# RoCS-MT: Robustness Challenge Set for Machine Translation

Rachel Bawden, Benoît Sagot

<figure><img src="image_1.png" /></figure>

## To cite this version:

Rachel Bawden, Benoît Sagot. RoCS-MT: Robustness Challenge Set for Machine Translation. WMT23 - Eighth Conference on Machine Translation, Dec 2023, Singapore, Singapore. pp.198–216. [hal-04300824](https://hal.science/hal-04300824v1)

HAL Id: hal-04300824

https://hal.science/hal-04300824v1

Submitted on 22 Nov 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# RoCS-MT: Robustness Challenge Set for Machine Translation

**Rachel Bawden**

Inria, Paris, France
rachel.bawden@inria.fr

**Benoît Sagot**

Inria, Paris, France
benoit.sagot@inria.fr

## Abstract

RoCS-MT, a Robust Challenge Set for Machine Translation (MT), is designed to test MT systems' ability to translate user-generated content (UGC) that displays non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. RoCS-MT is composed of English comments from Reddit, selected for their non-standard nature, which have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. In the context of the WMT23 test suite shared task, we analyse the models submitted to the general MT task for all from-English language pairs, offering some insights into the types of problems faced by state-of-the-art MT models when dealing with non-standard UGC texts. We compare automatic metrics for MT quality, including quality estimation to see if the same conclusions can be drawn without references. In terms of robustness, we find that many of the systems struggle with non-standard variants of words (e.g. due to phonetically inspired spellings, contraction, truncations, etc.), but that this depends on the system and the amount of training data, with the best overall systems performing better across all phenomena. GPT4 is the clear front-runner. However we caution against drawing conclusions about generalisation capacity as it and other systems could be trained on the source side of RoCS and also on similar data.

## 1 Introduction

As the quality of state-of-the-art machine translation (MT) systems is becoming indistinguishable in certain scenarios and domains from that of human translators (Kocmi et al., 2022), the task of tackling the translation non-standard texts is becoming an increasingly realisable aim. A considerable proportion of texts produced today are done so online in informal, unedited settings, e.g. on forums such as Twitter and Reddit, and MT is frequently to make posts accessible to a global audience. However, it

has been shown that MT still struggles with user-generated content (UGC) (Gupta et al., 2023), as the type of language can differ considerably from the edited texts that have traditionally been used to train and evaluate MT models.

The RoCS-MT challenge set (Robust Challenge Set for Machine Translation) is designed to provide a test bed for the automatic translation of non-standard UGC phenomena. It contains approximately 2k sentences from the online forum Reddit that have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. The sentences were selected specifically for the presence of non-standard phenomena, of which we provide manual annotations (e.g. spelling errors, devowelling, capitalisations, acronymisms, etc.). Inspired by other datasets such as the French Social Media Bank (Seddah et al., 2012) and its parallel component (Rosales Núñez et al., 2019), our aim is to provide an evaluation set that is more challenging than certain previous efforts, such as the commonly used MTNT dataset (Michel and Neubig, 2018). We also make different choices from most previous efforts concerning the guidelines for normalisation and translation of the source sentences. We choose to first normalise the source sentences before translation in order to optimise the quality of the translation and to reduce the arbitrariness that may be introduced when transferring non-standard variation to the target language (e.g. on which characters to apply spelling errors, how many characters to duplicate when elongating words). For normalisation, we aim to strike a balance between normalisation as much as possible while making sure that the normalised text remains natural.<fnref n="1" />

In this paper, we describe the creation of the challenge set, and in the context of the WMT23 test suite shared task, we analyse the models submitted

<footnote n="1">¹E.g. We choose to not normalise the acronym lol 'laughing out loud', as it is rarely/never used in its expanded form.</footnote>
to the general MT shared task for the from-English shared task language pairs: English→{Czech, German, Hebrew, Japanese, Russian, Ukrainian, Chinese} (en→{cs, de, he, ja, ru, uk, zh}). Through automatic and manual analysis of system outputs, we find that many of the phenomena remain challenging for most systems (in particular those that create potential out-of-vocabulary or rare words such as phonetically inspired spellings, contractions, devowelling and truncation). However, the difficulty varies depending on the phenomenon, the particular instance (notably how frequent the non-standard word is) and the system, especially with respect to the quantity of training data. The highest performing systems overall generally do better across the board on all phenomena, whereas the weaker systems struggle in particular with certain phenomena. GPT4 has a clear lead over other systems, correctly translating even some of the most challenging examples and sometimes (although inconsistently) reproducing non-standardness in its outputs. However conclusions are limited given that the training data is unknown (as is the case of other unconstrained systems).

We make the challenge set, system outputs, evaluation code and guidelines (for the normalisation, annotation and translation) openly available for research purposes.<fnref n="2" />

## 2 Related Work

Several parallel UGC datasets exist across different language pairs. While some are extracted automatically from crawled data (Ling et al., 2013; Vicente et al., 2016; Mubarak et al., 2020), a majority are based on monolingual sentences that are then translated into the target language (Sluyter-Gäthje et al., 2018; Michel and Neubig, 2018; Rosales Núñez et al., 2019; Fujii et al., 2020; McNamee and Duh, 2022). The closest to our RoCS-MT dataset are (Michel and Neubig, 2018) and (Rosales Núñez et al., 2019), which were designed to contain challenging non-standard phenomena, whereas many of the existing datasets do not apply any such filter. Like RoCS-MT, the MTNT dataset (Michel and Neubig, 2018) contains texts from Reddit. To target non-standard language, they select sentences that have a low probability using a language model trained on standard data. In practice, and as shown by Rosales Núñez et al. (2019), the amount of non-standard language remains limited with this

method. Rosales Núñez et al. (2019) base their parallel dataset on the French Social Media Bank dataset (Seddah et al., 2012), which targets non-standard language by searching for specific non-standard keywords. They show that this leads to a higher level of non-standard language, although the method is by nature more biased towards the keywords and phenomena used for data selection. An error analysis of the dataset was conducted in (Rosales Núñez et al., 2021), showing MT quality (using BLEU) for different UGC phenomena.

Despite significant effort to describe and classify UGC phenomena (Michel and Neubig, 2018; San-guinetti et al., 2020), there is no consensus as to how texts should be normalised (and indeed translated). One extreme is to normalise all phenomena to standard forms, as is often done in lexical normalisation tasks (Han and Baldwin, 2011; van der Goot et al., 2021), but which in several cases would lead to unnatural outputs (e.g. if *lol* and *lmao*, were systematically normalised to *laughing out loud* and *laughing my ass off*). This makes translation difficult too, as the translations would also be unnatural. At the other end of the spectrum is the choice to not normalise source texts and in addition to attempt to translate the phenomena into the target language, with the disadvantage that some phenomena are language-specific<fnref n="3" /> and others would result in arbitrary decisions being made such as to which characters to apply spelling errors. The current datasets targeting particularly non-standard phenomena choose to at least in part transfer some phenomena to the target language, whereas we adopt a higher degree of normalisation (see Section 3.1.1 for more details), producing standard but natural-sounding translations.

## 3 Challenge Set Creation

### 3.1 Data Sourcing and Selection

The source sentences are taken from English posts on discussion platform Reddit⁴ using the API.⁵ We do not target a particular variety of English

<footnote n="3">³Two examples of this are French *verlan*, which consists in inverting syllables in words (e.g. *louche→chelou* 'bizarre') and English cockney rhyming slang (e.g. *loaf* meaning *head* thanks to its rhyme with the expression *loaf of bread*). However, even phenomena that do exist crosslingually do not necessarily apply to the same words (e.g. the use of digits to replace their homophones as in *2day* 'today', where the translation does not necessarily contain a homophone of a digit in the target language).

⁴www.refox.com

⁵Using the free version of the API (December 2022).</footnote>

<footnote n="2">²https://github.com/rbawden/RoCS-MT</footnote>
(e.g. British, American, etc.) and even include some non-native English,<fnref n="6" /> although we do not include code-switched texts. We get an initial pool of posts by searching for specific keywords from a manually drawn-up list as in (Sanguinetti et al., 2020), e.g. *ttyl*, *ppl*, *gr8*, *alot*. The full list is given in Appendix A. For each keyword, we crawled both Reddit-wide and 3 specific subreddits (CasualUK, MadeMeSmile and entertainment) to ensure a diversity of informal topics<fnref n="7" /> at 6-month intervals between 2017 and 2022.

Once we had the initial pool of examples, in order to reduce the number of posts to manually review, we applied a very coarse-grained in-house ‘non-standardness’ classifier that we had trained on a small set of manually annotated tweets according to 4 labels (standard, mildly non-standard, moderately non-standard and very non-standard), and look at posts whose title or text was marked as anything other than ‘standard’. From those posts, we manually select titles and passages from the text that contain interesting non-standard phenomena, including sentences not containing the initial keyword associated with the post. This means that although our initial search process is biased to our word list, the effect is diminished by taking additional non-standard phenomena. We automatically filter out any 18+ content (using the Reddit meta-information), and manually filter out any content that is sexually inappropriate, insulting or deals with sensitive (potentially triggering) topics such as suicide or drug addiction.

### 3.1.1 Sentence Splitting and Normalisation

We start by manually splitting the texts into sentences. In many cases, this corresponds to splitting on final punctuation (e.g. full stop, exclamation marks, etc.). However, the non-standard nature of the texts increases the number of cases where texts are split in places that are not marked by punctuation or where punctuation or newlines are added unexpectedly in the middle of what would ordinarily be considered a sentence.

For instance, the sequence *I went grocery shopping I'm down to my last dollars soon (...)* was split into the first sentence *I went grocery shopping*

and the second sentence beginning with *I'm down to my last dollars soon*, despite the lack of a final punctuation between *shopping* and *I'm*.

The first author (a native English speaker) manually normalised each of the sentences produced by our manual sentence splitting, seeking help from people knowledgeable in the topics (e.g. video gaming) where necessary. The complete normalisation guidelines with examples can be found in the dedicated Github repository.<fnref n="8" /> As with any guidelines for dealing with complex and evolving non-standard phenomena, the decisions made are certainly not bulletproof and are likely to evolve in future work. Our aim was to reach a compromise between (i) normalising as much as possible of the text while (ii) rendering the output natural and realistic and (iii) not over-normalising such as to remove the style of the original text. We therefore normalise words such that the normalised variant could be spontaneously and naturally used.

### 3.1.2 Translation

Translation of the English sentences was carried out by paid professional translators. They had access to the original posts and both the raw and normalised versions of each sentence. Translation was carried out at the sentence level (following the manual segmentation and using as the source the normalised translation), although the translators had access to surrounding linguistic context, as well as additional context and translation notes provided by the first author during the normalisation step. There were also several exchanges between the first author and the translators in order to provide additional context and to answer questions. In order to preserve author anonymity, translators did not have access to meta-information about the authors (e.g. their gender). A single translation was produced for each sentence (we left the choice of speaker gender to the translators) with the exception of Ukrainian, for which two translations were produced for sentences where the speaker gender has an impact.

The target languages were chosen to cover four of those in the WMT2023 general translation task (Czech, German, Ukrainian and Russian), as well as French, which is an important language for our own research, although we do not analyse the French portion of the data in this article.

**Translation Guidelines** Translators were provided with guidelines (see Appendix B). They were

<footnote n="6">⁶We do not have access to any personal information about the post authors, but we know this because some posters apologise for their level of English in the posts included.</footnote>

<footnote n="7">⁷The subreddits were chosen to have topics that were informal and could have a reasonable number of posts, although in reality, the number of non-standard posts found from these specific subreddits was limited.</footnote>

<footnote n="8">⁸https://github.com/rbawden/RoCS-MT</footnote>
instructed to translate the normalised versions of each sentence into the target language, using standard language but best matching the intention, naturalness and familiarity level of the sentence, similar to the guidelines set out in (McNamee and Duh, 2022). The decision to use standard language was to avoid the arbitrariness associated with attempting to reproduce non-standard phenomena in translation, which would make comparisons, particularly automatic ones, more difficult (e.g. which characters to alter to reproduce a spelling error, how many characters to repeat in the case of expressive repetition, etc.). They were also instructed to respect the manual segmentation provided,<fnref n="9" /> to respect punctuation choices made in the source where appropriate (e.g. conserving full stops) and to preserve English words in meta-linguistic discussions (i.e. where authors are writing specifically about English words). As in the normalisation guidelines, abbreviations, acronyms and simplifications were to be expanded unless the result would not make a natural sentence that could realistically be found. However, abbreviations linked to the names of places and institutions were to be kept as they were if used as such in the target language (e.g. French *OTAN* for English *NATO*). They were requested not to use MT systems to help them translate in order not to bias the translations produced.

## 3.2 Challenge Set Subsets

We create four subsets of the challenge set to test the impact of sentence segmentation (manual or automatic using spaCy) and of normalisation (manual or none, i.e. the original raw text):

* manseg-raw: Manual segmentation with original (raw) text

* manseg-norm: Manual segmentation with manual normalisation

* spacyseg-raw: spaCy segmentation with original (raw) text

* spacyseg-norm: spaCy segmentation with manual normalisation<fnref n="10" />

As shown in Section 3.3, the two different segmentation methods result in different numbers of

individual sentences, and automatic segmentation with spaCy differs depending on whether the text has been normalised or not. In practice, in this article, we focus only on the manseg-raw and manseg-norm subsets, although we also release the system outputs for the spacyset- subset. We leave research on these other subsets (i.e. looking at the impact of sentence segmentation) to future work.

## 3.3 Dataset Characteristics

Some basic quantitative characteristics of the data are given Table 1.

**Impact of sentence splitting** While the number of sentences is fixed for the manual segmentation, spaCy segmentation is highly dependent on whether the text has been normalised or not, likely due to the tool being less well adapted to non-standard text; when applied to raw text, the resulting number of sentences is far lower than manual segmentation (1660 vs. 1922), whereas the resulting number of sentences is more similar to manual segmentation when applied to the normalised text.

**Tokenisation** Normalisation impacts the number of tokens in the texts, as well as the number of unique tokens. When comparing the two normalised subsets on the one hand and the two raw subsets on the other (i.e. differing only in the sentence splitting), the number of tokens differs due to the fact that automatic segmentation tends to oversplit sentences on punctuation that in the manual segmentation would remain part of a token in the preceding sentence. The number of unique tokens inevitably drops after normalising, due to the homogenisation of non-standard forms (7175 vs. 6612) for manual segmentation.

**Normalisation Types** We manually annotated the texts for non-standard phenomena (e.g. spelling errors, acronyms, devowelling, capitalisation, pronoun drop, etc.), with the possibility of there being several types for a single span of text. Our annotations are at the word-level, with some phenomena spanning several words (e.g. capitalisation). Table 2 provides some statistics for the annotations occurring in at least 10 sentences, and some examples are given in Examples 1-4.

(1) **_btw_** I **_wud_** prefer them **_rllly_** quick.
By the way, I **_would_** prefer them **_really_** quick.
**_acronym_** contraction **_devow._**
**capitalisation**
**punct_diff**

<footnote n="9">⁹A segment’s translation can contain several sentences but sentence boundaries cannot be overridden.</footnote>

<footnote n="10">¹⁰The spaCy segmentation was obtained by concatenating all normalised sentences from a single text and then automatically splitting.</footnote>
<table><caption>Table 1: Basic statistics of the four subsets of the test suite. Tokens are defined as whitespace delimited character sequences. Sentences can either come from post titles or the body of the post.</caption><thead><tr><th>Subset</th><th>Seg.</th><th>Norm.</th><th>#sents.</th><th>#toks.</th><th>#toks. (unique)</th><th>Ave. sent. len.</th><th>#posts</th><th>#titles</th><th>#body</th></tr></thead><tbody><tr><td>manseg-raw</td><td>Manual</td><td>×</td><td>1922</td><td>27971</td><td>7175</td><td>14.55</td><td rowspan="4">391</td><td rowspan="4">80</td><td rowspan="4">263</td></tr><tr><td>manseg-norm</td><td>Manual</td><td>✓</td><td>1922</td><td>28800</td><td>6612</td><td>14.98</td></tr><tr><td>spacyseg-raw</td><td>spaCy</td><td>×</td><td>1660</td><td>28095</td><td>7297</td><td>16.92</td></tr><tr><td>spacyseg-norm</td><td>spaCy</td><td>✓</td><td>1996</td><td>28881</td><td>6615</td><td>14.47</td></tr></tbody></table>

<table><caption>Table 2: For each annotation appearing in at least 10 sentences, the number of words, unique words (lower-cased) and sentences for which it appears.</caption><thead><tr><th>Annotation</th><th>#toks</th><th>#diff toks</th><th>#sents</th></tr></thead><tbody><tr><td>punct_diff</td><td>2500</td><td>136</td><td>1259</td></tr><tr><td>capitalisation</td><td>2122</td><td>802</td><td>1059</td></tr><tr><td>norm_punct</td><td>542</td><td>46</td><td>339</td></tr><tr><td>acronymisation</td><td>329</td><td>100</td><td>277</td></tr><tr><td>phonetic_distance</td><td>566</td><td>285</td><td>268</td></tr><tr><td>spelling_error</td><td>345</td><td>306</td><td>261</td></tr><tr><td>spacing</td><td>294</td><td>111</td><td>250</td></tr><tr><td>truncation</td><td>203</td><td>104</td><td>169</td></tr><tr><td>contraction</td><td>161</td><td>37</td><td>146</td></tr><tr><td>devowelling</td><td>137</td><td>33</td><td>122</td></tr><tr><td>elongation</td><td>139</td><td>96</td><td>117</td></tr><tr><td>pronoun_drop</td><td>114</td><td>1</td><td>110</td></tr><tr><td>word_drop</td><td>97</td><td>2</td><td>85</td></tr><tr><td>grammar</td><td>75</td><td>54</td><td>73</td></tr><tr><td>inflection</td><td>78</td><td>64</td><td>67</td></tr><tr><td>lex</td><td>65</td><td>52</td><td>63</td></tr><tr><td>article_drop</td><td>69</td><td>1</td><td>63</td></tr><tr><td>scrambled</td><td>38</td><td>36</td><td>37</td></tr><tr><td>words_to_digits</td><td>45</td><td>18</td><td>37</td></tr><tr><td>word_to_symbol</td><td>26</td><td>12</td><td>22</td></tr><tr><td>dialectism</td><td>24</td><td>15</td><td>22</td></tr><tr><td>double_to_single_character</td><td>17</td><td>10</td><td>17</td></tr><tr><td>word_add</td><td>16</td><td>13</td><td>15</td></tr><tr><td>digits_to_words</td><td>16</td><td>13</td><td>14</td></tr><tr><td>interjection</td><td>13</td><td>8</td><td>10</td></tr><tr><td>surrounding'emphasis</td><td>12</td><td>11</td><td>10</td></tr><tr><td>word_order</td><td>11</td><td>11</td><td>10</td></tr><tr><td>emoticon</td><td>10</td><td>10</td><td>10</td></tr></tbody></table>

(2) So any idea s on wot I shud be
So any ideas on what I should be ?
spacing phon._dst. contraction punct.

(3) Dhat kwik beizh fawks jmmpd
That quick beige fox jumped
phon._dst. phon._dst. phon._dst. phon._dst.

(4) Em HOW DARE YOU SWEAR IN
EM: How dare you swear in
caps. caps. caps. caps. caps.
punct_diff

FRONT OF MY SUN
front of my son ?
caps. caps. caps. spelling punct_diff

## 4 Translation Systems

In this article, we evaluate the systems submitted to the general translation task at WMT2023. There

are both constrained and unconstrained systems, the two settings presenting significant differences in training data that should be taken into account when comparing systems.

**Constrained systems** Constrained systems followed similar strategies, with many systems doing data filtering/cleaning and data augmentation, using either bilingual or multilingual models and reranking. The constrained systems submitted were AIRC (Rikters and Miwa, 2023), ANVITA, CUNI-Transformer and CUNI-DocTransformer (Popel, 2020) (we refer to these system as CUNI-Trans and CUNI-DocTrans to save space in the results tables), CUNI-GA (Jon et al., 2023), HW-TSC (Wu et al., 2023b), IOL_Research (Zhang, 2023), NAIST-NICT (Deguchi et al., 2023), Samsung_Research.Philippines (Cruz, 2023) (hereafter Samsung_RP), SKIM (Kudo et al., 2023) and UvA-LTL (Wu et al., 2023a).

**Unconstrained systems** As in previous years of the shared task, translations were produced from anonymised online systems, corresponding in this addition to ONLINE-{A,B,G,M,W,Y} sub-missions. This year, translations from GPT4 were also produced using 5 few-shot examples (GPT4-5shot).<fnref n="11" /> Note that caution should be taken when comparing results from GPT4, given that it is very possible that source sentences from RoCS-MT are included in GPT4’s training data. Two systems based on NLLB (Team et al., 2022) were also submitted in the context of the metrics shared task: NLLB_Greedy and NLLB_MBR_BLEU (hereafter NLLB_MBR), which both rely on the same model but differ by the decoding strategy, either standard (greedy) or based on the Minimum Bayes Risk strategy (Freitag et al., 2022). A number of unconstrained systems were also submitted by participants, namely Lan-BridgeMT (Wu and Hu, 2023), KYB, GTCOM (Zong, 2023), (Li et al., 2023), PROMT (Molchanov and Kovalenko, 2023), Yishu

<footnote n="11">¹¹The prompt used is the sentence-level prompt from (Hendy et al., 2023), which is also shown in Appendix C.</footnote>
(Min et al., 2023) and ZengHuiMT (Zeng, 2023).

# 5 Evaluation and Analysis

Evaluation of UGC translation is more challenging than standard text; a correct translation can either be standard or non-standard in the target language, and there may be multiple ways of being non-standard that may not all be covered by available references. In our case, we chose to produce standard reference translations (See Section 3.1.2). Any system that produces non-standard language may therefore be underestimated using reference-based metrics.

We test three different metrics (BLEU, COMET and COMET-QE) to evaluate the systems' translations of RoCS-MT, looking at how coherent they are between each other, and whether it is possible to use quality estimation to evaluate MT robustness in order to remove the need for reference translations (Section 5.1). We also look at the MT quality of each system per phenomenon by calculating COMET scores over subsets of the data. Finally, we perform a qualitative analysis, manually looking at how the different systems handle UGC phenomena, and confirming some of the trends using some simple automatic analyses (Section 5.2).

## 5.1 Automatic evaluation

BLEU (Papineni et al., 2002), as a surface-level metric, is intuitively not robust to variation. It is therefore likely to be particularly ill-adapted to MT robustness evaluation, since MT systems' outputs can display standard or non-standard characteristics. We choose nevertheless to test this here, calculating BLEU scores using the sacreBLEU toolkit (Post, 2018).<fnref n="12" /> We compare BLEU to reference-based COMET (Rei et al., 2020)¹³ for those language pairs for which we have a reference, and to COMET's reference-less (quality estimation) version, which we refer to as COMET-QE (Rei et al., 2022).¹⁴ We notably aim to test whether it is possible to use COMET-QE for evaluation rather than reference-based COMET, which would remove the dependency on reference translations and make evaluation possible for a wider range of languages.

Ukrainian has two reference translations for sentences for which the speaker's gender results in different translations is ambiguous between male

and female. While BLEU is designed to handle multiple references, this is not built into COMET. For these sentences, we choose to take the best COMET score of the two references. For COMET and COMET-QE, which also use the source sentence, we choose to evaluate system outputs against both the manseg-norm and manseg-raw source sentences, regardless of which set was translated by the system and take the highest score of all the source-reference combinations. This covers the case where non-standard (i.e. raw) sentences are normalised during the translation process.

We provide full results for COMET and COMET-QE in Table 3 and 4 respectively, and we include results for BLEU in Table 7 in Appendix D.

**How coherent are the metrics?** The trends of the three metrics are similar but not at all systematic (in terms of rankings) when evaluating translations of the normalised data (manseg-norm), with the same systems getting the highest scores across language pairs (amongst the best systems being ONLINE-W, ONLINE-B, GPT4). However, there are some clear inconsistencies between BLEU and the two COMET metrics when evaluating non-standard data (manseg-raw). For example GPT4 is ranked above other systems by COMET and COMET-QE, whereas the BLEU scores of other systems (and in particular ONLINE-W and sometimes ONLINE-B) are higher. This indicates that GPT4 outputs are more surfacically different from the reference translations, which could be a result of paraphrasing or non-standard translations rather than a reflection of MT quality, especially given the high scores by COMET.

This confirms that BLEU is poorly adapted to evaluating MT robustness and could even lead to misleading conclusions, confirming previous conclusions drawn by Rosales Núñez et al. (2021) about the inadequacy of BLEU for the evaluation of UGC MT. On the other hand, COMET-QE scores show more similar trends to COMET, suggesting that it could be possible to use it to evaluate without having to produce reference translations. We nevertheless add that COMET remains an automatic metric that does not produce perfect correlation with human judgments, more research would be necessary to stress-test the metric for MT robustness evaluation, particularly in terms of evaluating which of COMET and COMET-QE is better correlated with human judgments.

<footnote n="12">¹²case:mixed|eff:no|tok:13a|smooth:exp|v:2.2.1
¹³We use the default wmt22-comet-da model.
¹⁴We use the default wmt22-cometkiwi-da.</footnote>
**Which systems come out on top?** The highest performing systems are the unconstrained online systems, with GPT4 getting significantly higher COMET and COMET-QE scores than other systems when translating non-standard (raw) text for all languages tested. Other systems that tend to produce high scores are ONLINE-W and to a lesser extent ONLINE-B. Apart from these online models, both NLLB models are the best-scoring ones, which might come from the fact that they are highly multilingual and therefore could be more robust to language variation. The constrained systems, whilst not the highest performing systems, appear to get comparable scores to at least some of the online systems.

**Which systems are most robust?** This question is linked to the previous question about MT quality on non-standard data. To take into account the base performance of the systems, we look at the difference in score between each system's translation of the non-standard sentences and their normalised versions (also in the previously mentioned Tables 3 and 4. While there is a general trend that the higher performing systems also have a smaller difference in quality (i.e. they are also more robust), there are some stand-out systems. GPT4 is the system with the lowest quality difference between original and normalised sentences for all language pairs tested. The NLLB models also have a low delta between the two subset, lower than or comparable to some of the more robust online systems. Similarly to the previous question, constrained systems are not the most robust in terms of their score difference. Notably for en–cs and en–de, the score differences are amongst the highest. However, some of the systems do show performance in the same ballpark as some of the online systems.

**Automatic analysis by UGC phenomenon** In order to analyse how systems handle different non-standard phenomena, we evaluate sentences by annotation types, by calculating COMET and COMET-QE scores for sentences containing at least one occurrence of a particular normalisation annotation. COMET results are given in Table 5 and we include a fuller analysis for COMET-QE results in Table 8 in Appendix E. Note that we only include annotation types that appear in at least 50 sentences, and that the ‘all’ column refers to the scores over all sentences and not just the ones annotated for UGC phenomena.

Scores are not directly comparable across annotation type. Performance by annotation type is consistent with previous conclusions, with GPT4 getting the highest scores across the board, and online systems and NLLB also doing well. It is striking that the systems that have higher scores in general tend to do better across the board on all annotation types, whereas the lower-scoring systems struggle with certain non-standard phenomena. They correspond in particular to phonetic distance, where a word is spelt differently according to how it is pronounced (e.g. HEERE'Z A QWESHCHUN FER YA 'Here's a question for you'), contractions (e.g. wud 'would'), devowelling (e.g. nvr 'never'), truncation (e.g. intro 'introductory') and spelling errors. These are notably phenomena that could well result in out-of-vocabulary words.

**Are certain language pairs more difficult than others?** It is tricky to compare across language pairs, since scores are not comparable. However, there are some indications that the en–cs set is more challenging, given the low scores across multiple annotation types for all systems other than GPT4. The fact that GPT4 has high scores for all annotation types listed shows that the lower scores of other models are not due to quality issues in the reference translations, and provides an upper bound against which other systems can be compared, thereby indicating that the systems struggled more.

## 5.2 Qualitative analysis

**Non-standard variants of words** Many of the non-standard phenomena that characterise the texts (e.g. acronyms, truncations, contractions, devowelling) represent a similar difficulty to unknown or rare tokens in MT. The treatment of these words differs according to the system used, and inevitably largely on the training data of the model. Many of the constrained systems struggle to translate such words, either copying the words into the translation or omitting them entirely. The degree to which the systems succeed in correctly translating these words appears to depend on how common it is. For example, *tho*, phonetically-inspired spelling of *though*, was translated successfully by multiple systems, although the devowelled word *tmro* 'tomorrow' proved more difficult.

**Markers of expressivity** It is common for UGC texts to have markers of expressivity such as capitalisation or repetition of letters. We removed these markers in our normalised versions and reference
<table><caption>Table 3: COMET scores of systems on the manseg-norm and manseg-raw subsets.</caption><thead><tr><th rowspan="2">Systems</th><th colspan="3">en-cs</th><th colspan="3">en-de</th><th colspan="3">en-ru</th><th colspan="3">en-uk</th></tr><tr><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th></tr></thead><tbody><tr><td colspan="13" style="text-align: left; font-weight: bold;">Unconstrained</td></tr><tr><td>GPT4-5shot</td><td>0.857</td><td>0.825</td><td><strong>0.031</strong></td><td>0.869</td><td>0.837</td><td><strong>0.032</strong></td><td>0.818</td><td>0.793</td><td><strong>0.025</strong></td><td>0.858</td><td>0.838</td><td><strong>0.021</strong></td></tr><tr><td>ONLINE-A</td><td>0.836</td><td>0.724</td><td><strong>0.112</strong></td><td>0.858</td><td>0.771</td><td><strong>0.087</strong></td><td>0.806</td><td>0.730</td><td><strong>0.076</strong></td><td>0.830</td><td>0.741</td><td><strong>0.090</strong></td></tr><tr><td>ONLINE-B</td><td>0.844</td><td>0.760</td><td><strong>0.084</strong></td><td>0.867</td><td>0.815</td><td><strong>0.052</strong></td><td>0.812</td><td>0.748</td><td><strong>0.063</strong></td><td>0.856</td><td>0.787</td><td><strong>0.069</strong></td></tr><tr><td>ONLINE-G</td><td>0.812</td><td>0.699</td><td><strong>0.113</strong></td><td>0.847</td><td>0.763</td><td><strong>0.084</strong></td><td>0.828</td><td>0.773</td><td><strong>0.055</strong></td><td>0.853</td><td>0.803</td><td><strong>0.050</strong></td></tr><tr><td>ONLINE-M</td><td>0.838</td><td>0.720</td><td><strong>0.118</strong></td><td>0.847</td><td>0.714</td><td><strong>0.133</strong></td><td>0.787</td><td>0.686</td><td><strong>0.102</strong></td><td>-</td><td>-</td><td>-</td></tr><tr><td>ONLINE-W</td><td>0.865</td><td>0.782</td><td><strong>0.082</strong></td><td>0.892</td><td>0.809</td><td><strong>0.083</strong></td><td>0.834</td><td>0.786</td><td><strong>0.048</strong></td><td>0.862</td><td>0.819</td><td><strong>0.043</strong></td></tr><tr><td>ONLINE-Y</td><td>0.819</td><td>0.725</td><td><strong>0.095</strong></td><td>0.862</td><td>0.795</td><td><strong>0.067</strong></td><td>0.814</td><td>0.756</td><td><strong>0.058</strong></td><td>0.823</td><td>0.750</td><td><strong>0.073</strong></td></tr><tr><td>NLLB_MBR</td><td>0.837</td><td>0.792</td><td><strong>0.045</strong></td><td>0.836</td><td>0.786</td><td><strong>0.049</strong></td><td>0.799</td><td>0.755</td><td><strong>0.045</strong></td><td>0.826</td><td>0.778</td><td><strong>0.049</strong></td></tr><tr><td>NLLB_Greedy</td><td>0.839</td><td>0.791</td><td><strong>0.049</strong></td><td>0.837</td><td>0.783</td><td><strong>0.054</strong></td><td>0.798</td><td>0.753</td><td><strong>0.046</strong></td><td>0.827</td><td>0.775</td><td><strong>0.052</strong></td></tr><tr><td>Lan-BridgeMT</td><td>0.820</td><td>0.723</td><td><strong>0.097</strong></td><td>0.830</td><td>0.737</td><td><strong>0.094</strong></td><td>0.784</td><td>0.699</td><td><strong>0.084</strong></td><td>0.795</td><td>0.705</td><td><strong>0.090</strong></td></tr><tr><td>GTCOM_Peter</td><td>0.822</td><td>0.725</td><td><strong>0.098</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.807</td><td>0.714</td><td><strong>0.092</strong></td></tr><tr><td>PROMT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.780</td><td>0.685</td><td><strong>0.095</strong></td><td>-</td><td>-</td><td>-</td></tr><tr><td>ZengHuiMT</td><td>0.811</td><td>0.717</td><td><strong>0.094</strong></td><td>0.833</td><td>0.760</td><td><strong>0.073</strong></td><td>0.772</td><td>0.706</td><td><strong>0.066</strong></td><td>0.786</td><td>0.709</td><td><strong>0.077</strong></td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">Unconstrained</td></tr><tr><td>AIRC</td><td>-</td><td>-</td><td>-</td><td>0.779</td><td>0.669</td><td><strong>0.110</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CUNI-Trans</td><td>0.831</td><td>0.719</td><td><strong>0.112</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CUNI-DocTrans</td><td>0.840</td><td>0.694</td><td><strong>0.146</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CUNI-GA</td><td>0.840</td><td>0.694</td><td><strong>0.146</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

<table><caption>Table 4: COMET-QE scores of systems on the manseg-norm and manseg-raw subsets.</caption><thead><tr><th rowspan="2">Systems</th><th colspan="3">en-cs</th><th colspan="3">en-de</th><th colspan="3">en-he</th><th colspan="3">en-ja</th><th colspan="3">en-ru</th><th colspan="3">en-uk</th><th colspan="3">en-zh</th></tr><tr><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th></tr></thead><tbody><tr><td colspan="13" style="text-align: left; font-weight: bold;">Unconstrained</td></tr><tr><td>GPT4-5shot</td><td>0.817</td><td>0.800</td><td><strong>0.018</strong></td><td>0.822</td><td>0.805</td><td><strong>0.017</strong></td><td>0.806</td><td>0.793</td><td><strong>0.013</strong></td><td>0.846</td><td>0.838</td><td><strong>0.008</strong></td><td>0.806</td><td>0.789</td><td><strong>0.017</strong></td><td>0.809</td><td>0.797</td><td><strong>0.012</strong></td><td>0.797</td><td>0.786</td><td><strong>0.011</strong></td></tr><tr><td>ONLINE-A</td><td>0.807</td><td>0.724</td><td><strong>0.083</strong></td><td>0.816</td><td>0.765</td><td><strong>0.050</strong></td><td>0.807</td><td>0.737</td><td><strong>0.070</strong></td><td>0.824</td><td>0.772</td><td><strong>0.052</strong></td><td>0.807</td><td>0.750</td><td><strong>0.058</strong></td><td>0.791</td><td>0.726</td><td><strong>0.065</strong></td><td>0.786</td><td>0.725</td><td><strong>0.061</strong></td></tr><tr><td>ONLINE-B</td><td>0.814</td><td>0.756</td><td><strong>0.058</strong></td><td>0.821</td><td>0.793</td><td><strong>0.028</strong></td><td>0.812</td><td>0.767</td><td><strong>0.045</strong></td><td>0.848</td><td>0.822</td><td><strong>0.027</strong></td><td>0.808</td><td>0.761</td><td><strong>0.047</strong></td><td>0.805</td><td>0.759</td><td><strong>0.046</strong></td><td>0.805</td><td>0.766</td><td><strong>0.039</strong></td></tr><tr><td>ONLINE-G</td><td>0.791</td><td>0.705</td><td><strong>0.086</strong></td><td>0.812</td><td>0.766</td><td><strong>0.045</strong></td><td>0.786</td><td>0.720</td><td><strong>0.067</strong></td><td>0.782</td><td>0.700</td><td><strong>0.082</strong></td><td>0.821</td><td>0.784</td><td><strong>0.036</strong></td><td>0.809</td><td>0.775</td><td><strong>0.034</strong></td><td>0.765</td><td>0.704</td><td><strong>0.062</strong></td></tr><tr><td>ONLINE-M</td><td>0.807</td><td>0.710</td><td><strong>0.096</strong></td><td>0.810</td><td>0.724</td><td><strong>0.086</strong></td><td>-</td><td>-</td><td>-</td><td>0.798</td><td>0.711</td><td><strong>0.088</strong></td><td>0.790</td><td>0.702</td><td><strong>0.089</strong></td><td>-</td><td>-</td><td>-</td><td>0.762</td><td>0.692</td><td><strong>0.069</strong></td></tr><tr><td>ONLINE-W</td><td>0.822</td><td>0.765</td><td><strong>0.057</strong></td><td>0.822</td><td>0.780</td><td><strong>0.042</strong></td><td>-</td><td>-</td><td>-</td><td>0.822</td><td>0.790</td><td><strong>0.031</strong></td><td>0.819</td><td>0.786</td><td><strong>0.033</strong></td><td>0.812</td><td>0.782</td><td><strong>0.030</strong></td><td>0.802</td><td>0.767</td><td><strong>0.036</strong></td></tr><tr><td>ONLINE-Y</td><td>0.799</td><td>0.732</td><td><strong>0.067</strong></td><td>0.822</td><td>0.786</td><td><strong>0.036</strong></td><td>0.808</td><td>0.753</td><td><strong>0.056</strong></td><td>0.842</td><td>0.811</td><td><strong>0.031</strong></td><td>0.814</td><td>0.764</td><td><strong>0.050</strong></td><td>0.787</td><td>0.731</td><td><strong>0.056</strong></td><td>0.796</td><td>0.752</td><td><strong>0.044</strong></td></tr><tr><td>NLLB_MBR</td><td>0.802</td><td>0.762</td><td><strong>0.040</strong></td><td>0.801</td><td>0.763</td><td><strong>0.038</strong></td><td>0.796</td><td>0.756</td><td><strong>0.040</strong></td><td>0.721</td><td>0.682</td><td><strong>0.039</strong></td><td>0.794</td><td>0.754</td><td><strong>0.040</strong></td><td>0.784</td><td>0.744</td><td><strong>0.040</strong></td><td>0.617</td><td>0.596</td><td><strong>0.021</strong></td></tr><tr><td>NLLB_Greedy</td><td>0.806</td><td>0.765</td><td><strong>0.041</strong></td><td>0.802</td><td>0.761</td><td><strong>0.041</strong></td><td>0.795</td><td>0.756</td><td><strong>0.039</strong></td><td>0.749</td><td>0.711</td><td><strong>0.058</strong></td><td>0.795</td><td>0.754</td><td><strong>0.041</strong></td><td>0.786</td><td>0.745</td><td><strong>0.041</strong></td><td>0.664</td><td>0.645</td><td><strong>0.019</strong></td></tr><tr><td>Lan-BridgeMT</td><td>0.799</td><td>0.724</td><td><strong>0.075</strong></td><td>0.800</td><td>0.741</td><td><strong>0.064</strong></td><td>0.797</td><td>0.757</td><td><strong>0.040</strong></td><td>0.827</td><td>0.774</td><td><strong>0.053</strong></td><td>0.796</td><td>0.724</td><td><strong>0.071</strong></td><td>0.769</td><td>0.696</td><td><strong>0.072</strong></td><td>0.803</td><td>0.799</td><td><strong>0.011</strong></td></tr><tr><td>GTCOM_Peter</td><td>0.796</td><td>0.722</td><td><strong>0.074</strong></td><td>-</td><td>-</td><td>-</td><td>0.797</td><td>0.719</td><td><strong>0.077</strong></td><td>-</td><td>0.788</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.774</td><td>0.704</td><td><strong>0.070</strong></td><td>-</td><td>-</td><td>-</td></tr><tr><td>KYB</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PROMT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Yishu</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ZengHuiMT</td><td>0.781</td><td>0.713</td><td><strong>0.067</strong></td><td>0.792</td><td>0.748</td><td><strong>0.045</strong></td><td>0.790</td><td>0.734</td><td><strong>0.055</strong></td><td>0.828</td><td>0.791</td><td><strong>0.037</strong></td><td>0.772</td><td>0.724</td><td><strong>0.048</strong></td><td>0.748</td><td>0.696</td><td><strong>0.052</strong></td><td>0.772</td><td>0.711</td><td><strong>0.061</strong></td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">Constrained</td></tr><tr><td>AIRC</td><td>-</td><td>-</td><td>-</td><td>0.763</td><td>0.684</td><td><strong>0.079</strong></td><td>-</td><td>-</td><td>-</td><td>0.779</td><td>0.701</td><td><strong>0.078</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ANVITA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.797</td><td>0.716</td><td><strong>0.089</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.630</td><td>0.536</td><td><strong>0.094</strong></td></tr><tr><td>CUNI-Trans</td><td>0.798</td><td>0.705</td><td><strong>0.093</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CUNI-DocTrans</td><td>0.803</td><td>0.677</td><td><strong>0.126</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CUNI-GA</td><td>0.803</td><td>0.677</td><td><strong>0.126</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.793</td><td>0.740</td><td><strong>0.054</strong></td></tr><tr><td>HW-TSC</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>IOL_Research</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NAIST-NICT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Samsung_RP</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.797</td><td>0.732</td><td><strong>0.065</strong></td><td>-</td><td>0.830</td><td>0.764</td><td><strong>0.066</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SKIM</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.837</td><td>0.785</td><td><strong>0.052</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>UvA-LTL</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.799</td><td>0.731</td><td><strong>0.068</strong></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

translations for consistency. However, there are no guidelines as to how the different systems should translate them: either preserving the markers in the translation or normalising as we do in the reference. What we observe is variable behaviour depending on both the system and on the specific textual instances. Table 6 shows one of the more extreme examples of character repetition. Most of the systems fail to translate the words with character repetition and instead copy them (the case of *moooorreeee* 'more', *poolllliitte* 'polite' and *Discouuurrrse* 'Discourse'). There is greater robustness for the word *neeeeed* 'need', which is translated correctly as *brauchen* by a majority of systems, but not by AIRC, ONLINE-M and ONLINE-W, which is interesting since ONLINE-W obtains very high BLEU scores for en-de. When systems do translate the words, it tends to be the standard form that is generated (i.e. without repetition). The only example here is GPT4, which translates *moooorrrreeee* as *viiiieeeeel* 'viel', conserving the expressivity of the source sentence. What is interesting is that this behaviour is far from consistent for GPT4, with the other non-standard words in the same example not undergoing the same treatment. This could suggest that while the system succeeded in deciphering the
<table><caption>Table 5: COMET scores by normalisation annotation type and averaged over all RoCS-MT sentences ('all').</caption><thead><tr><th>System</th><th>acronymisation</th><th>article_drop</th><th>capitalisation</th><th>contraction</th><th>devoelling</th><th>elongation</th><th>grammar</th><th>infection</th><th>lex.choice</th><th>norm_punct</th><th>phonetic_distance</th><th>pronoun_drop</th><th>punct_diff</th><th>spacing</th><th>spelling_error</th><th>truncation</th><th>word_drop</th><th>all</th></tr></thead><tbody><tr><td colspan="20">en-cs</td></tr><tr><td>GPT4-5shot</td><td>0.785</td><td>0.823</td><td>0.811</td><td>0.815</td><td>0.848</td><td>0.810</td><td>0.816</td><td>0.817</td><td>0.802</td><td>0.809</td><td>0.798</td><td>0.833</td><td>0.819</td><td>0.821</td><td>0.824</td><td>0.795</td><td>0.810</td><td>0.825</td></tr><tr><td>ONLINE-A</td><td>0.606</td><td>0.663</td><td>0.696</td><td>0.591</td><td>0.599</td><td>0.665</td><td>0.655</td><td>0.721</td><td>0.649</td><td>0.715</td><td>0.567</td><td>0.712</td><td>0.702</td><td>0.698</td><td>0.619</td><td>0.624</td><td>0.676</td><td>0.724</td></tr><tr><td>ONLINE-B</td><td>0.664</td><td>0.726</td><td>0.733</td><td>0.665</td><td>0.667</td><td>0.717</td><td>0.719</td><td>0.761</td><td>0.695</td><td>0.751</td><td>0.629</td><td>0.756</td><td>0.746</td><td>0.750</td><td>0.698</td><td>0.697</td><td>0.728</td><td>0.760</td></tr><tr><td>ONLINE-G</td><td>0.591</td><td>0.644</td><td>0.682</td><td>0.572</td><td>0.581</td><td>0.658</td><td>0.636</td><td>0.657</td><td>0.625</td><td>0.680</td><td>0.529</td><td>0.693</td><td>0.677</td><td>0.653</td><td>0.588</td><td>0.608</td><td>0.653</td><td>0.699</td></tr><tr><td>ONLINE-M</td><td>0.601</td><td>0.661</td><td>0.689</td><td>0.616</td><td>0.586</td><td>0.659</td><td>0.676</td><td>0.693</td><td>0.633</td><td>0.720</td><td>0.574</td><td>0.700</td><td>0.693</td><td>0.695</td><td>0.636</td><td>0.627</td><td>0.671</td><td>0.720</td></tr><tr><td>ONLINE-W</td><td>0.682</td><td>0.730</td><td>0.754</td><td>0.700</td><td>0.708</td><td>0.748</td><td>0.704</td><td>0.771</td><td>0.702</td><td>0.772</td><td>0.663</td><td>0.763</td><td>0.766</td><td>0.754</td><td>0.717</td><td>0.726</td><td>0.727</td><td>0.782</td></tr><tr><td>ONLINE-Y</td><td>0.617</td><td>0.668</td><td>0.703</td><td>0.616</td><td>0.626</td><td>0.678</td><td>0.652</td><td>0.707</td><td>0.660</td><td>0.709</td><td>0.589</td><td>0.728</td><td>0.708</td><td>0.708</td><td>0.635</td><td>0.643</td><td>0.672</td><td>0.725</td></tr><tr><td>NLLB_MBR</td><td>0.711</td><td>0.786</td><td>0.778</td><td>0.757</td><td>0.771</td><td>0.787</td><td>0.745</td><td>0.785</td><td>0.754</td><td>0.778</td><td>0.724</td><td>0.794</td><td>0.786</td><td>0.773</td><td>0.757</td><td>0.738</td><td>0.774</td><td>0.792</td></tr><tr><td>NLLB_Greedy</td><td>0.718</td><td>0.770</td><td>0.775</td><td>0.750</td><td>0.781</td><td>0.759</td><td>0.750</td><td>0.789</td><td>0.745</td><td>0.773</td><td>0.721</td><td>0.793</td><td>0.785</td><td>0.766</td><td>0.760</td><td>0.747</td><td>0.765</td><td>0.791</td></tr><tr><td>Lan-BridgeMT</td><td>0.618</td><td>0.670</td><td>0.701</td><td>0.612</td><td>0.594</td><td>0.674</td><td>0.650</td><td>0.693</td><td>0.651</td><td>0.709</td><td>0.573</td><td>0.730</td><td>0.703</td><td>0.693</td><td>0.632</td><td>0.636</td><td>0.674</td><td>0.723</td></tr><tr><td>GTCOM_Peter</td><td>0.597</td><td>0.675</td><td>0.704</td><td>0.628</td><td>0.619</td><td>0.673</td><td>0.671</td><td>0.689</td><td>0.635</td><td>0.704</td><td>0.577</td><td>0.714</td><td>0.703</td><td>0.691</td><td>0.640</td><td>0.632</td><td>0.666</td><td>0.725</td></tr><tr><td>ZengHuiMT</td><td>0.610</td><td>0.674</td><td>0.695</td><td>0.605</td><td>0.593</td><td>0.649</td><td>0.658</td><td>0.723</td><td>0.645</td><td>0.693</td><td>0.576</td><td>0.707</td><td>0.702</td><td>0.692</td><td>0.635</td><td>0.653</td><td>0.690</td><td>0.717</td></tr><tr><td>CUNI-Trans</td><td>0.605</td><td>0.655</td><td>0.683</td><td>0.591</td><td>0.612</td><td>0.663</td><td>0.635</td><td>0.670</td><td>0.644</td><td>0.719</td><td>0.563</td><td>0.699</td><td>0.693</td><td>0.677</td><td>0.622</td><td>0.632</td><td>0.679</td><td>0.719</td></tr><tr><td>CUNI-DocTrans</td><td>0.583</td><td>0.617</td><td>0.660</td><td>0.553</td><td>0.562</td><td>0.660</td><td>0.609</td><td>0.653</td><td>0.610</td><td>0.692</td><td>0.521</td><td>0.669</td><td>0.667</td><td>0.654</td><td>0.589</td><td>0.610</td><td>0.644</td><td>0.694</td></tr><tr><td>CUNI-GA</td><td>0.583</td><td>0.617</td><td>0.660</td><td>0.553</td><td>0.562</td><td>0.660</td><td>0.609</td><td>0.653</td><td>0.610</td><td>0.692</td><td>0.521</td><td>0.669</td><td>0.667</td><td>0.654</td><td>0.589</td><td>0.610</td><td>0.644</td><td>0.694</td></tr><tr><td colspan="20">en-de</td></tr><tr><td>GPT4-5shot</td><td>0.800</td><td>0.801</td><td>0.824</td><td>0.814</td><td>0.836</td><td>0.810</td><td>0.803</td><td>0.835</td><td>0.797</td><td>0.821</td><td>0.812</td><td>0.834</td><td>0.831</td><td>0.824</td><td>0.823</td><td>0.809</td><td>0.817</td><td>0.837</td></tr><tr><td>ONLINE-A</td><td>0.670</td><td>0.708</td><td>0.751</td><td>0.660</td><td>0.698</td><td>0.727</td><td>0.726</td><td>0.783</td><td>0.693</td><td>0.771</td><td>0.765</td><td>0.642</td><td>0.774</td><td>0.759</td><td>0.758</td><td>0.699</td><td>0.728</td><td>0.742</td></tr><tr><td>ONLINE-B</td><td>0.746</td><td>0.768</td><td>0.801</td><td>0.767</td><td>0.778</td><td>0.788</td><td>0.779</td><td>0.823</td><td>0.771</td><td>0.809</td><td>0.734</td><td>0.812</td><td>0.810</td><td>0.800</td><td>0.770</td><td>0.780</td><td>0.802</td><td>0.815</td></tr><tr><td>ONLINE-G</td><td>0.663</td><td>0.701</td><td>0.743</td><td>0.679</td><td>0.725</td><td>0.725</td><td>0.711</td><td>0.759</td><td>0.708</td><td>0.754</td><td>0.667</td><td>0.760</td><td>0.751</td><td>0.745</td><td>0.711</td><td>0.724</td><td>0.732</td><td>0.763</td></tr><tr><td>ONLINE-M</td><td>0.592</td><td>0.615</td><td>0.679</td><td>0.591</td><td>0.586</td><td>0.665</td><td>0.650</td><td>0.694</td><td>0.631</td><td>0.714</td><td>0.548</td><td>0.679</td><td>0.686</td><td>0.671</td><td>0.609</td><td>0.630</td><td>0.643</td><td>0.714</td></tr><tr><td>ONLINE-W</td><td>0.717</td><td>0.725</td><td>0.784</td><td>0.749</td><td>0.762</td><td>0.775</td><td>0.758</td><td>0.808</td><td>0.765</td><td>0.797</td><td>0.718</td><td>0.822</td><td>0.797</td><td>0.802</td><td>0.768</td><td>0.765</td><td>0.787</td><td>0.809</td></tr><tr><td>ONLINE-Y</td><td>0.710</td><td>0.746</td><td>0.777</td><td>0.745</td><td>0.762</td><td>0.760</td><td>0.753</td><td>0.797</td><td>0.746</td><td>0.789</td><td>0.703</td><td>0.794</td><td>0.784</td><td>0.778</td><td>0.767</td><td>0.746</td><td>0.755</td><td>0.795</td></tr><tr><td>NLLB_MBR</td><td>0.702</td><td>0.749</td><td>0.776</td><td>0.751</td><td>0.756</td><td>0.763</td><td>0.724</td><td>0.786</td><td>0.729</td><td>0.768</td><td>0.718</td><td>0.780</td><td>0.778</td><td>0.767</td><td>0.744</td><td>0.735</td><td>0.755</td><td>0.786</td></tr><tr><td>NLLB_Greedy</td><td>0.699</td><td>0.745</td><td>0.772</td><td>0.748</td><td>0.761</td><td>0.749</td><td>0.732</td><td>0.788</td><td>0.743</td><td>0.765</td><td>0.715</td><td>0.767</td><td>0.776</td><td>0.767</td><td>0.745</td><td>0.742</td><td>0.759</td><td>0.783</td></tr><tr><td>Lan-BridgeMT</td><td>0.624</td><td>0.652</td><td>0.722</td><td>0.627</td><td>0.629</td><td>0.701</td><td>0.678</td><td>0.726</td><td>0.656</td><td>0.725</td><td>0.602</td><td>0.716</td><td>0.720</td><td>0.705</td><td>0.637</td><td>0.673</td><td>0.678</td><td>0.737</td></tr><tr><td>ZengHuiMT</td><td>0.671</td><td>0.698</td><td>0.741</td><td>0.686</td><td>0.694</td><td>0.724</td><td>0.719</td><td>0.765</td><td>0.699</td><td>0.743</td><td>0.649</td><td>0.755</td><td>0.752</td><td>0.741</td><td>0.706</td><td>0.734</td><td>0.726</td><td>0.760</td></tr><tr><td>AIRC</td><td>0.556</td><td>0.588</td><td>0.646</td><td>0.533</td><td>0.557</td><td>0.625</td><td>0.597</td><td>0.666</td><td>0.595</td><td>0.646</td><td>0.522</td><td>0.631</td><td>0.647</td><td>0.636</td><td>0.557</td><td>0.595</td><td>0.609</td><td>0.669</td></tr><tr><td colspan="20">en-ru</td></tr><tr><td>GPT4-5shot</td><td>0.751</td><td>0.794</td><td>0.780</td><td>0.788</td><td>0.811</td><td>0.795</td><td>0.766</td><td>0.797</td><td>0.773</td><td>0.781</td><td>0.760</td><td>0.780</td><td>0.787</td><td>0.755</td><td>0.781</td><td>0.758</td><td>0.771</td><td>0.793</td></tr><tr><td>ONLINE-A</td><td>0.633</td><td>0.687</td><td>0.716</td><td>0.641</td><td>0.689</td><td>0.668</td><td>0.712</td><td>0.751</td><td>0.688</td><td>0.719</td><td>0.609</td><td>0.713</td><td>0.721</td><td>0.709</td><td>0.653</td><td>0.680</td><td>0.703</td><td>0.730</td></tr><tr><td>ONLINE-B</td><td>0.664</td><td>0.724</td><td>0.732</td><td>0.684</td><td>0.730</td><td>0.709</td><td>0.721</td><td>0.770</td><td>0.712</td><td>0.744</td><td>0.652</td><td>0.740</td><td>0.740</td><td>0.747</td><td>0.705</td><td>0.706</td><td>0.722</td><td>0.748</td></tr><tr><td>ONLINE-G</td><td>0.689</td><td>0.762</td><td>0.759</td><td>0.730</td><td>0.754</td><td>0.757</td><td>0.749</td><td>0.791</td><td>0.747</td><td>0.767</td><td>0.700</td><td>0.766</td><td>0.765</td><td>0.765</td><td>0.736</td><td>0.743</td><td>0.768</td><td>0.773</td></tr><tr><td>ONLINE-M</td><td>0.566</td><td>0.636</td><td>0.657</td><td>0.584</td><td>0.583</td><td>0.634</td><td>0.653</td><td>0.691</td><td>0.636</td><td>0.687</td><td>0.655</td><td>0.668</td><td>0.661</td><td>0.657</td><td>0.608</td><td>0.604</td><td>0.640</td><td>0.686</td></tr><tr><td>ONLINE-W</td><td>0.724</td><td>0.769</td><td>0.770</td><td>0.761</td><td>0.773</td><td>0.759</td><td>0.769</td><td>0.797</td><td>0.753</td><td>0.777</td><td>0.721</td><td>0.784</td><td>0.780</td><td>0.781</td><td>0.763</td><td>0.762</td><td>0.781</td><td>0.786</td></tr><tr><td>ONLINE-Y</td><td>0.668</td><td>0.738</td><td>0.742</td><td>0.708</td><td>0.723</td><td>0.714</td><td>0.730</td><td>0.781</td><td>0.727</td><td>0.741</td><td>0.668</td><td>0.747</td><td>0.747</td><td>0.751</td><td>0.724</td><td>0.706</td><td>0.730</td><td>0.756</td></tr><tr><td>NLLB_MBR</td><td>0.679</td><td>0.748</td><td>0.743</td><td>0.731</td><td>0.749</td><td>0.763</td><td>0.721</td><td>0.763</td><td>0.718</td><td>0.738</td><td>0.690</td><td>0.737</td><td>0.746</td><td>0.732</td><td>0.717</td><td>0.703</td><td>0.733</td><td>0.755</td></tr><tr><td>NLLB_Greedy</td><td>0.675</td><td>0.738</td><td>0.740</td><td>0.720</td><td>0.745</td><td>0.734</td><td>0.718</td><td>0.758</td><td>0.710</td><td>0.729</td><td>0.695</td><td>0.734</td><td>0.744</td><td>0.733</td><td>0.720</td><td>0.703</td><td>0.723</td><td>0.753</td></tr><tr><td>Lan-BridgeMT</td><td>0.604</td><td>0.653</td><td>0.679</td><td>0.614</td><td>0.635</td><td>0.659</td><td>0.672</td><td>0.714</td><td>0.636</td><td>0.694</td><td>0.571</td><td>0.699</td><td>0.687</td><td>0.683</td><td>0.623</td><td>0.635</td><td>0.668</td><td>0.699</td></tr><tr><td>PROMT</td><td>0.574</td><td>0.636</td><td>0.664</td><td>0.590</td><td>0.645</td><td>0.646</td><td>0.635</td><td>0.686</td><td>0.637</td><td>0.677</td><td>0.553</td><td>0.678</td><td>0.667</td><td>0.656</td><td>0.601</td><td>0.620</td><td>0.644</td><td>0.685</td></tr><tr><td>ZengHuiMT</td><td>0.623</td><td>0.694</td><td>0.691</td><td>0.615</td><td>0.669</td><td>0.647</td><td>0.687</td><td>0.730</td><td>0.663</td><td>0.683</td><td>0.600</td><td>0.700</td><td>0.696</td><td>0.690</td><td>0.658</td><td>0.657</td><td>0.688</td><td>0.706</td></tr><tr><td colspan="20">en-uk</td></tr><tr><td>GPT4-5shot</td><td>0.804</td><td>0.838</td><td>0.828</td><td>0.834</td><td>0.834</td><td>0.818</td><td>0.817</td><td>0.836</td><td>0.804</td><td>0.831</td><td>0.809</td><td>0.834</td><td>0.832</td><td>0.835</td><td>0.826</td><td>0.811</td><td>0.819</td><td>0.838</td></tr><tr><td>ONLINE-A</td><td>0.626</td><td>0.688</td><td>0.725</td><td>0.635</td><td>0.665</td><td>0.679</td><td>0.705</td><td>0.744</td><td>0.680</td><td>0.736</td><td>0.608</td><td>0.728</td><td>0.721</td><td>0.721</td><td>0.656</td><td>0.676</td><td>0.721</td><td>0.741</td></tr><tr><td>ONLINE-B</td><td>0.701</td><td>0.744</td><td>0.770</td><td>0.723</td><td>0.748</td><td>0.742</td><td>0.766</td><td>0.790</td><td>0.750</td><td>0.767</td><td>0.788</td><td>0.679</td><td>0.774</td><td>0.779</td><td>0.788</td><td>0.734</td><td>0.744</td><td>0.764</td></tr><tr><td>ONLINE-G</td><td>0.718</td><td>0.788</td><td>0.791</td><td>0.771</td><td>0.773</td><td>0.779</td><td>0.773</td><td>0.807</td><td>0.767</td><td>0.799</td><td>0.736</td><td>0.736</td><td>0.795</td><td>0.795</td><td>0.766</td><td>0.775</td><td>0.808</td><td>0.803</td></tr><tr><td>ONLINE-W</td><td>0.763</td><td>0.783</td><td>0.802</td><td>0.805</td><td>0.805</td><td>0.789</td><td>0.807</td><td>0.831</td><td>0.808</td><td>0.818</td><td>0.757</td><td>0.805</td><td>0.814</td><td>0.817</td><td>0.787</td><td>0.793</td><td>0.812</td><td>0.819</td></tr><tr><td>ONLINE-Y</td><td>0.647</td><td>0.712</td><td>0.737</td><td>0.703</td><td>0.706</td><td>0.706</td><td>0.706</td><td>0.754</td><td>0.685</td><td>0.742</td><td>0.656</td><td>0.749</td><td>0.738</td><td>0.734</td><td>0.710</td><td>0.691</td><td>0.713</td><td>0.750</td></tr><tr><td>NLLB_MBR</td><td>0.698</td><td>0.760</td><td>0.763</td><td>0.768</td><td>0.760</td><td>0.765</td><td>0.730</td><td>0.777</td><td>0.743</td><td>0.730</td><td>0.762</td><td>0.715</td><td>0.759</td><td>0.768</td><td>0.765</td><td>0.742</td><td>0.731</td><td>0.756</td></tr><tr><td>NLLB_Greedy</td><td>0.692</td><td>0.740</td><td>0.761</td><td>0.741</td><td>0.758</td><td>0.724</td><td>0.732</td><td>0.767</td><td>0.730</td><td>0.765</td><td>0.703</td><td>0.768</td><td>0.766</td><td>0.762</td><td>0.742</td><td>0.742</td><td>0.735</td><td>0.775</td></tr><tr><td>Lan-BridgeMT</td><td>0.591</td><td>0.655</td><td>0.685</td><td>0.606</td><td>0.603</td><td>0.657</td><td>0.660</td><td>0.672</td><td>0.632</td><td>0.679</td><td>0.572</td><td>0.689</td><td>0.684</td><td>0.675</td><td>0.620</td><td>0.628</td><td>0.660</td><td>0.705</td></tr><tr><td>GTCOM_Peter</td><td>0.591</td><td>0.670</td><td>0.702</td><td>0.613</td><td>0.645</td><td>0.671</td><td>0.677</td><td>0.706</td><td>0.671</td><td>0.707</td><td>0.580</td><td>0.693</td><td>0.695</td><td>0.685</td><td>0.637</td><td>0.636</td><td>0.658</td><td>0.714</td></tr><tr><td>ZengHuiMT</td><td>0.608</td><td>0.684</td><td>0.696</td><td>0.620</td><td>0.639</td><td>0.650</td><td>0.672</td><td>0.712</td><td>0.660</td><td>0.698</td><td>0.600</td><td>0.701</td><td>0.697</td><td>0.696</td><td>0.652</td><td>0.662</td><td>0.687</td><td>0.709</td></tr></tbody></table>

non-standard English texts, there is no systematic notion of generating non-standard translations; it is possible that an expressive version of *viel* was seen far more often in the training data, thus being a probable translation in this case.

**Number of copied words**

Bearing in mind the fact that unknown source words are often either copied or removed, we apply a simple automatic analysis to judge how often systems copy source words to diagnose a potential problem with out-of-vocabulary words. We calculate how often a sentence's translation contains words that appear in both the source and system's translations (counting occurrences for each sentences and taking the average). We calculate the difference in the average number of copied words for themanseg-raw and manseg-norm subsets (i.e. how much more are

words copied when systems are translating non-standard data). Results are shown in Table 9 of Appendix F. Note that a certain number of 'copied' words is normal, as certain words are expected to be identical in source and target texts (e.g. some named entities, numbers, etc.). There are similar trends across languages: the same systems tend to have relatively fewer/more copied words in raw than norm. GPT4 has the smallest difference in number of copied words between norm and raw translations. Unsurprisingly, most other systems have more copying in raw than in norm, indicating a problem handling unknown vocabulary and confirming our previous qualitative analysis.

On a similar note, we also analysed the source-translation length ratios for both raw and norm subsets. While this could be linked to copying or omit-
<table><caption>Table 6: Example of character repetition linked to a mark of expressivity for en-de.</caption><thead><tr><th></th><th>norm/raw</th><th>Text</th></tr></thead><tbody><tr><td>Source</td><td>norm</td><td>"politics used to be more polite. We need a return to the discourse"</td></tr><tr><td>Source</td><td>raw</td><td>"politics used to be mooooorrreeee poolllliitte we neeeeed a return to The Discouuurrрр"</td></tr><tr><td>Reference (de)</td><td>norm</td><td>„Politik früher höflicher war. Wir brauchen eine Rückkehr zum Diskurs.“</td></tr><tr><td>AIRC</td><td>norm<br>raw</td><td>"Politik war früher höflicher. Wir brauchen eine Rückkehr zum Diskurs"<br>"Politiker waren früher mooooorrreeee poolllliitte wir neeeeed eine Rückkehr zu The Discouuurrрр"</td></tr><tr><td>GPT4-5shot</td><td>norm<br>raw</td><td>„die Politik früher höflicher war. Wir brauchen eine Rückkehr zur Diskussion"<br>„die Politik früher viiieeel höflicher war, wir brauchen eine Rückkehr zur Diskussion"</td></tr><tr><td>Lan-BridgeMT</td><td>norm<br>raw</td><td>„die Politik früher höflicher war. Wir brauchen eine Rückkehr zum Diskurs"<br>„die Politik früher mooooorrreeee poolllliitte war, dass wir eine Rückkehr zu The Discouuurrрр brauchten"</td></tr><tr><td>NLLB_MBR</td><td>norm<br>raw</td><td>Politik früher höflicher war<br>"Politik war früher mooooorrreeeeeee poolllliitte, wir brauchen eine Rückkehr zu The Discouuurrрр"</td></tr><tr><td>ONLINE-A</td><td>norm<br>raw</td><td>„Politik früher höflicher war. Wir brauchen eine Rückkehr zum Diskurs"<br>„Politik früher mooooorrreeee poolllliitte wir brauchten eine Rückkehr zu The Discouuurrрр"</td></tr><tr><td>ONLINE-B</td><td>norm<br>raw</td><td>„die Politik früher höflicher war.“ Wir brauchen eine Rückkehr zum Diskurs"<br>„Politik früher mal mooooorrreeee poolllliitte war, wir brauchen eine Rückkehr zu The Discouuurrрр"</td></tr><tr><td>ONLINE-G</td><td>norm<br>raw</td><td>„Politik früher höflicher war. Wir brauchen eine Rückkehr zum Diskurs"<br>„Politik früher mooooorrreeee poolllliitte war, wir brauchen eine Rückkehr zur Discouuurrрр"</td></tr><tr><td>ONLINE-M</td><td>norm<br>raw</td><td>„Politik war früher mooooorrreeee poolllliitte wir neeeeed a return to The Discouuurrрр"<br>„die Politik früher höflicher war. Wir brauchen eine Rückkehr zum Diskurs"</td></tr><tr><td>ONLINE-W</td><td>norm<br>raw</td><td>„Politik früher mooooorrreeee poolllliitte we neeeeed a return to The Discouuurrрр"<br>„Politik früher höflicher war. Wir brauchen eine Rückkehr zum Diskurs"</td></tr><tr><td>ONLINE-Y</td><td>norm<br>raw</td><td>„Politik früher mooooorrreeee poolllliitte war, wir brauchen eine Rückkehr zu The Discouuurrрр"<br>„Politik früher höflicher war“, war früher höflicher. Wir brauchen eine Rückkehr zum Diskurs"</td></tr><tr><td>ZengHuiMT</td><td>norm<br>raw</td><td>„Politik früher mooooorrreeee poolllliitte war, wir brauchten eine Rückkehr zu The Discouuurrрр"</td></tr></tbody></table>

ting unknown words (as we have seen), it is more likely to be linked to overgeneration problems, linked to systems encountering text that is out-of-domain, which we occasionally observed in the system outputs. We observed that for all systems, the length ratio between manseg-raw translations and their source sentences was greater than those for manseg-norm. The effect was even greater for the texts when automatic sentence segmentation was applied (i.e. for spacyseg- subsets).

## 6 Conclusion

We have presented a new resource, RoCS-MT, a robustness challenge set for MT, designed to test MT systems on non-standard UGC. Our automatic and manual analysis show that non-standard texts are still a problem for many of the systems, including the unconstrained ones, and that certain phenomena such as phonetically inspired spellings pose a problem in particular. The comparison of COMET and COMET-QE metrics suggest that it may be possible to draw similar conclusions from automatic scoring without using references, although future work could go into more depth into analysing what is captured by the different metrics.

### Limitations

The current test set is available for five from-English directions and it would be interesting to study other language directions, including those

not involving English. The current version of the challenge set only contains variants for speaker gender for one of the language pairs, and we plan to add these for the other target languages in a future version.

Finally, a major limitation is one that is becoming widespread nowadays, which is that many of the systems trained and even used in research are trained on an unknown quantity of data for which the sources are unknown. Without being able to verify the fact, GPT4 and potentially some of the other systems are likely to be trained on some of the source sentences in the challenge set, and future models may even be trained on the reference translations we provide, despite it being indicated as a test set. This is a blocking factor for scientific comparison and one that goes beyond this particular resource.

### Acknowledgements

This paper was funded by both authors chair positions in the PRAIRIE institute funded by the French national agency ANR as part of the “Investissements d’avenir” programme under the reference ANR-19-P3IA-0001. It was also funded by Rachel Bawden’s Emergence project, DadaNMT, funded by Sorbonne Université.
References

Jan Christian Blaise Cruz. 2023. Samsung R&D Institute Philippines at WMT 2023. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Hiroyuki Deguchi, Kenji Imamura, Yuto Nishida, Yusuke Sakai, Justin Vasselli, and Taro Watanabe. 2023. NAIST-NICT WMT'23 General MT Task Submission. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. *Transactions of the Association for Computational Linguistics*, 10:811–825.

Ryo Fujii, Masato Mita, Kaori Abe, Kazuaki Hanawa, Makoto Morishita, Jun Suzuki, and Kentaro Inui. 2020. PheMT: A phenomenon-wise dataset for machine translation robustness on user-generated contents. In *Proceedings of the 28th International Conference on Computational Linguistics*, pages 5929–5943, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Ananya Gupta, Jae Takeuchi, and Bart Knijnenburg. 2023. On the real-world performance of machine translation: Exploring social media post-authors' perspectives. In *Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)*, pages 302–310, Toronto, Canada. Association for Computational Linguistics.

Bo Han and Timothy Baldwin. 2011. Lexical normalization of short text messages: Makn sens a #twitter. In *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*, pages 368–378, Portland, Oregon, USA. Association for Computational Linguistics.

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hasan Awadalla. 2023. How good are GPT models at machine translation? A comprehensive evaluation. *arXiv preprint arXiv:2302.09210*.

Josef Jon, Martin Popel, and Ondřej Bojar. 2023. CUNI-GA submission at WMT23. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022

conference on machine translation (WMT22). In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Keito Kudo, Takumi Ito, Makoto Morishita, and Jun Suzuki. 2023. SKIM at WMT 2023 General Translation Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Ben Li, Yoko Matsuzaki, and Shivam Kalkar. 2023. KYB General Machine Translation Systems for WMT23. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In *Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 176–186, Sofia, Bulgaria. Association for Computational Linguistics.

Paul McNamee and Kevin Duh. 2022. The multilingual microblog translation corpus: Improving and evaluating translation of user-generated text. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 910–918, Marseille, France. European Language Resources Association.

Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 543–553, Brussels, Belgium. Association for Computational Linguistics.

Luo Min, yixin tan, and Qiulin Chen. 2023. Yishu: Yishu At WMT2023 Translation Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Alexander Molchanov and Vladislav Kovalenko. 2023. PROMT Systems for WMT23 Shared General Translation Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2020. Constructing a bilingual corpus of parallel tweets. In *Proceedings of the 13th Workshop on Building and Using Comparable Corpora*, pages 14–21, Marseille, France. European Language Resources Association.

Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics*, pages 311–318, Philadelphia,
Pennsylvania, USA. Association for Computational Linguistics.

Martin Popel. 2020. CUNI English-Czech and English-Polish systems in WMT20: Robust document-level training. In *Proceedings of the Fifth Conference on Machine Translation*, pages 269–273, Online. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2685–2702, Online. Association for Computational Linguistics.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Matíss Rikters and Makoto Miwa. 2023. AIST AIRC Submissions to the WMT23 Shared Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

José Carlos Rosales Núñez, Djamé Seddah, and Guil-laume Wisniewski. 2019. Comparison between NMT and PBSMT performance for translating noisy user-generated content. In *Proceedings of the 22nd Nordic Conference on Computational Linguistics*, pages 2–14, Turku, Finland. Linköping University Electronic Press.

José Carlos Rosales Núñez, Djamé Seddah, and Guil-laume Wisniewski. 2021. Understanding the impact of UGC specificities on translation quality. In *Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)*, pages 189–198, Online. Association for Computational Linguistics.

Manuela Sanguinetti, Cristina Bosco, Lauren Cas-sidy, Özlem Çetinoğlu, Alessandra Teresa Cignarella, Teresa Lynn, Ines Rehbein, Josef Ruppenhofer, Djamé Seddah, and Amir Zeldes. 2020. Treebank-ing user-generated content: A proposal for a uni-fied representation in Universal Dependencies. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 5240–5250, Marseille, France. European Language Resources Association.

Djamé Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. The French Social Media Bank: a treebank of noisy user generated content. In *Proceedings of COLING 2012*, pages 2441–2458, Mumbai, India. The COLING 2012 Or-ganizing Committee.

Henny Sluyter-Gäthje, Pintu Lohar, Haithem Afli, and Andy Way. 2018. FooTweets: A bilingual paral-lel corpus of world cup tweets. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan. European Language Resources Association (ELRA).

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef-fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar-rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.

Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Bar-bara Plank, Benjamin Muller, Iñaki San Vicente Ron-cal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Ma-hendra, Talha Çolakoğlu, Timothy Baldwin, Tom-maso Caselli, and Wladimir Sidorenko. 2021. Mul-tiLexNorm: A shared task on multilingual lexical normalization. In *Proceedings of the Seventh Work-shop on Noisy User-generated Text (W-NUT 2021)*, pages 493–509, Online. Association for Computa-tional Linguistics.

Iñaki San Vicente, Iñaki Alegría, Cristina España-Bonet, Pablo Gamallo, Hugo Gonçalo Oliveira, Eva Martínez Garcia, Antonio Toral, Arkaitz Zubiaga, and Nora Aranberri. 2016. TweetMT: A parallel microblog corpus. In *Proceedings of the Tenth In-ternational Conference on Language Resources and Evaluation (LREC'16)*, pages 2936–2941, Portorož, Slovenia. European Language Resources Association (ELRA).

Di Wu, Shaomu Tan, David Stap, Ali Araabi, and Christof Monz. 2023a. UvA-MT's Participation in the WMT 2023 General Translation Shared Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Yangjian Wu and Gang Hu. 2023. Exploring Prompt En-gineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.
Zhanglin Wu, Daimeng Wei, Zongyao Li, Zhengzhe YU, Shaojun Li, Xiaoyu Chen, Hengchao Shang, Jiaxin GUO, Yuhao Xie, Lizhi Lei, Hao Yang, and Yanfei Jiang. 2023b. Treating General MT Shared Task as a Multi-Domain Adaptation Problem: HW-TSC's Submission to the WMT23 General MT Shared Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Hui Zeng. 2023. Achieving State-of-the-Art Multilingual Translation Model with Minimal Data and Parameters. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Wenbo Zhang. 2023. IOL Research Machine Translation Systems for WMT23 General Machine Translation Shared Task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Hao Zong. 2023. Gtcom neural machine translation systems for wmt23. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.
## A Non-standard keywords for sourcing of posts

The full list of keywords we searched for using the Reddit API is as follows: yyy, iii, eee, ppl, btw, imo, wtf, shes, hes, ima, shud, wud, cud, afaik, bcuz, hahaaa, dat, wen, wot, woz, bout, bro, gonna, lmao, ppl, smh, yall, omg, barley, fyi, beleive, seperate, lol, ttyl, muaha, mwah, air, afaik, fr, fyi, idk, ikr, irl, jk, nvm, plz, pls, cu, tbh, ur, wth, kk, 2mo, 2moro, tmrw, fwiw, nvm, thx, b4, ruok, m8, l8r, 2nite, gr8, lk, wt, w/, peeps, sooo, verry, innit, wasnt, ain't, definately, yous, nae, awfull, freind, untill, wierd, aweful, wether and alot.

The keywords were chosen as they illustrate well known non-standard phenomena, including:

* spelling errors (e.g. wierd 'weird', wether 'whether', alot)

* acronymisation (nvm 'never mind', fyi 'for your information)

* repetition of characters (e.g. hahaaa, eee, sooo)

* contractions (e.g. cud, gonna, shud))

* dialectisms (e.g. ain't, yous, nae, innit, yall)

* devowelling (e.g. tmrw 'tomorrow', pls 'please', jk 'joke')

* truncations, including abbreviations (e.g. peeps 'people', w/ "with")

* digit phonetisation (e.g. 2nite 'tonight', b4 'before', l8r 'later', cu 'see you', ruok 'are you ok')

* other phonetic spellings (e.g. wot 'what', thx 'thanks', dat 'that')

* missing whitespace (e.g. cu 'see you', ruok 'are you ok', both examples also corresponding to phonetic spellings)

* missing punctuation (e.g. ur (sometimes) 'you're', wasnt 'wasn't')

* etc.

Although the choice of keywords does create a certain bias in the types of language retrieved (especially given that several variants of some keywords are included), these keywords are used to identify posts that likely to contain other non-standard phenomena, so the final selected sentences are not restricted to those containing these keywords.

## B Translation Guidelines

These guidelines are included because there are some specific constraints as to how the translations are to be carried out, and some particularities of the dataset to explain. The sentences to be translated are found in the excel spreadsheet in the column "Normalised segment". However, we also provide additional information that can help translation (see below for more information).

**Origin of the text** The texts to be translated are from the Reddit online forum (extracted using the API), taken from a range of different subreddits (so of different genres of text, e.g. relationship advice, advice about pets, video gaming strategy, etc.). They were selected due to their non-standard nature (spelling mistakes, abbreviations, lack of punctuation etc.).

**Preprocessing of the text** The texts have been manually pseudo-anonymised (usernames and names other than those representing celebrities and other well-known public figures are replaced with new names), split into "sentences" and normalised. It is the normalised versions of the sentences that are to be translated.

The sentences have been filtered to remove offensive or sensitive content (hate speech, taking drugs, suicide, etc.). However, profanities were kept as they were taken to be illustrative of the sociolect of online language. If however, you do not feel comfortable with translating something, please leave it blank and write a comment indicating that you have not translated it.

**Additional context provided to help translation**

The text is split into short documents with one or several sentences per document. In the excel document, a sentence's document is indicated by the value in the column "Post number", and the cells are also coloured such that it is visually easier to see which sentences belong to the same document (alternating grey and white). A Reddit post is associated with a title and a text with the main content of the post. The documents can contain either the title or a subset of the text or even both. The type of text associated with each sentence is indicated in the column "Text type". Titles are marked in bold to make them visually easier to see. Although the normalised text may be sufficient to carry out the translation, we also give access to the additional information just in case:
* the title of the post

* the entire body of text associated with the post

* the raw version of the sentence (after pseudonymisation and segmentation into sentences)

* some translation notes have been added to provide some context about the posts (e.g. to give an idea of what is the subject of conversation, the meaning of some expressions and abbreviations, etc. in order to make translation easier). Very occasionally there are indications about how to translate (for instance for meta-linguistic questions where people discuss particular words, it is best to keep the English words, e.g. *One word I simply can't say properly is water...* → water should be kept in English in the translation.

**Constraints (important)** The dataset will be used to evaluate machine translation systems on their ability to handle non-standard texts. This crucially means that: the sentence boundaries that have been defined must not be modified. It is possible to translate a sentence using several sentences if that is what is natural. However, it is not possible to merge several source sentences to produce a single translation of both (i.e. one translation per row). translators should not use machine translation systems or other computational systems to aid translation as this could bias the translations to look like translations produced by Google Translate, DeepL, ChatGPT, etc.

**More specific guidelines** There are multiple posts that use slang terms (e.g. gaming or general online slang such as *lol*) and it possible that the correct translation will be an English borrowing. It is fine to use an English borrowing in this case, if this is what is generally used online. The punctuation choices should be kept as much as possible, as appropriate for the target language of translation (e.g. conserving full stops, exclamation marks, quotes, etc.). As described above, there are some instances of people talking about English words, and in this case, the English words should be kept as is. Another example: *One says “Let’s eat granny” making it seem like someone’s going to eat their nan. However, the other example says “Lets eat, granny”, implying a different meaning to the sentence.* The phrases “Let’s eat granny” and “Let’s

eat, granny” should be kept in English. These are indicated in the translation notes.

Use of “non-standard” language:

* Any spelling mistakes that were in the raw sentence should not be reproduced in the translation (i.e. the normalised version should be used as the source sentence to translate).

* Formatting, including things like capitalisation, should (for the same reasons) follow the conventions of the normalised translation.

* Abbreviations, acronyms and simplifications (e.g. in English *wdym = what do you mean*, *bc = because*, *rly = really*, etc.) should be expanded, unless the result would not make a natural sentence that could realistically be found. An example of a non-natural expansion would be *lol = laughing out loud*, since this is not practically used.

* However, abbreviations linked to the names of places (e.g. USA, UK, UCL (=University College London) should be kept as they are if the acronym is also commonly used in the target language. In other cases, the most frequent equivalent translation should be used. (e.g. English *UN* = French *ONU*, English *NATO* = French *OTAN*).

The overall idea is that the translations should be natural and not contain the types of non-standard language that were normalised in the English versions, although they should match as best possible the style and familiarity.

**Additional questions** If you have any doubts or questions about the meaning of the sentences, please contact me at rachel.bawden@inria.fr to discuss things further.

## C Prompt used for GPT4-5-shot

The prompt used for the GPT4-shot is the one from (Hendy et al., 2023), i.e. the following:

Translate this into 1. [target language]:
[shot n source]

1. [shot n reference]

Translate this into 1. [target language]:
 input

1.
## D BLEU scores

We provide BLEU scores for language pairs with reference translations in Table 7. The results are provided (as with the COMET scores in the main part of the paper) for the original raw subset (`manseg-raw`) and for its normalised version (`(manseg-norm)`)) as well as the difference between the two scores ($\delta$).

## E COMET-QE scores by annotation type

We provide in Table 8 COMET-QE scores per annotation type for all from-English language pairs of the shared task.

## F Copying analysis

Table 9 shows results for our automatic analysis of the number of source words that are found in the output translations. We calculate the number of such words, averaged over the number of sentences for each of the subsets `manseg-raw` and `manseg-norm` and we calculate the difference between the two. Positive numbers indicate that more copied words are found when systems translate the non-standard output and negative numbers indicate that more copied words are found when systems translated the normalised sentences.
<table><caption>Table 7: BLEU scores of systems on the manseg-norm and manseg-raw subsets.</caption><thead><tr><th rowspan="2">Systems</th><th colspan="3">en-cs</th><th colspan="3">en-de</th><th colspan="3">en-ru</th><th colspan="3">en-uk</th></tr><tr><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th><th>norm</th><th>raw</th><th>Δ</th></tr></thead><tbody><tr><td colspan="14" style="text-align: left; font-weight: bold;">Unconstrained</td></tr><tr><td>GPT4-5shot</td><td>25.9</td><td>22.5</td><td>3.4</td><td>46.6</td><td>40.8</td><td>5.8</td><td>23.4</td><td>19.6</td><td>3.9</td><td>27.8</td><td>25.4</td><td>2.4</td></tr><tr><td>ONLINE-A</td><td>27.1</td><td>19.3</td><td>7.8</td><td>49.0</td><td>38.5</td><td>10.5</td><td>26.1</td><td>20.4</td><td>5.8</td><td>31.3</td><td>25.0</td><td>6.3</td></tr><tr><td>ONLINE-B</td><td>28.4</td><td>20.9</td><td>7.5</td><td>47.7</td><td>40.7</td><td>7.1</td><td>25.7</td><td>20.6</td><td>5.1</td><td>39.0</td><td>29.6</td><td>9.4</td></tr><tr><td>ONLINE-G</td><td>25.0</td><td>17.4</td><td>7.6</td><td>46.2</td><td>35.5</td><td>10.7</td><td>27.9</td><td>22.5</td><td>5.4</td><td>29.2</td><td>25.1</td><td>4.0</td></tr><tr><td>ONLINE-M</td><td>27.8</td><td>19.1</td><td>8.7</td><td>44.5</td><td>29.5</td><td>15.0</td><td>23.6</td><td>16.5</td><td>7.1</td><td>-</td><td>-</td><td></td></tr><tr><td>ONLINE-W</td><td>30.0</td><td>22.9</td><td>7.2</td><td>66.0</td><td>47.1</td><td>18.9</td><td>29.3</td><td>23.7</td><td>5.6</td><td>31.5</td><td>27.4</td><td>4.0</td></tr><tr><td>ONLINE-Y</td><td>25.6</td><td>18.8</td><td>6.7</td><td>48.3</td><td>39.3</td><td>9.0</td><td>24.7</td><td>20.1</td><td>4.5</td><td>30.8</td><td>25.1</td><td>5.7</td></tr><tr><td>NLLB_MBR</td><td>25.5</td><td>20.8</td><td>4.7</td><td>41.5</td><td>34.1</td><td>7.4</td><td>22.3</td><td>18.2</td><td>4.2</td><td>26.9</td><td>22.3</td><td>4.6</td></tr><tr><td>NLLB_Greedy</td><td>25.4</td><td>20.8</td><td>4.6</td><td>42.0</td><td>34.0</td><td>8.0</td><td>22.1</td><td>18.4</td><td>3.7</td><td>26.2</td><td>22.0</td><td>4.2</td></tr><tr><td>Lan-BridgeMT</td><td>26.1</td><td>18.7</td><td>7.4</td><td>41.3</td><td>31.2</td><td>10.1</td><td>22.8</td><td>17.4</td><td>5.4</td><td>25.8</td><td>19.9</td><td>5.9</td></tr><tr><td>GTCOM_Peter</td><td>25.3</td><td>19.2</td><td>6.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>26.7</td><td>21.4</td><td>5.3</td></tr><tr><td>PROMT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>22.6</td><td>16.4</td><td>6.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td></td></tr><tr><td>ZengHuiMT</td><td>26.1</td><td>20.1</td><td>6.0</td><td>46.7</td><td>39.2</td><td>7.5</td><td>23.5</td><td>19.6</td><td>3.8</td><td>27.9</td><td>23.3</td><td>4.6</td></tr><tr><td colspan="14" style="text-align: left; font-weight: bold;">Constrained</td></tr><tr><td>AIRC</td><td>-</td><td>-</td><td>-</td><td>35.1</td><td>24.4</td><td>10.6</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td></td></tr><tr><td>CUNI-Trans</td><td>27.7</td><td>19.6</td><td>8.1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td></td></tr><tr><td>CUNI-DocTrans</td><td>28.9</td><td>18.0</td><td>10.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td></td></tr><tr><td>CUNI-GA</td><td>28.9</td><td>18.0</td><td>10.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td></td></tr></tbody></table>
<table><caption>Table 9: The difference in the number of source words present in the MT output between the manseg-raw and manseg-norm subsets, averaged across all sentences for each system. This indicates how much more (or less) source words are copied in the raw (unnormalised) sentences with respect to their normalised versions.</caption><thead><tr><th>Lang. pair</th><th>en-cs</th><th>en-de</th><th>en-he</th><th>en-ja</th><th>en-ru</th><th>en-uk</th><th>en-zh</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>0.14</td><td>-0.04</td><td>-0.04</td><td>-0.06</td><td>-0.07</td><td>-0.06</td><td>-0.04</td></tr><tr><td>NLLB_Greedy</td><td>0.08</td><td>-0.09</td><td>0.03</td><td>-0.01</td><td>-0.03</td><td>-0.01</td><td>-0.01</td></tr><tr><td>NLLB_MBR</td><td>0.06</td><td>-0.04</td><td>0.03</td><td>-0.00</td><td>-0.02</td><td>-0.01</td><td>-0.02</td></tr><tr><td>ONLINE-W</td><td>0.62</td><td>0.31</td><td>-</td><td>0.04</td><td>0.12</td><td>0.04</td><td>0.06</td></tr><tr><td>ONLINE-B</td><td>0.51</td><td>0.07</td><td>0.20</td><td>0.06</td><td>0.15</td><td>0.23</td><td>0.17</td></tr><tr><td>ONLINE-Y</td><td>0.54</td><td>0.10</td><td>0.27</td><td>0.01</td><td>0.07</td><td>0.11</td><td>0.25</td></tr><tr><td>Yishu</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.17</td></tr><tr><td>Lan-BridgeMT</td><td>0.69</td><td>0.46</td><td>0.02</td><td>0.44</td><td>0.28</td><td>0.42</td><td>-0.04</td></tr><tr><td>Samsung_Research.Philippines</td><td>-</td><td>-</td><td>0.32</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>HW-TSC</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.18</td></tr><tr><td>ONLINE-G</td><td>0.77</td><td>0.26</td><td>0.01</td><td>0.74</td><td>0.11</td><td>0.11</td><td>0.54</td></tr><tr><td>GTCOM_Peter</td><td>0.57</td><td>-</td><td>0.63</td><td>-</td><td>-</td><td>0.34</td><td>-</td></tr><tr><td>ONLINE-A</td><td>0.68</td><td>0.34</td><td>0.40</td><td>0.25</td><td>0.23</td><td>0.35</td><td>0.26</td></tr><tr><td>UvA-LTL</td><td>-</td><td>-</td><td>0.38</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SKIM</td><td>-</td><td>-</td><td>-</td><td>0.33</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ZengHuiMT</td><td>0.72</td><td>0.32</td><td>0.39</td><td>0.27</td><td>0.33</td><td>0.42</td><td>0.26</td></tr><tr><td>ONLINE-M</td><td>0.64</td><td>0.74</td><td>-</td><td>0.49</td><td>0.36</td><td>-</td><td>0.46</td></tr><tr><td>AIRC</td><td>-</td><td>0.71</td><td>-</td><td>0.56</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PROMT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.46</td><td>-</td><td>-</td></tr><tr><td>CUNI-Trans</td><td>0.88</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NAIST-NICT</td><td>-</td><td>-</td><td>-</td><td>0.61</td><td>-</td><td>-</td><td>-</td></tr><tr><td>IOL_Research</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.54</td></tr><tr><td>CUNI-DocTrans</td><td>1.53</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ANVITA</td><td>-</td><td>-</td><td>-</td><td>0.62</td><td>-</td><td>-</td><td>1.90</td></tr><tr><td>CUNI-GA</td><td>1.53</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>KYB</td><td>-</td><td>-</td><td>-</td><td>1.15</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>