Venue: EMNLP 2020

# Automatic Machine Translation Evaluation in Many Languages

via Zero-Shot Paraphrasing

Brian Thompson

Johns Hopkins University

brian.thompson@jhu.edu &Matt Post

Johns Hopkins University

post@cs.jhu.edu

###### Abstract

We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser's output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages (excluding Gujarati where the model had no training data). We also explore using our model for the task of quality estimation as a metric--conditioning on the source instead of the reference--and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair.

## 1 Introduction

Machine Translation (MT) systems have improved dramatically in the past several years. This is largely due to advances in neural MT (NMT) methods, but the pace of improvement would not have been possible without automatic MT metrics, which provide immediate feedback on MT quality without the time and expense associated with obtaining human judgments of MT output.

However, the improvements that existing automatic metrics helped enable are now causing the correlation between human judgments and automatic metrics to break down Ma et al. (2019); Mathur et al. (2020) especially for BLEU Papineni et al. (2002), which has been the de facto standard metric since its introduction almost two decades ago. The problem currently appears limited to very strong systems, but as hardware, modeling, and available training data improve, it is likely BLEU will fail more frequently in the future. This could prove extremely detrimental if the MT community fails to adopt an improved metric, as good ideas could quietly be discarded or rejected from publication because they do not correlate with BLEU. In fact, this may already be happening.

We propose using a sentential, sequence-to-sequence paraphraser to force-decode and score MT outputs conditioned on their corresponding human references. Our model implicitly represents the entire (exponentially large) set of potential paraphrases of a sentence, both valid and invalid; by "querying" the model wit

Figure 1: Our model is trained on multilingual parallel examples such as “Ciao amico” translated to French is “Salut l’ami.” At evaluation time, the model is used in zero-shot mode to score MT system outputs conditioned on their corresponding human references. For example, the MT system output “Hi world” conditioned on the human reference “Hello world” is found to have token probabilities [0.3, 0.6].



put, we can use the model score to measure how well the system output paraphrases the human reference translation. Our model is not trained on any human quality judgements, which are not available in many domains and/or language pairs.

The best possible MT output is one which perfectly matches a human reference; therefore, for evaluation, an ideal paraphraser would be one with an output distribution centered around a copy of its input sentence. We denote such a model a "lexically/syntactically unbiased paraphraser" to distinguish it from a standard paraphraser trained to produce output which conveys the meaning of the input while also being _lexically and/or syntactically different from it_. For this reason, we propose using a multilingual NMT system as an unbiased paraphraser by treating paraphrasing as zero-shot "translation" (e.g., Czech to Czech). We show that a multilingual NMT model is much closer to an ideal lexically/syntactically unbiased paraphraser than a generative paraphraser trained on synthetic paraphrases. It also allows a single model to work in many languages, and can be applied to the task of "Quality estimation (QE) as a metric" Fonseca et al. (2019) by conditioning on the source instead of the reference. Figure 1 illustrates our method, which we denote _Prism_ (Probability is the metric).

We train a single model in 39 languages and show that it:

* Outperforms or ties with prior metrics and several contrastive neural methods on the segment-level WMT 2019 MT metrics task in every language pair;1 Footnote 1: Except for Gujarati, where we had no training data.
* Is able to discriminate between very strong neural systems at the system level, addressing a problem raised at WMT 2019; and
* Significantly outperforms all QE metrics submitted to the WMT 2019 QE shared task

Finally, we contrast the effectiveness of our model when scoring MT output using the source vs the human reference. We observe that human references substantially improve performance, and, crucially, allow our model to rank systems that are _substantially better than our model at the task of translation_. This is important because it establishes that our method does not require building a state-of-the-art multilingual NMT model in order to produce a state-of-the-art MT metric capable of evaluating state-of-the-art MT systems.

We release our model, metrics toolkit, and pre-processed training data.2

Footnote 2: [https://github.com/thompsonb/prism](https://github.com/thompsonb/prism)

## 2 Related Work

MT MetricsEarly MT metrics like BLEU Papineni et al. (2002) and NIST Doddington (2002) use token-level n-gram overlap between the MT output and the human reference. Overlap can also be measured at the character level Popovic (2015, 2017) or using edit distance Snover et al. (2006). Many metrics use word- and/or sentence-level embeddings, including ReVal Gupta et al. (2015), RUSE Shimanaka et al. (2018), WWDO Chow et al. (2019), and ESIM Mathur et al. (2019). MEANT Lo and Wu (2011) and MEANT 2.0 Lo (2017) measure similarity between semantic frames and role fillers. State-of-the-art methods including YiSi Lo (2019) and BERTscore Zhang et al. (2019, 2020) rely on contextualized embeddings Devlin et al. (2019) trained on large (non-parallel) corpora. BLEURT Sellam et al. (2020) applies fine tuning of BERT, including training on prior human judgements. In contrast, our work exploits parallel bitext and doesn't require training on human judgements.

Paraphrase DatabasesPrior work explored using parallel bitext to identify phrase level paraphrases Bannard and Callison-Burch (2005); Ganitkevitch et al. (2013) including bitext in multiple language pairs Ganitkevitch and Callison-Burch (2014). Paraphrase tables were, in turn, used in MT metrics to reward systems for paraphrasing words Banerjee and Lavie (2005) or phrases Zhou et al. (2006); Denkowski and Lavie (2010) from the human reference. Our work can be viewed as extending this idea to the sentence level, without having to enumerate the millions or billions of paraphrases Dreyer and Marcu (2012) for each sentence.

Multilingual NMTMultilingual NMT Dong et al. (2015) has been shown to rival performance of single language pair models in high-resource languages Aharoni et al. (2019); Arivazhagan et al. (2019) while also improving low-resource translation via transfer learning from higher-resource languages Zoph et al. (2016); Nguyen and Chiang (2017); Neubig and Hu (2018). An extreme low-resource setting is where the system translates between languages seen during training, but in a language _pair_ where it did not see _any_ training 

data, denoted 'zero-shot' translation. Despite evidence that intermediate representations are not truly language-agnostic Kudugunta et al. (2019), zero-shot translation has been shown successful, especially between related languages Johnson et al. (2017); Gu et al. (2018); Pham et al. (2019).

Generative ParaphrasingSentential paraphrasing can be accomplished by training an MT system on paraphrase examples instead of translation pairs Quirk et al. (2004). While natural paraphrase datasets do exist Quirk et al. (2004); Coster and Kauchak (2011); Fader et al. (2013); Lin et al. (2014); Federmann et al. (2019), they are somewhat limited. An alternative is to start with much more plentiful bitext and back-translate one side into the language of the other to create synthetic paraphrases on which to train Prakash et al. (2016); Wieting and Gimpel (2018); Hu et al. (2019, 2019, 2019). Tiedemann and Scherrer (2019) propose using paraphrasing as a way to measure the semantic abstraction of multilingual NMT. They also propose using a multilingual NMT model as a generative paraphraser.3

Footnote 3: We find that generating from a well trained multilingual NMT system tends to produce copies of the input, as opposed to interesting paraphrases (see Appendix A).

Semantic SimilarityParallel corpora in many language pairs have been used to produce fixed-size, multilingual sentence representations Schwenk and Douze (2017); Wieting et al. (2017); Artetxe and Schwenk (2018); Wieting et al. (2019); Raganato et al. (2019). LASERArtetxe and Schwenk (2018), for example, trains a variant of NMT with a fixed-size intermediate representation in 93 languages. Embeddings produced by the encoder can be compared to measure intra- or interlingual semantic similarity.

## 3 Method

We propose using a paraphraser to force-decode and estimate probabilities of MT system outputs, conditioned on their corresponding human references. Let \(p(y_{t}|y_{i<t},x)\) be the probability our paraphraser assigns to the \(t^{\text{th}}\) token in output sequence \(y\), given the previous output tokens \(y_{i<t}\) and the input sequence \(x\). Table 1 shows an example of how token-level probabilities from our model (described in SS4) penalize both fluency and adequacy errors given a human reference. We consider two ways of combining token-level probabilities from the model--sequence-level log probability (\(G\)) and average token-level log probability (\(H\)):

\[G(y|x)=\sum_{t=1}^{|y|}\log p(y_{t}|y_{i<t},x)\] \[H(y|x)=\frac{1}{|y|}G(y|x)\]

Let \(\operatorname*{sys}\) denote an MT system output, \(\operatorname*{ref}\) denote a human reference, and \(\operatorname*{src}\) denote the source. We expect scoring \(\operatorname*{sys}\) conditioned on \(\operatorname*{ref}\) to be most indicative of the quality of \(\operatorname*{sys}\). However, we also explore scoring \(\operatorname*{ref}\) conditioned on \(\operatorname*{sys}\) as we find qualitatively that output sentences which drop some

\begin{table}
\begin{tabular}{l l c c c c c c c c c c} \hline \hline  & \multicolumn{5}{c}{**Word-level paraphraser log probabilities**} & \multicolumn{5}{c}{**H(out\(|\mathbf{in}\))****sBLEU**} & **LASER** \\ \hline Copy & Jason went to school & at & the University of Madrid & . & \(<\)EOS\(>\) & & & & & & \\  & -0.08 & -0.26 & -0.16 & -0.16 & -0.12 & -0.11 & -0.14 & -0.10 & -0.10 & -0.11 & -0.10 & -0.13 & 100.0 & 1.000 \\ \hline Disfluent & Jason went & **school** & at & **University** & of Madrid & . & \(<\)EOS\(>\) & & & & & & \\  & -0.08 & -0.26 & **-7.21** & -0.12 & **-4.81** & -0.10 & -0.11 & -0.10 & & & & & \\ \hline Inadequate & Jason & **will** & **go** & to school at & the University of Madrid & . & \(<\)EOS\(>\) & & & & & \\  & -0.08 & **-9.77** & **-0.76** & -0.22 & -0.19 & -0.14 & -0.15 & -0.16 & -0.10 & -0.10 & -0.12 & -0.10 & -0.10 & -0.99 & 70.8 & 0.960 \\ \cline{2-11}  & Jason went to school & at & the University of & & **Berlin** & . & \(<\)EOS\(>\) & & & & & & \\  & -0.08 & -0.26 & -0.16 & -0.16 & -0.12 & -0.11 & -0.14 & -0.10 & **-10.34** & -0.12 & -0.10 & -0.10 & -0.10 & 78.3 & 0.957 \\ \hline Fluent \& & Jason & **attended** & the University of & Madrid & . & \(<\)EOS\(>\) & & & & & & & \\ Adequate & -0.08 & **-2.01** & -1.63 & -0.42 & -0.10 & -0.09 & -0.16 & -0.10 & & & & -0.57 & 41.1 & 0.918 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Example token-level log probabilities from our model for various output sentences, conditioned on input sentence (i.e., human reference) “_Jason went to school at the University of Madrid.”_ H(out\(|\mathbf{in}\)) denotes the average token-level log probability. We observe that our model generally penalizes any deviations (**bolded**) from the input sentence, but tends to penalize deviations which change the meaning of the sentence or introduce a disfluency more harshly than those which are fluent and adequate. Sentence-level BLEU with smoothing=1 (“sBLEU”) and LASER embedding cosine similarity (“LASER”) are shown for comparison. We note that LASER appears fairly insensitive to disfluencies, and sentenceBLEU struggles to reward valid paraphrases.

 

meaning conveyed by the input sentence are penalized less harshly by the model than output sentences which contain extra information not present in the input. Scoring in both directions to penalize the presence of information in one sentence but not the other is similar, in spirit, to methods which use bi-directional textual entailment as an MT metric (Pado et al., 2009; Khobragade et al., 2019).4

Footnote 4: Conditional probabilities of MT systems in each direction have been shown effective at filtering MT training data (Junczys-Dowmunt, 2018).

We postulate that the output sentence that best represents the meaning of an input sentence is, in fact, simply a copy of the input sentence, as precise word order and choice often convey subtle connotations. As such, we seek a model whose output distribution is _centered around a copy of the input sentence_, which we denote a "lexically/syntactically unbiased paraphraser." While a standard generative paraphraser is trained to retain semantic meaning, it does not meet our criteria because it is _simultaneously_ trained to produce output which is lexically/syntactically different than its input, a key element in generative paraphrasing (Bhagat and Hovy, 2013).

We propose using a multilingual NMT system as a lexically/syntactically unbiased paraphrase. A multilingual NMT system consists of an encoder which maps a sentence in to an (ideally) language-agnostic semantic representation, and decoder to map that representation back to a sentence. The model has only seen bitext in training, but we propose to treat paraphrasing as a zero-shot "translation" (e.g., Czech to Czech).

Because our model is multilingual, we can also score MT system output conditioned on the source sentence instead of the human reference. This task is known as "quality estimation (QE) as a metric," and was part of the WMT19 QE shared task (Fonseca et al., 2019). We use "Prism-ref" to denote our reference-based metric and "Prism-src" to denote our system applied as a QE metric.

Our final metric and QE metric are defined based on results on our development set (see SS5.2) as follows:

\[\text{Prism-ref}=\frac{1}{2}H(\mathrm{sys|ref})+\frac{1}{2}H( \mathrm{ref|sys})\] \[\text{Prism-src}=H(\mathrm{sys|src})\]

To obtain system-level scores, we average segment-level scores over all segments in the test set.

## 4 Experiments

We train a multilingual NMT model and explore the extent to which it functions as a lexically/syntactically unbiased paraphraser. We then conduct several preliminary experiments on the WMT18 MT metrics data (Ma et al., 2018) to determine how to best utilize the token-level probabilities from the paraphraser, and report results on the WMT19 system- and segment-level metric tasks (Ma et al., 2019) and QE as a metric task (Fonseca et al., 2019).

### Data Preparation

Our method requires a model, which in turn relies heavily on the data on which it is trained, so we describe here the rationale behind the design decisions made regarding the training data. Full details sufficient for replication are provided in Appendix B.

Language-Agnostic RepresentationsTo encourage our intermediate representation to be as language-agnostic as possible, we choose datasets with as much language pair diversity as possible (i.e., not just en-* and *-en), as Kudugunta et al. (2019) has shown that encoder representation is affected by both the source language and target language. While it is common to append the target language token to the source sentence, we instead prepend it to the target sentence so that the encoder cannot do anything target-language specific with this tag. At test time, we force-decode the desired language tag prior to scoring.

NoiseNMT systems are known to be sensitive to noise, including sentence alignment errors (Khayrallah and Koehn, 2018), so we perform filtering with LASER (Schwenk, 2018; Chaudhary et al., 2019). We also perform language ID filtering using FastText (Joulin et al., 2016) to avoid training the decoder with incorrect language tags.

Number of LanguagesAharoni et al. (2019) found that performance of zero-shot translation in a related language pair increased substantially when increasing the number of languages from 5 languages and 25, with a performance plateau somewhere between 25 and 50 languages. We view paraphrasing as zero-shot translation between sentences in the same language, so we expect to need a similar number of languages 

CopiesWe filter sentence pairs with excessive copies and partial copies, as multiple studies (Ott et al., 2018; Khayrallah and Koehn, 2018) have noted that MT performance degrades substantially when systems are exposed to copies in training.

### Model Training

We train a Transformer Vaswani et al. (2017) model with approximately 745M parameters to translate between 39 languages. The full list of languages and data amounts used is provided in Appendix B, and model training details sufficient for replication are given in Appendix C. Training a single large model consumed the majority of our compute budget, thus performing ablations is beyond the scope of this work.

Our data comes primarily from WikiMatrix Schwenk et al. (2019), Global Voices,6 EuroParl Koehn (2005), SETimes,7 and United Nations Eisele and Chen (2010). The data processing described above and in Appendix B results in 99.8M sentence pairs in 39 languages.8 The most common language is English, at 16.7% of our data, while the least common 20 languages account for 21.9%.

Footnote 6: [http://casmacat.eu/corpus/global-voices.html](http://casmacat.eu/corpus/global-voices.html)

Footnote 7: [http://nlp.ffzg.hr/resources/corpora/setimes/](http://nlp.ffzg.hr/resources/corpora/setimes/)

Footnote 8: For every sentence pair (a,b) in our 99.8M examples, we train on both (a,b) and (b,a)

### Baselines and Contrastive Methods

We compare to all systems from the WMT19 shared metrics task, as well as BERTscore Zhang et al. (2020) and the recent BLEURT method Sellam et al. (2020). We also explore several contrastive methods. Training details sufficient for replication for each model/baseline are given in Appendix C.

Generative Sentential ParaphraserWe compare scoring with our Prism model vs a standard, English-only paraphraser trained on the ParaBank 2 dataset Hu et al. (2019). ParaBank 2 contains \(\sim\) 50M synthetic paraphraster pairs derived from back-translating a Czech-English corpus, and the authors report state-of-the-art paraphrasing results.

Auto-encoderAuto-encoders provide an alternative means of training seq2seq models, without the need for parallel bitext. We compare to scoring with the "multilingual denoising pre-trained model" (mBART) of Liu et al. (2020), as it works in all languages of interest.

LASERWe explore using the cosine distance between LASER embeddings of the MT output and human reference, using the pretrained 93-language model provided by the authors.9 We are particularly interested in LASER as it, like our model, is trained on parallel bitext in many languages.

Footnote 9: [http://casmacat.eu/corpus/global-voices.html](http://casmacat.eu/corpus/global-voices.html)

Language ModelWe find qualitatively that LASER is fairly insensitive to disfluencies (see Table 1), so we also explore augmenting it with language model (LM) scores of the system outputs. We train a multilingual language model (see Appendix C) on the same data as our multilingual NMT system.

### Paraphraser Bias

We expect that a lexically/syntactically unbiased measure of translation quality should (on average) increase with increased lexical similarity between a translation and reference. To explore the extent to which Prism and the model trained on ParaBank 2 are biased, we consider average \(H({\rm sys}|{\rm ref})\) as a function of binned lexical similarity (approximated by sentBLEU, with smoothing=1) for all \(({\rm sys},{\rm ref})\) pairs for all systems submitted to WMT19 in all language pairs into English. We also contrast the conditional probabilities of three outputs for the same input: (1) the sequence generated by the model via beam search; (2) a copy of the input; and (3) a human paraphrase of the input. Finally, we generate from the model using beam search and examine the outputs to see how much they differ from the inputs.

### MT Metrics Evaluation

We report results and statistical significance using scripts released with the WMT19 shared task. Segment-level performance is reported as the Kendall's \(\tau\) variant used in the shared task, and system-level performance is reported as Pearson correlation with the mean of the human judgments. Bootstrap resampling Koehn (2004); Graham et al. (2014) is used to estimate confidence intervals for each metric, and metrics with non-overlapping 95% confidence intervals are identified as having a statistically significant difference in performance 



## 5 Results

### Paraphraser Bias Results

We find \(H({\rm sys}|{\rm ref})\) increases monotonically with sentBLEU for the Prism model, but the model trained on ParaBank 2 has nearly the same scores for output with sentBLEU in the range of 60 to 100; however that range accounts for only about 8.5% of all system outputs (see Figure 2). We find that a copy of the input is almost as probable as beam search output for the Prism model. In contrast, the model trained on ParaBank 2 prefers its own beam search output to a copy of the input. Additionally, beam search from our model produces output which is more lexically similar to the input (BLEU of 82.8 with respect to input, vs 31.9 for ParaBank 2). ParaBank 2 tends to change the output in ways which occasionally significantly alter the meaning of the sentence. See Appendix A for more details. All of these findings support our hypothesis that our model is closer to an ideal lexically/syntactically unbiased paraphraser than the contrastive model trained on synthetic paraphrases.

\begin{table}
\begin{tabular}{l c c c c c c c c c|c c c} \hline \hline  & **en-cs** & **en-de** & **en-fi** & **en-gu** & **en-kk** & **en-lt** & **en-ru** & **en-zh** & **de-cs** & **de-fr** & **fr-de** \\ \hline BERTscore[4] & 0.485 & 0.345 & 0.524 & **0.558** & **0.533** & 0.463 & 0.580 & 0.347 & 0.352 & 0.325 & 0.274 \\ EED\({}^{\ddagger}\)[10] & 0.431 & 0.315 & 0.508 & **0.568** & 0.518 & 0.425 & 0.546 & 0.257 & 0.345 & 0.301 & 0.267 \\ YiSi-1°[13] (Loe, 2019) & 0.475 & 0.351 & 0.537 & **0.551** & **0.546** & 0.470 & 0.585 & **0.355** & 0.376 & 0.349 & 0.310 \\ YiSi-1.srl\({}^{\ddagger}\)[13] & – & 0.368 & — & — & — & — & — & **0.361** & — & — & 0.299 \\ \hline Prism-ref (This Work) & **0.582** & **0.427** & **0.591** & 0.313 & **0.531** & **0.558** & 0.584 & **0.376** & **0.458** & **0.453** & **0.426** \\ LASER + LM (Contrastive) & 0.535 & 0.401 & 0.568 & 0.306 & 0.408 & 0.503 & **0.640** & **0.356** & 0.431 & 0.401 & **0.381** \\ mBART (Contrastive) & 0.345 & 0.302 & 0.401 & 0.528 & 0.462 & 0.365 & 0.443 & 0.280 & 0.262 & 0.255 & 0.236 \\ \hline \hline  & & & **de-en** & **fi-en** & **gu-en** & **kk-en** & **lt-en** & **ru-en** & **zh-en** & \\ \hline BERtscore[4] & & 0.176 & 0.345 & **0.320** & **0.432** & **0.381** & **0.223** & **0.430** & \\ BLEURT[14] & & **0.204** & **0.367** & **0.311** & **0.447** & **0.387** & **0.228** & **0.423** & \\ ESIM\({}^{\dagger}\)[13] (Chen et al., 2017; Mathur et al., 2019) & 0.167 & 0.337 & 0.303 & **0.435** & 0.359 & 0.201 & 0.396 & \\ YiSi-1°[13] (Lo, 2019) & 0.164 & 0.347 & **0.312** & **0.440** & **0.376** & **0.217** & **0.426** & \\ YiSi-1.srl\({}^{\ddagger}\)[13] & **0.199** & 0.346 & 0.306 & **0.442** & **0.380** & **0.222** & **0.431** & \\ \hline Prism-ref (This Work) & **0.204** & **0.357** & **0.313** & **0.434** & **0.382** & **0.225** & **0.438** \\ Prism-ref w/ ParaBank 2 (Contrastive) & 0.184 & **0.341** & **0.326** & 0.425 & **0.373** & 0.207 & **0.432** \\ LASER + LM (Contrastive) & 0.190 & 0.335 & **0.319** & 0.428 & **0.368** & 0.207 & 0.416 & \\ mBART (Contrastive) & 0.136 & 0.255 & 0.246 & 0.377 & 0.298 & 0.162 & 0.349 & \\ \hline \hline \end{tabular}
\end{table}
Table 2: WMT19 segment-level human correlation (\(\tau\)), to non-English (top) and to English (bottom). **Gold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(\ddagger\):WMT19 Metric Submission. For brevity, only competitive baselines are shown. For complete results see Appendix E. Our models were not trained on Gujarati (gu). “LASER + LM” denotes the optimal linear combination found on the development set.

Figure 2: Average \(H({\rm sys}|{\rm ref})\) as a function of average lexical difference (as measured by sentBLEU) for every English \(({\rm sys},{\rm ref})\) pair submitted to WMT19, for both the Prism and ParaBank 2 paraphraisers. \((sys,ref)\) pairs are split into 10 sentBLEU bins of uniform width. Fraction of total data in each bin is shown on x-axis (in parentheses).





### Preliminary (Development) Results

We find that length-normalized log probability (\(H\)) slightly outperforms un-normalized log probability (\(G\)). When using the reference, we find an equal weighting of \(H({\rm sys}|{\rm ref})\) and \(H({\rm ref}|{\rm sys})\) to be approximately optimal, but we find that when using the source, \(H({\rm src}|{\rm sys})\) does not appear to add useful information to \(H({\rm sys}|{\rm src})\). Full results can be found in Appendix D. These findings were used to select the Prism-ref and Prism-src definitions (SS3).

We find that the probability of \({\rm sys}\) as estimated by an LM, as well as and the cosine distance between LASER embeddings of \({\rm sys}\) and \({\rm ref}\), both have decent correlation with human judgments and are complementary. However, cosine distance between LASER embeddings of \({\rm sys}\) and \({\rm src}\) have only weak correlation.

### Segment-Level Metric Results

Segment-level metric results are shown in Table 2. On language pairs into non-English, we outperform prior work by a statistically significant margin in 7 of 11 language pairs9 and are statistically tied for best in the rest, with the exception of Gujarati (gu) where the model had no training data. Into English, our metric is statistically tied with the best prior work in every language pair. Our metric tends to significantly outperform our contrastive LASER + LM and mBART methods, although LASER + LM performs surprisingly well in en-ru.

Footnote 9: In en–ru, Prism-ref is statistically tied with YiSi-1, ESIM, and BERTscore.

### System-Level Metric Results

Table 3 shows system-level metric performance on the top four systems submitted to WMT19 compared to selected metrics. While correlations are not high in all cases for Prism, they are at least all positive. In contrast, BLEU has negative correlation in 5 language pairs, and BERTscore and YiSi-1 variants are each negative in at least two. BLEURT has positive correlations in all language pairs into English, but is English-only. Note that Pearson's correlation coefficient may be unstable in this setting Mathur et al. (2020). For full top four system-level results see Appendix F.

We do not find the system-level results computed against _all_ submitted MT systems (see Appendix G) to be particularly interesting; as noted by Ma et al. (2019), a single weak system can result in high overall system-level correlation even for a very poor metric.

### QE as a Metric Results

We find that our reference-less Prism-src outperforms all QE as a metrics systems from the WMT19 shared task by a statistically significant margin, in every language pair at segment-level human correlation (Table 4), and outperforms or statistically ties at system-level human correlation (Appendix G).

## 6 Analysis and Discussion

How helpful are human references?The fact that our model is multilingual allows us to explore the extent to which the human reference actually improves our model's ability to judge MT system output, compared to using the source instead. The underlying assumption with any MT metric is that the work done by the human translator makes it easier to automatically judge the quality of MT output. However, if our model or the MT systems being judged were strong enough, we would expect this assumption to break down.

Comparing the performance of our method with access to the human reference (Prism-ref) vs our method with access to only the source (Prism-src), we find that the reference-based method statistically outperforms the source-based method in all but one language pair. We find the case where they are not statistically different, de-cs, to be particularly interesting: de-cs was the only language pair in WMT19 where the systems were unsupervised (i.e., did not use parallel training data). As a result, it is the only language pair where our model outperformed the best WMT system at translation. In most cases, our model is _substantially worse at translation_ than the best WMT systems. For example, in en-de and zh-en, two language pairs where strong NMT systems were especially problematic for MT metrics, the Prism model is 6.8 and 19.2 BLEU points behind the strongest WMT systems, respectively (see Table 5 for the Prism model compared to the best system submitted in each WMT19 language pair). Thus the performance difference between Prism-ref and Prism-src would suggest that the model needs no help in judging MT systems which are weaker than it is, but _the human references are assisting our model in evaluating MT systems which are stronger than it is._ This means that we have not simply reduced the task of MT evaluation to that of building a state-of-the-art MT

system. We see that a good (but not state-of-the-art) multilingual NMT system can be a state-of-the-art MT metric and judge state-of-the-art MT systems.

Finally, with the exception of de-cs discussed above, we see statistically significant improvements for Prism-ref over Prism-src both into English (where human judgments were reference-based) and into non-English (where human judgments were source-based). This suggests that the high correlation of Prism-ref with human judgements is not simply the result of reference bias [10].

Does paraphraser bias matter?Our lexically/syntactically unbiased paraphrase tends to outperforms the generative English-only ParaBank 2 paraphraser, but usually not by a statistically significant margin. Analysis indicate the lexical/syntactic bias is only harmful in somewhat infrequent cases where MT systems match or nearly match the reference, suggesting it would be more detrimental with stronger systems or multiple references. Our multilingual training method is much simpler than the alternative of creating synthetic paraphrases and training individual models in 39 languages, and our model may benefit from transfer learning to lower-resource languages.

Does fluency matter?Despite NMT being very fluent, our results suggest that fluency is fairly discriminative, especially in non-English: LM scoring outperforms sentenceBLEU at segment-level correlation in 7/10 language pairs to non-English languages (excluding Gujarati), for example. This is consistent with recent findings that LM scores can be used to augment BLEU [4].

\begin{table}
\begin{tabular}{l c c c c c c c|c c c c} \hline \hline  & **en–es** & **en–de** & **en–fi** & **en–gu** & **en–kk** & **en–lt** & **en–ru** & **en–zh** & **de–cs** & **de–fr** & **fr–de** \\ \hline Best WMT19 QE as Metric & 0.069\({}^{\text{d}}\) & 0.236\({}^{\text{b}}\) & 0.351\({}^{\text{e}}\) & 0.147\({}^{\text{a}}\) & 0.187\({}^{\text{a}}\) & 0.003\({}^{\text{a}}\) & 0.226\({}^{\text{c}}\) & 0.044\({}^{\text{a}}\) & 0.199\({}^{\text{a}}\) & 0.186\({}^{\text{a}}\) & 0.066\({}^{\text{a}}\) \\ Prism-src (This work) & **0.470** & **0.402** & **0.555** & **0.215** & **0.507** & **0.499** & **0.486** & **0.287** & **0.444** & **0.371** & **0.316** \\ \hline  & & & **de-en** & **fi-en** & **eu-en** & **kk-en** & **lt-en** & **ru-en** & **zh-en** & **zh-en** & \\ \cline{2-10} Best WMT19 QE as Metric & 0.068\({}^{\text{b}}\) & 0.211\({}^{\text{d}}\) & –\(0.001^{\text{i}}\) & 0.096\({}^{\text{d}}\) & 0.075\({}^{\text{a}}\) & 0.089\({}^{\text{d}}\) & 0.253\({}^{\text{a}}\) & & & & & \\ Prism-src (This work) & **0.109** & **0.300** & **0.102** & **0.391** & **0.356** & **0.178** & **0.336** & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 4: WMT19 segment-level human correlation (\(\tau\)) for QE as Metric systems (which have access to the source only, not the reference). **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. Our models were not trained on Gujarati (gu). For brevity, only the best QE-metric for each language pair is shown—for full results see Appendix G. a:YISI-2 [10] b:YISI-2_srl 

Can we measure adequacy and fluency separately?The proposed method significantly outperforms the contrastive LASER-based method in most language pairs, even when LASER is augmented with a language model. This suggests that jointly optimizing a model for adequacy and fluency is better than optimizing them independently and combining after the fact--this is unsurprising given that neural MT has shown significant improvements over statistical MT, where a phrase table and language model were trained separately.

Can we train on monolingual data instead of bitext?The proposed method significantly outperforms scoring with the mBART auto-encoder, which is trained on large amounts of monolingual data, despite using substantially less compute power (1.3 weeks on 8 V100s for Prism vs 2.5 weeks on 256 V100s for mBART).

## 7 Conclusion and Future Work

We show that a multilingual NMT system can be used as a lexically/syntactically unbiased, multilingual paraphraser, and that the resulting paraphraser can be used as an MT metric and QE metric. Our method achieves state-of-the-art performance on the most recent WMT shared metrics and QE tasks, without training on prior human judgements.

We release a single model which supports 39 languages. To the best of our knowledge, we are the first to release a large multilingual NMT system, and we hope others follow suit. We are optimistic our method will improve further as stronger multilingual NMT models become publicly available.

We compare our method to several contrastive methods and present analysis showing that we have not simply reduced the task of evaluation to that of building a state-of-the-art MT system; the work done by the human translator to create references helps the evaluation model to judge systems that are stronger (at translation) than it is.

Nothing in our method is specific to sentence-level MT. In future work, we would like to extend Prism to paragraph- or document-level evaluation by training a paragraph- or document-level multilingual NMT system, as there is growing evidence that MT evaluation would be better conducted at the document level, rather than the sentence level Laubli et al. (2018).

## Acknowledgments

Brian Thompson is supported by the National Defense Science and Engineering Graduate (NDSEG) Fellowship.

## References

* Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
* Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. _arXiv preprint arXiv:1907.05019_.
* Berman et al. (2018)

\begin{table}
\begin{tabular}{l r r r} \hline \hline Lang & \multicolumn{3}{c}{BLEU} \\ Pair & WMT19 Best & Multilingual & \(\Delta\) \\ \hline de–cs & 20.1\(\dagger\) & 21.8 & +1.7 \\ de–en & 42.8 & 35.5 & -7.3 \\ de–fr & 37.3 & 33.9 & -3.4 \\ en–cs & 29.9 & 24.2 & -5.7 \\ en–de & 44.9 & 38.1 & -6.8 \\ en–fi & 27.4 & 21.9 & -5.5 \\ en–gu & 28.2 & 0.0\(\ddagger\) & -28.2 \\ en–kk & 11.1 & 8.6 & -2.5 \\ en–lt & 20.1 & 15.0 & -5.1 \\ en–ru & 36.3 & 28.1 & -8.2 \\ en–zh & 44.6 & 30.1 & -14.5 \\ fi–en & 33.0 & 26.2 & -6.8 \\ fr–de & 35.0 & 26.4 & -8.6 \\ gu–en & 24.9 & 0.4\(\ddagger\) & -24.5 \\ kk–en & 30.5 & 27.7 & -2.8 \\ lt–en & 36.3 & 28.5 & -7.8 \\ ru–en & 40.1 & 36.1 & -4.0 \\ zh–en & 39.9 & 20.6 & -19.3 \\ \hline \hline \end{tabular}
\end{table}
Table 5: BLEU scores for our multilingual NMT system on WMT19 testsets, compared to best system from WMT19. Our multilingual system achieves state-of-the-art performance as an MT metric despite substantially under performing all the best WMT19 MT systems at translation (excluding unsupervised). \(\dagger\): WMT systems were unsupervised (no parallel data). \(\ddagger\): Multilingual system did not train on Gujarati (gu). Systems are not trained on the same data, so this should not be interpreted as a comparison between multilingual and single-language pair MT. ISO 639-1 language codes.

 

Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. _arXiv preprint arXiv:1812.10464_.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
* Bannard and Callison-Burch (2005) Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)_, pages 597-604, Ann Arbor, Michigan. Association for Computational Linguistics.
* Bhagat and Hovy (2013) Rahul Bhagat and Eduard Hovy. 2013. Squibs: What is a paraphrase? _Computational Linguistics_, 39(3):463-472.
* Bojar et al. (2017) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In _Proceedings of the Second Conference on Machine Translation_, pages 169-214, Copenhagen, Denmark. Association for Computational Linguistics.
* Chaudhary et al. (2019) Vishrav Chaudhary, Yuqing Tang, Francisco Guzman, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_, pages 261-266, Florence, Italy. Association for Computational Linguistics.
* Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics.
* Chow et al. (2019) Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019. WMDO: Fluency-based word mover's distance for machine translation evaluation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 494-500, Florence, Italy. Association for Computational Linguistics.
* Coster and Kauchak (2011) William Coster and David Kauchak. 2011. Simple English Wikipedia: A new text simplification task. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 665-669, Portland, Oregon, USA. Association for Computational Linguistics.
* Denkowski and Lavie (2010) Michael Denkowski and Alon Lavie. 2010. Extending the METEOR machine translation evaluation metric to the phrase level. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 250-253, Los Angeles, California. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Doddington (2002) George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In _Proceedings of the second international conference on Human Language Technology Research_, pages 138-145.
* Dong et al. (2015) Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1723-1732, Beijing, China. Association for Computational Linguistics.
* Dreyer and Marcu (2012) Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalent semantics for translation evaluation. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 162-171, Montreal, Canada. Association for Computational Linguistics.
* Duma and Menzel (2017) Melania Duma and Wolfgang Menzel. 2017. UHH submission to the WMT17 metrics shared task. In _Proceedings of the Second Conference on Machine Translation_, pages 582-588, Copenhagen, Denmark. Association for Computational Linguistics.
* Edunov et al. (2020) Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020. On the evaluation of machine translation systems trained with back-translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2836-2846, Online. Association for Computational Linguistics.
* Eisele and Chen (2010) Andreas Eisele and Yu Chen. 2010. Multiun: A multi-lingual corpus from united nation documents. In _LREC_.
* Fader et al. (2013) Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question 

answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria, pp. 1608-1618. External Links: Link Cited by: SS2.
* C. Federmann, O. Elachapar, and C. Quirk (2019)Multilingual whispers: generating paraphrases with translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (WCNUT 2019), Hong Kong, China, pp. 17-26. External Links: Link Cited by: SS3.4.
* M. Fomicheva and L. Specia (2016)Reference bias in monolingual machine translation evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Berlin, Germany, pp. 77-82. External Links: Link Cited by: SS2.
* E. Fonseca, L. Yankovskaya, A. F. T. Martins, M. Fishel, and C. Federmann (2019)Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), Florence, Italy, pp. 1-10. External Links: Link Cited by: SS2.
* J. Ganitkevitch and C. Callison-Burch (2014)The multilingual paraphrase database. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), Reykjavik, Iceland, pp. 4276-4283. External Links: Link Cited by: SS2.
* J. Gantikevitch, B. Van Durme, and C. Callison-Burch (2013)PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia, pp. 758-764. External Links: Link Cited by: SS2.
* Y. Graham, N. Mathur, and T. Baldwin (2014)Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland, USA, pp. 266-274. External Links: Link Cited by: SS2.
* J. Gu, H. Hassan, J. Devlin, and V. O.K. Li (2018)Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 344-354. External Links: Link Cited by: SS2.
* Y. Guo and J. Hu (2019)Meteor++ 2.0: adopt syntactic level paraphrase knowledge into machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 501-506. External Links: Link Cited by: SS2.
* R. Gupta, C. Orasan, and J. van Genabith (2015)ReVal: a simple and effective machine translation evaluation metric based on recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 1066-1072. External Links: Link Cited by: SS2.
* A. L. Fan, D. Wong, L. Chao, L. He, Y. Lu, J. Xing, and X. Zeng (2013)Mt summit13.language-independent model for machine translation evaluation with reinforced factors.. Cited by: SS2.
* A. L. F. Han, D. F. Wong, and L. S. Chao (2012)LEPOR: a robust evaluation metric for machine translation with augmented factors. In Proceedings of COLING 2012: Posters, Vol. 441-450, Mumbai, India, pp. 441-450. External Links: Link Cited by: SS2.
* H. Hassan, A. Aue, C. Chen, V. Chowdhary, J. Clark, C. Federmann, Xuedong Huang, M. Junczys-Dowmunt, W. Lewis, M. Li, S. Liu, T. Liu, R. Luo, A. Menezes, T. Qin, F. Seide, X. Tan, F. Tian, L. Wu, S. Wu, Y. Xia, D. Zhang, Z. Zhang, and M. Zhou (2018)Achieving human parity on automatic chinese to english news translation. CoRRabs/1803.05567. External Links: Link Cited by: SS2.
* J. Hu, H. Khayrallah, R. Culkin, P. Xia, T. Chen, M. Post, and B. Van Durme (2019)Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 839-850. External Links: Link Cited by: SS2.
* J. Howard, R. Rudinger, M. Post, and B. Van Durme (2019)PARABAR: monolingual bitext generation and sentential paraphrasing via lexically-constrained neural machine translation. In Proceedings of AAAI, Cited by: SS2.
* J. Howard, A. Singh, N. Holzenberger, M. Post, and B. Van Durme (2019)Large-scale, diverse, paraphrastic bittexts via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Hong Kong, China, pp. 44-54. External Links: Link Cited by: SS2.
* Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiamu, Q. V Le, Yonghui Wu, and zhifeng Chen (2019)Gpipe: efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. Garnett (Eds.), Lecture Notes in Computer Science, Vol. 32, pp. 103-112. External Links: Link Cited by: SS2.

 

Julia Ive, Frederic Blain, and Lucia Specia. 2018. deep-Quest: A framework for neural-based quality estimation. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3146-3157, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. _Transactions of the Association for Computational Linguistics_, 5:339-351.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. _arXiv preprint arXiv:1607.01759_.
* Junczys-Dowmunt (2018) Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 888-895, Belgium, Brussels. Association for Computational Linguistics.
* Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 74-83, Melbourne, Australia. Association for Computational Linguistics.
* Khobragade et al. (2019) Rakesh Khobragade, Heaven Patel, Anand Namdev, Anish Mishra, and Pushpak Bhattacharyya. 2019. Machine translation evaluation using bi-directional entailment.
* Koehn (2004) Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _MT summit_, volume 5, pages 79-86. Citeseer.
* Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
* Kudugunta et al. (2019) Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1565-1575, Hong Kong, China. Association for Computational Linguistics.
* Laubli et al. (2018) Samuel Laubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics.
* Leusch et al. (2006) Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDER: Efficient MT evaluation using block movements. In _11th Conference of the European Chapter of the Association for Computational Linguistics_, Trento, Italy. Association for Computational Linguistics.
* ECCV 2014_, pages 740-755, Cham. Springer International Publishing.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation.
* Lo (2017) Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In _Proceedings of the Second Conference on Machine Translation_, pages 589-597, Copenhagen, Denmark. Association for Computational Linguistics.
* a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 507-513, Florence, Italy. Association for Computational Linguistics.
* Lo and Wu (2011) Chi-kiu Lo and Dekai Wu. 2011. MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 220-229, Portland, Oregon, USA. Association for Computational Linguistics.
* Ma et al. (2018) Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 671-688, Belgium, Brussels. Association for Computational Linguistics.
* Ma et al. (2017) Qingsong Ma, Yvette Graham, Shugu Wang, and Qun Liu. 2017. Blend: a novel combined MT metric based on direct assessment -- CASICT-DCU submission to WMT17 metrics task. In _Proceedings of the Second Conference on Machine Translation_, pages 598-603, Copenhagen, Denmark. Association for Computational Linguistics.

 

* Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 62-90, Florence, Italy. Association for Computational Linguistics.
* Mathur et al. (2019) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
* Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4984-4997, Online. Association for Computational Linguistics.
* Neubig and Hu (2018) Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 875-880, Brussels, Belgium. Association for Computational Linguistics.
* Nguyen and Chiang (2017) Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 296-301, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In _International Conference on Machine Learning_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Pado et al. (2009) Sebastian Pado, Michel Galley, Dan Jurafsky, and Christopher D. Manning. 2009. Robust machine translation evaluation with entailment features. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 297-305, Suntec, Singapore. Association for Computational Linguistics.
* Panja and Nakkar (2018) Joybrata Panja and Sudip Kumar Nakkar. 2018. ITER: Improving translation edit rate through optimizable edit costs. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 746-750, Belgium, Brussels. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pham et al. (2019) Ngoc-Quan Pham, Jan Niehues, Thanh-Le Ha, and Alexander Waibel. 2019. Improving zero-shot translation with language-independent constraints. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 13-23, Florence, Italy. Association for Computational Linguistics.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram f-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Popovic (2017) Maja Popovic. 2017. chrF++: words helping character n-grams. In _Proceedings of the Second Conference on Machine Translation_, pages 612-618, Copenhagen, Denmark. Association for Computational Linguistics.
* Popovic et al. (2011) Maja Popovic, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. 2011. Evaluation without references: IBM1 scores as evaluation metrics. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_, pages 99-103, Edinburgh, Scotland. Association for Computational Linguistics.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
* Prakash et al. (2016) Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual LSTM networks. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 2923-2934, Osaka, Japan. The COLING 2016 Organizing Committee.
* Quirk et al. (2004) Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 142-149, Barcelona, Spain. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8):9.
* Raganato et al. (2019) Alessandro Raganato, Raul Vazquez, Mathias Creutz, and Jorg Tiedemann. 2019. An evaluation of language-agnostic inner-attention-based representations in machine translation. In _Proceedings of the 

4th Workshop on Representation Learning for NLP (RepLANLP-2019)_, pages 27-32, Florence, Italy. Association for Computational Linguistics.
* Schwenk [2018] Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 228-234, Melbourne, Australia. Association for Computational Linguistics.
* Schwenk et al. [2019] Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzman. 2019. WikiMatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. _CoRR_, abs/1907.05791.
* Schwenk and Douze [2017] Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations with neural machine translation. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_, pages 157-167, Vancouver, Canada. Association for Computational Linguistics.
* Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Shimanaka et al. [2018] Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 751-758, Belgium, Brussels. Association for Computational Linguistics.
* Snover et al. [2006] Matthew Snower, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In _Proceedings of association for machine translation in the Americas_, volume 200.
* Stanchev et al. [2019] Peter Stanchev, Weiyue Wang, and Hermann Ney. 2019. EED: Extended edit distance measure for machine translation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 514-520, Florence, Italy. Association for Computational Linguistics.
* Stanojevic and Sima'an [2015] Milos Stanojevic and Khalil Sima'an. 2015. BEER 1.1: ILLC UvA submission to metrics and tuning task. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 396-401, Lisbon, Portugal. Association for Computational Linguistics.
* Tiedemann and Scherrer [2019] Jorg Tiedemann and Yves Scherrer. 2019. Measuring semantic abstraction of multilingual NMT with paraphrase recognition and generation tasks. In _Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP_, pages 35-42, Minneapolis, USA. Association for Computational Linguistics.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wang et al. [2016] Weiyue Wang, Jan-Thorbsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation edit rate on character level. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_, pages 505-510, Berlin, Germany. Association for Computational Linguistics.
* Wieting and Gimpel [2018] John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 451-462, Melbourne, Australia. Association for Computational Linguistics.
* Wieting et al. [2019] John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Simple and effective paraphrastic similarity from parallel translations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4602-4608, Florence, Italy. Association for Computational Linguistics.
* Wieting et al. [2017] John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 274-285, Copenhagen, Denmark. Association for Computational Linguistics.
* Yankovskaya et al. [2019] Elizaveta Yankovskaya, Andre Tattar, and Mark Fishel. 2019. Quality estimation and translation metrics via pre-trained word and sentence embeddings. In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_, pages 101-105, Florence, Italy. Association for Computational Linguistics.
* Yoshimura et al. [2019] Ryoma Yoshimura, Hiroki Shimanaka, Yukio Matsumura, Hayahide Yamagishi, and Mamoru Komachi. 2019. Filtering pseudo-references by paraphrasing for automatic evaluation of machine translation. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 521-525, Florence, Italy. Association for Computational Linguistics.
* Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zhang et al. [2020] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_ 

* Zhou et al. (2006) Liang Zhou, Chin-Yew Lin, Dragos Stefan Munteanu, and Eduard Hovy. 2006. ParaEval: Using paraphrases to evaluate summaries automatically. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_, pages 447-454, New York City, USA. Association for Computational Linguistics.
* Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1568-1575, Austin, Texas. Association for Computational Linguistics.

 

Generation Examples

Figure 3 shows sentences generated from both our model and the model trained on ParaBank 2.

We also contrast the conditional probabilities of three outputs for the same input: (1) the sequence generated by the model via beam search; (2) a copy of the input; and (3) a human paraphrase of the input. We use the English side of the zh-en newstest17 (Bojar et al., 2017) as input, so that we can use the second human reference released by Hassan et al. (2018) as a human paraphrase. Table 6 shows the results of scoring a copy of the input, a human paraphrase of the input, and a model's beam search output, for both our multilingual paraphraser and the ParaBank 2 model.

\begin{table}
\begin{tabular}{l c r} \hline \hline  & ParaBank 2 & This Work \\ \hline \(H(BS|r0)\) & -0.501 & -0.225 \\ \(H(r0|r0)\) & -1.157 & -0.303 \\ \(H(r1|r0)\) & -2.246 & -2.187 \\ \hline \(BLEU(BS,r0)\) & 31.9 & 82.8 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Average token log probability (\(H\)) for a sequence generated via beam search (\(BS\)), a copy of the input (\(r0\)), and a high-quality human paraphrase of the input (\(r1\)), for a generative paraphraser vs our model, conditioned on \(r0\) in all cases. BLEU is also computed for the beam search output of each model, with respect to \(r0\). Note that BLEU for \(r1\) with respect to \(r0\) is 17.1.

 



## Appendix A

Figure 3: Sentences generated via beam search (beamwidth 5) for the multilingual model presented in this work vs ParaBank 2. We note that our model tends to produces copies or near copies of the input, which is the desired behavior for our application. Changes are emphasized with **bold** or **strikethrough**. The model trained on ParaBank 2 tends to produce output with lexical/syntactic changes, which occasionally also significantly change the meaning of the sentence (denoted in red). References (paraphraser inputs) are the first ten sentences of WMT17 zh-en.



Data Details for Replication

Much of our data comes from WikiMatrix (Schwenk et al., 2019), a large collection of parallel data extracted from Wikipedia, and for more domain variety, we added Global Voices,10 EuroParl (Koehn, 2005) (random subset of to 100k sentence pairs per language pair), SETimes,11 United Nations (Eisele and Chen, 2010) (random sample of 1M sentence pairs per language pair). We also included WMT Kazakh-English and Kazakhstan from WMT, to be able to evaluate on Kazakh.

Footnote 10: [http://casmacat.eu/corpus/global-voices.html](http://casmacat.eu/corpus/global-voices.html)

Footnote 11: [http://nlp.ffzg.hr/resources/corpora/setimes/](http://nlp.ffzg.hr/resources/corpora/setimes/)

WMT Kazakh-English and Kazakh-Russian were limited to the best 1M and 200k sentence pairs, respectively, as judged by LASER. We used a margin threshold of 1.05 for WikiMatrix and a threshold of 1.04 for the remaining datasets, as we expect them to be cleaner. We find that FastText classifies many sentences as non-English when they contain mostly English but also contain a few non-English words, especially from lower resource languages. To remedy this, we performed language identification (LID) on 5-grams and filtered out sentences for which LID did not classify at least half of the 5-grams as the expected language.

We filtered out sentences where there was more than 60% overlap in 3-grams or 40% overlap in 4-grams. Via manual inspection, this seemed to provide a good trade-off between allowing numbers and named entities to be copied, and filtering out sentences that were clearly not translated. We perform tokenization with SentencePiece (Kudo and Richardson, 2018) prior to filtering, using a 200k vocabulary for all language pairs, to account for languages like Chinese which do not denote word boundaries. Note that this vocabulary was used only for filtering, not for training the final model.

We limited training to languages with at least 1M examples, which resulted in 39 languages. Figure 4 shows the languages and amount of data in each language.

Figure 4: Distribution of the 39 languages (ISO 639-1 language code) of the 99.8M training sentences. English accounts for 16.7%. Spanish, French, Russian, Portuguese, German, and Italian account for a combined 34.3%. The bottom 20 languages account for only 21.9% combined.

 

Model Training Details for Replication

### Primary Model

We train a SentencePiece (Kudo and Richardson, 2018) model with a 64k vocabulary size on the concatenation of all data, and filter sentences with length greater than 200 subwords. Multilingual NMT performance has been found to increase significantly with model size - tor example, the best performance of Huang et al. (2019) is with their largest model which has 6 billion parameters. Training such a model is well beyond the scope of this work, but we train a model as large a feasible given our compute budget constraints. We train a Transformer (Vaswani et al., 2017) in fairseq (Ott et al., 2019) with eight encoder layers, eight decoder layers, an embedding size of 1280, feed forward layer size of 12288, 20 attention heads, learning rate of \(0.0004\), batch size of 1800 tokens with gradient accumulation over 200 batches, gradient clipping of 1.2, and dropout of 0.1. The model has approximately 745M parameters for 39 languages. We train for 6 epochs, which takes approximately 9 days on ap3.16xlarge instance rented from Amazon AWS, which has 8 Volta V100 GPUs with 16 GB of memory each. No hyperparameters were swept, as training a single model used the majority of our compute budget (the total cost for training this model was approximately $13,000 USD). However, we did restart training after discovering that LID was not performing well and adding the 5-gram LID filtering.

### ParaBank 2 Model

We train a contrastive, English-only paraphraser on the ParaBank 2 dataset (Hu et al., 2019c). We train a Transformer with an 8-layer encoder, 8-layer decoder, \(1024\) dimensional embeddings, embedding sizes of \(1024\), feed-forward size of \(4096\), and \(16\) attention heads. We use a SentencePiece model with a 16k vocabulary size. Dropout is \(0.3\), label smoothing is \(0.1\), and learning rate is \(0.0005\). The model has approximately 253M parameters for 1 language. Batch size is 31200 tokens, and the model trains for approximately 6 weeks (33 epochs) on 4 Nvidia 2080 GPUs.

### Language Model

We train a multilingual language model on the same data as our multilingual NMT system.

The model architecture is based on GPT-2 (Radford et al., 2019), and we use the fairseq transformer_lm_gpt2_small implementation. We train for 200k updates (18 epochs) of approximately 131k tokens. The model has 369M parameters for 39 languages. We train with shared embeddings and a learning rate of \(0.0005\), and we stop gradients at sentence boundaries, using --sample-break-mode eos as the model will be used to evaluate individual sentences. Other parameters match the fairseq defaults. The model trained for approximately 4 weeks on 4 Nvidia TITAN RTX GPUs.

### Autoencoder

We use the pretrained "multilingual denoising pre-trained model" (mBART) model of Liu et al. (2020), as it works in all languages of interest. Their model is designed to be fine-tuned to translation tasks, and their fine-tuning introduces subtle changes to the decoder that are required for inference. In order to adapt it to our task, we therefore fine-tune for a single update with a learning rate of 0. We then produce scores with the model in the same manner as Prism-ref. The model has approximately 680M parameters for 25 languages. We did not train this model but note that doing so required substantial compute power - Liu et al. (2020) note that they trained for approximately 2.5 weeks on 256 Nvidia V100 GPUS, each with 32GB of memory.

### Baselines

We compare to BLEURT (Sellam et al., 2020) using the authors' recommended "BLEURT-Base 128"12 We compare to BERTscore F1 (Zhang et al., 2020) using the model and code provided by the authors.13 

The remaining baseline results are computed using the metric scores as submitted to (Ma et al., 2019)14

Footnote 14: [http://data.statmt.org/wmt19/translation-task/wmt19-submitted-data-v3.tgz](http://data.statmt.org/wmt19/translation-task/wmt19-submitted-data-v3.tgz) 

WMT 2018 (Development set) Results: System-level, Segment-level, and Sweeps

Figure 5 shows results on the development set (WMT18) for sweeping various linear combinations.

Table 7, Table 8, Table 9 and Table 10, show full segment- and system- level results, into and out of English, for the WMT 2018 MT metrics shared task, along with all baselines and submitted systems.

 



\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **cs-en** & **de-en** & **et-cn** & **fi-en** & **ru-en** & **tr-en** & **zh-en** \\ n & 5110 & 77811 & 56721 & 15648 & 10404 & 8525 & 33357 \\ \hline BEER\({}^{\dagger}\)Stanojevic and Sima’an, 2015 & 0.295 & 0.481 & 0.341 & 0.232 & 0.288 & 0.229 & 0.214 \\ BERTscoreZhang et al. (2019, 2020) & **0.404** & 0.550 & **0.397** & **0.296** & **0.340** & **0.292** & **0.253** \\ BLEND\({}^{\ddagger}\)Ma et al. (2017) & 0.322 & 0.492 & 0.354 & 0.226 & 0.290 & 0.232 & 0.217 \\ CharactTER\({}^{\ddagger}\)Wang et al. (2016) & 0.256 & 0.450 & 0.286 & 0.185 & 0.244 & 0.172 & 0.202 \\ chrF\({}^{\ddagger}\)Popovic (2015) & 0.288 & 0.479 & 0.328 & 0.229 & 0.269 & 0.210 & 0.208 \\ chrF+\({}^{\ddagger}\)Popovic (2017) & 0.288 & 0.479 & 0.332 & 0.234 & 0.279 & 0.218 & 0.207 \\ ITER\({}^{\ddagger}\)Pana and Nakar (2018) & 0.198 & 0.396 & 0.235 & 0.128 & 0.139 & -0.029 & 0.144 \\ meteor++Shimamaka et al. (2018) & 0.270 & 0.457 & 0.329 & 0.207 & 0.253 & 0.204 & 0.179 \\ RUSE\({}^{\ddagger}\)Shimanaka et al. (2018) & 0.347 & 0.498 & 0.368 & 0.273 & 0.311 & 0.259 & 0.218 \\ sentBLEU\({}^{\dagger}\)Papineni et al. (2002) & 0.233 & 0.415 & 0.285 & 0.154 & 0.228 & 0.145 & 0.178 \\ UHH\_TSKM\({}^{\ddagger}\)Duma and Menzel (2017) & 0.274 & 0.436 & 0.300 & 0.168 & 0.235 & 0.154 & 0.151 \\ YiSi-0\({}^{\ddagger}\)Lo (2019) & 0.301 & 0.474 & 0.330 & 0.225 & 0.294 & 0.215 & 0.205 \\ YiSi-1\({}^{\dagger}\)Lo (2019) & 0.319 & 0.488 & 0.351 & 0.231 & 0.300 & 0.234 & 0.211 \\ YiSi-1\({}^{\ddagger}\)Lo (2019) & 0.317 & 0.483 & 0.345 & 0.237 & 0.306 & 0.233 & 0.209 \\ \hline Prism-ref (This Work) & **0.423** & **0.560** & **0.409** & **0.317** & **0.366** & **0.309** & **0.263** \\ Prism-ref w/ ParaBank 2 (Contrastive) & **0.386** & 0.538 & **0.399** & **0.309** & **0.340** & 0.275 & 0.244 \\ LASER + LM (Contrastive) & 0.364 & 0.526 & 0.378 & 0.265 & 0.305 & 0.257 & 0.243 \\ Prism-src (This work) & 0.355 & 0.515 & 0.370 & 0.257 & 0.308 & 0.213 & 0.194 \\ LM & 0.285 & 0.438 & 0.285 & 0.198 & 0.280 & 0.123 & 0.192 \\ LASER & 0.310 & 0.494 & 0.364 & 0.232 & 0.257 & 0.248 & 0.207 \\ mBART (Contrastive) & 0.251 & 0.455 & 0.315 & 0.199 & 0.248 & 0.196 & 0.181 \\ \hline \hline \end{tabular}
\end{table}
Table 7: WMT18 Segment-level results, to English. n denotes number of pairwise judgments. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. We exclude BLEURTSellam et al. (2020) as it was directly trained on WMT18 judgements. \(\dagger\):WMT18 Baseline Ma et al. (2018) \(\ddagger\):WMT18 Metric Submission Ma et al. (2018)

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **en-cs** & **en-de** & **en-et** & **en-fi** & **en-ru** & **en-tr** & **enh-zh** \\ n & 5413 & 19711 & 32202 & 9809 & 22181 & 1358 & 28602 \\ \hline BEER\({}^{\ddagger}\)Stanojević and Sima’an (2015) & 0.518 & 0.686 & 0.558 & 0.511 & 0.403 & 0.374 & 0.302 \\ BERIscoreZhang et al. (2019, 2020) & 0.559 & 0.727 & 0.584 & 0.538 & 0.424 & 0.389 & **0.364** \\ BLEND\({}^{\ddagger}\)Ma et al. (2017) & – & — & — & — & 0.394 & — & — \\ CharacterTER\({}^{\ddagger}\)Wang et al. (2016) & 0.414 & 0.604 & 0.464 & 0.403 & 0.352 & 0.404 & 0.313 \\ chrF\({}^{\ddagger}\)Popovic (2015) & 0.516 & 0.677 & 0.572 & 0.520 & 0.383 & 0.409 & 0.328 \\ chrF+\({}^{\ddagger}\)Popović (2017) & 0.513 & 0.680 & 0.573 & 0.525 & 0.392 & 0.405 & 0.328 \\ ITER\({}^{\dagger}\)Panja and Nakar (2018) & 0.333 & 0.610 & 0.392 & 0.311 & 0.291 & 0.236 & — \\ sentBLEU\({}^{\dagger}\)Papinen et al. (2002) & 0.389 & 0.620 & 0.414 & 0.355 & 0.330 & 0.261 & 0.311 \\ YiSI-0\({}^{\ddagger}\)Lo (2019) & 0.471 & 0.661 & 0.531 & 0.464 & 0.394 & 0.376 & 0.318 \\ YiSI-1\({}^{\ddagger}\)Lo (2019) & 0.496 & 0.691 & 0.546 & 0.504 & 0.407 & 0.418 & 0.323 \\ ViSI-1\({}^{\ddagger}\)Lo (2019) & — & 0.696 & — & — & — & — & 0.310 \\ \hline Prism-ref (This Work) & **0.667** & **0.799** & **0.705** & **0.667** & **0.469** & **0.574** & **0.371** \\ LASER + LM (Contrastive) & 0.587 & 0.746 & 0.628 & 0.629 & 0.450 & **0.501** & **0.367** \\ Prism-src (This work) & 0.552 & 0.732 & 0.636 & 0.626 & 0.409 & **0.505** & 0.298 \\ LM & 0.459 & 0.655 & 0.408 & 0.511 & 0.375 & 0.331 & 0.221 \\ LASER & 0.480 & 0.677 & 0.585 & 0.511 & 0.402 & 0.432 & 0.338 \\ mBART (Contrastive) & 0.404 & 0.594 & 0.405 & 0.410 & 0.356 & 0.303 & 0.305 \\ \hline \hline \end{tabular}
\end{table}
Table 8: WMT18 Segment-level results, from English. n denotes number of pairwise judgments. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(\dagger\):WMT18 Baseline Ma et al. (2018) \(\ddagger\):WMT18 Metric Submission Ma et al. (2018)



\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline  & **en-cs** & **en–de** & **en–et** & **en-fi** & **en-ru** & **en–tr** & **en-zh** \\ n & 5 & 16 & 14 & 9 & 8 & 5 & 14 \\ \hline BEER\({}^{\ddagger}\)Stanojevic and Sima’an, 2015 & 0.958 & 0.994 & **0.985** & 0.991 & 0.982 & 0.870 & **0.976** \\ BERTScoreZhang et al. (2019, 2020) & **0.990** & **0.999** & **0.990** & **0.998** & 0.935 & 0.499 & 0.956 \\ BLEND\({}^{\ddagger}\)Ma et al. (2017) & 0.973 & 0.991 & 0.985 & **0.994** & **0.993** & **0.801** & **0.976** \\ BLEU\({}^{\dagger}\)Papineni et al. (2002) & 0.970 & 0.971 & **0.986** & 0.973 & 0.979 & **0.657** & **0.978** \\ CDER\({}^{\ddagger}\)Leusch et al. (2006) & 0.972 & 0.980 & **0.990** & 0.984 & 0.980 & **0.664** & **0.982** \\ CharactTER\({}^{\ddagger}\)Wang et al. (2016) & **0.970** & 0.993 & 0.979 & 0.989 & **0.991** & **0.782** & 0.950 \\ chrF\({}^{\dagger}\)Popovic (2015) & 0.966 & 0.994 & 0.981 & 0.987 & **0.990** & 0.452 & 0.960 \\ chrF+\({}^{\dagger}\)Popović (2017) & 0.966 & 0.993 & 0.981 & 0.989 & 0.990 & 0.174 & 0.964 \\ ITER\({}^{\dagger}\)Panja and Naskar (2018) & 0.975 & 0.990 & 0.975 & **0.996** & 0.937 & **0.861** & **0.980** \\ METOF++\({}^{\ddagger}\)Shimanaka et al. (2018) & 0.945 & 0.991 & 0.978 & 0.971 & **0.995** & 0.864 & 0.962 \\ NIST\({}^{\dagger}\)Doddington (2002) & 0.954 & 0.984 & 0.983 & 0.975 & 0.973 & **0.970** & 0.968 \\ PER\({}^{\dagger}\) & **0.970** & 0.985 & **0.983** & 0.993 & 0.967 & 0.159 & 0.931 \\ RUSE\({}^{\ddagger}\)Shimanaka et al. (2018) & 0.981 & 0.997 & **0.990** & 0.991 & **0.988** & **0.853** & **0.981** \\ TER\({}^{\dagger}\)Snover et al. (2006) & **0.950** & 0.970 & **0.990** & 0.968 & 0.970 & 0.533 & 0.975 \\ UHH_TSKM\({}^{\ddagger}\)Domain and Menzel (2017) & 0.952 & 0.980 & **0.989** & 0.982 & 0.980 & 0.547 & **0.981** \\ WER\({}^{\dagger}\) & **0.951** & 0.961 & **0.991** & 0.961 & 0.968 & 0.041 & 0.975 \\ YISI-0\({}^{\ddagger}\)Lo (2019) & 0.956 & 0.994 & 0.975 & 0.978 & 0.988 & **0.954** & 0.957 \\ YISI-1\({}^{\ddagger}\)Li (2019) & 0.950 & 0.992 & 0.979 & 0.973 & **0.991** & **0.958** & 0.951 \\ YISI-1\({}_{\text{SRL}}\)\({}^{\ddagger}\)Li (2019) & 0.965 & 0.995 & 0.981 & 0.977 & **0.992** & **0.869** & 0.962 \\ \hline Prism-ref (This Work) & 0.988 & 0.995 & 0.971 & **0.998** & **0.995** & 0.730 & **0.989** \\ Prism-ref w/ ParaBank 2 (Contrastive) & **0.992** & 0.989 & 0.964 & **0.998** & **0.996** & **0.896** & 0.986 \\ LASER + LM (Contrastive) & **0.988** & 0.991 & 0.965 & 0.994 & 0.745 & 0.297 & 0.890 \\ Prism-src (This work) & **0.984** & 0.991 & 0.964 & 0.987 & 0.970 & **0.896** & 0.958 \\ LM & **0.986** & 0.970 & 0.954 & 0.898 & 0.951 & **0.891** & 0.972 \\ LASER & **0.978** & 0.986 & 0.953 & 0.984 & 0.489 & **0.968** & 0.591 \\ mBART (Contrastive) & 0.955 & 0.996 & **0.987** & 0.995 & 0.981 & 0.721 & **0.980** \\ \hline \hline \end{tabular}
\end{table}
Table 10: WMT18 System-level results, from English. n denotes number of MT systems. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. †:WMT18 Baseline Ma et al. (2018) ‡:WMT18 Metric Submission Ma et al. (2018)

\begin{table}
\begin{tabular}{l c r r r r r r} \hline \hline  & **es–en** & **de–cn** & **e–en** & **fi–en** & **ru–en** & **fr–cn** & **zh–cn** \\ n & 5 & 16 & 14 & 9 & 8 & 5 & 14 \\ \hline BEER\({}^{\ddagger}\)Stanojevic and Sima’an (2015) & 0.958 & 0.994 & **0.985** & 0.991 & 0.982 & 0.870 & **0.976** \\ BERTscoreZhang et al. (2019, 2020) & **0.990** & **0.999** & **0.990** & **0.998** & 0.935 & 0.499 & 0.956 \\ BLEND\({}^{\ddagger}\)Ma et al. (2017) & 0.973 & 0.991 & 0.985 & **0.994** & **0.993** & **0.801** & **0.976** \\ BLEU\({}^{\dagger}\)Papineni et al. (2002) & 0.970 & 0.971 & **0.986** & 0.973 & 0.979 & **0.657** & **0.978** \\ CDER\({}^{\ddagger}\)Leusch et al. (2006) & 0.972 & 0.980 & **0.990** & 0.984 & 0.980 & **0.664** & **0.982** \\ CharactTER\({}^{\ddagger}\)Wang et al. (2016) & **0.970** & 0.993 & 0.979 & 0.989 & **0.991** & **0.782** & 0.950 \\ chrRF\({}^{\dagger}\)Popvić (2015) & 0.966 & 0.994 & 0.981 & 0.987 & **0.990** & 0.452 & 0.960 \\ chrFR\({}^{\ddagger}\)Popović (2017) & 0.966 & 0.993 & 0.981 & 0.989 & 0.990 & 0.174 & 0.964 \\ ITER\({}^{\ddagger}\)Panja and Naskar (2018) & 0.975 & 0.990 & 0.975 & **0.996** & 0.937 & **0.861** & **0.980** \\ METOF++\({}^{\ddagger}\)Shimanaka et al. (2018) & 0.945 & 0.991 & 0.978 & 0.971 & **0.995** & 0.864 & 0.962 \\ NIST\({}^{\dagger}\)Doddington (2002) & 0.954 & 0.984 & 0.983 & 0.975 & 0.973 & **0.970** & 0.968 \\ PER\({}^{\dagger}\)PER\({}^{\dagger}\)PE



## Appendix E WMT 2019 Metric and QE as Metric Segment-Level Results

Table 11, Table 12, and Table 13 show segment-level metrics (excluding QE as a metric) results, for language pairs into, out of, and not including English, for the WMT 2019 MT metrics shared task, along with all baselines and submitted systems.

Table 14, Table 15, and Table 16 show segment-level QE as a metric results, for language pairs into, out of, and not including English, for the WMT 2019 MT metrics shared task, along with all baselines and submitted systems.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **de-en** & **fi-en** & **gu-en** & **kk-en** & **lt-en** & **ru-en** & **zh-en** \\ n & 85365 & 38307 & 31139 & 27094 & 21862 & 46172 & 31070 \\ \hline BEER\({}^{\ddagger}\)Stanojevic and Sima’an [2015] & 0.128 & 0.283 & 0.260 & 0.421 & 0.315 & 0.189 & 0.371 \\ BERTr\({}^{\dagger}\)Mathur et al. [2019] & 0.142 & 0.331 & 0.291 & 0.421 & 0.353 & 0.195 & 0.399 \\ BERTScoreZhang et al. [2019, 2020] & 0.176 & 0.345 & **0.320** & **0.432** & **0.381** & **0.223** & **0.430** \\ BLEURTSellam et al. [2020] & **0.204** & **0.367** & **0.311** & **0.447** & **0.387** & **0.228** & **0.423** \\ CharactTER\({}^{\ddagger}\)Wang et al. [2016] & 0.101 & 0.253 & 0.190 & 0.340 & 0.254 & 0.155 & 0.337 \\ chrF\({}^{\dagger}\)Popovic [2015] & 0.122 & 0.286 & 0.256 & 0.389 & 0.301 & 0.180 & 0.371 \\ chRF+\({}^{\dagger}\)Popović [2017] & 0.125 & 0.289 & 0.257 & 0.394 & 0.303 & 0.182 & 0.374 \\ EDI\({}^{\ddagger}\)Stanchev et al. [2019] & 0.120 & 0.281 & 0.264 & 0.392 & 0.298 & 0.176 & 0.376 \\ ESIM\({}^{\ddagger}\)Chen et al. [2017, Mathur et al., 2019] & 0.167 & 0.337 & 0.303 & **0.435** & 0.359 & 0.201 & 0.396 \\ HELEPOR_baseline\({}^{\ddagger}\)Han et al. [2012, 2013] & – & – & – & 0.372 & – & – & 0.339 \\ Meteor++.2.0(syntax)\({}^{\ddagger}\)Guo and Hu [2019] & 0.084 & 0.274 & 0.237 & 0.395 & 0.291 & 0.156 & 0.370 \\ Meteor++.2.0(syntax+copy)\({}^{\ddagger}\)Guo and Hu [2019] & 0.094 & 0.273 & 0.244 & 0.402 & 0.287 & 0.163 & 0.367 \\ PReP\({}^{\ddagger}\)Yoshimura et al. [2019] & 0.030 & 0.197 & 0.192 & 0.386 & 0.193 & 0.124 & 0.267 \\ sentBLEU\({}^{\dagger}\)Papineni et al. [2002] & 0.056 & 0.233 & 0.188 & 0.377 & 0.262 & 0.125 & 0.323 \\ WDMD\({}^{\ddagger}\)Chow et al. [2019] & 0.096 & 0.281 & 0.260 & 0.420 & 0.300 & 0.162 & 0.362 \\ YiSi-0\({}^{\ddagger}\)Ho [2019] & 0.117 & 0.271 & 0.263 & 0.402 & 0.289 & 0.178 & 0.355 \\ YiSi-1\({}^{\ddagger}\)Lo [2019] & 0.164 & 0.347 & **0.312** & **0.440** & **0.376** & **0.217** & **0.426** \\ YiSi-1.srl\({}^{\ddagger}\)Lo [2019] & **0.199** & 0.346 & 0.306 & **0.442** & **0.380** & **0.222** & **0.431** \\ \hline Prism-ref (This Work) & **0.204** & **0.357** & **0.313** & **0.434** & **0.382** & **0.225** & **0.438** \\ Prism-ref w/ ParaBank 2 (Contrastive) & 0.184 & **0.341** & **0.326** & 0.425 & **0.373** & 0.207 & **0.432** \\ LASER + LM (Contrastive) & 0.190 & 0.335 & **0.319** & 0.428 & **0.368** & 0.207 & 0.416 \\ LM & 0.083 & 0.253 & 0.165 & 0.120 & 0.281 & 0.130 & 0.210 \\ LASER & 0.151 & 0.301 & 0.305 & 0.420 & 0.325 & 0.193 & 0.397 \\ mBART (Contrastive) & 0.136 & 0.255 & 0.246 & 0.377 & 0.298 & 0.162 & 0.349 \\ \hline \hline \end{tabular}
\end{table}
Table 11: WMT19 Segment-level results, metrics (excludes QE as metric), to English. n denotes number of pairwise judgments. **Gold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. †:WMT19 Baseline Ma et al. (2019) ‡WMT19 Metric Submission Ma et al. (2019)



\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & **e-cs** & **de–fr** & **fr-de** \\ n & 35793 & 4862 & 1369 \\ \hline BEER\({}^{\ddagger}\)Stanojević and Sima’an, [2015] & 0.337 & 0.293 & 0.265 \\ BERtscoreZhang et al. (2019, 2020) & 0.352 & 0.325 & 0.274 \\ CharacTER\({}^{\ddagger}\)Wang et al. (2016) & 0.232 & 0.251 & 0.224 \\ chrF\({}^{\sharp}\)Popovic (2015) & 0.326 & 0.284 & 0.275 \\ chrF+\({}^{\dagger}\)Popović (2017) & 0.326 & 0.284 & 0.278 \\ EED\({}^{\dagger}\)Stanchev et al. (2019) & 0.345 & 0.301 & 0.267 \\ ESIM\({}^{\ddagger}\)Chen et al. (2017); Mathur et al. (2019) & 0.331 & 0.290 & 0.289 \\ hLEPORA\_baseline\({}^{\natural}\)Han et al. (2012, 2013) & 0.207 & 0.239 & – \\ sentBLEU\({}^{\dagger}\)Papineni et al. (2002) & 0.203 & 0.235 & 0.179 \\ YiS1-0\({}^{\ddagger}\)Lo (2019) & 0.331 & 0.296 & 0.277 \\ YiS1-1\({}^{\ddagger}\)Lo (2019) & 0.376 & 0.349 & 0.310 \\ YiS1-1\_srL\({}^{\ddagger}\)Lo (2019) & – & – & 0.299 \\ \hline Prism-ref (This Work) & **0.458** & **0.453** & **0.426** \\ LASER + LM (Contrastive) & 0.431 & 0.401 & **0.381** \\ LM & 0.294 & 0.235 & 0.138 \\ LASER & 0.397 & 0.352 & **0.348** \\ mBART (Contrastive) & 0.262 & 0.255 & 0.236 \\ \hline \hline \end{tabular}
\end{table}
Table 13: WMT19 Segment-level results, metrics (excludes QE as metric), non-English. n denotes number of pairwise judgments. **bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(\dagger\):WMT19 Baseline Ma et al. (2019) \(\ddagger\):WMT19 Metric Submission Ma et al. (2019)

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & **en–cs** & **en–de** & **en–fi** & **en–gu** & **en–kk** & **en–lt** & **en–ru** & **en–zh** \\ n & 27178 & 99840 & 31820 & 11355 & 18172 & 17401 & 24334 & 18658 \\ \hline BEER\({}^{\ddagger}\)Stanojević and Sima’an, [2015] & 0.443 & 0.316 & 0.514 & 0.537 & 0.516 & 0.441 & 0.542 & 0.232 \\ BERTscoreZhang et al. (2019, 2020) & 0.485 & 0.345 & 0.524 & **0.558** & **0.533** & 0.463 & 0.580 & 0.347 \\ CharacTER\({}^{\ddagger}\)Wang et al. (2016) & 0.349 & 0.264 & 0.404 & 0.500 & 0.351 & 0.311 & 0.432 & 0.094 \\ chrF\({}^{\dagger}\)Popovic (2015) & 0.455 & 0.326 & 0.514 & 0.534 & 0.479 & 0.446 & 0.539 & 0.301 \\ chrF+\({}^{\dagger}\)Popovic (2017) & 0.458 & 0.327 & 0.514 & 0.538 & 0.491 & 0.448 & 0.543 & 0.296 \\ ECD\({}^{\dagger}\)Stanchev et al. (2019) & 0.431 & 0.315 & 0.508 & **0.568** & 0.518 & 0.425 & 0.546 & 0.257 \\ ESMM\({}^{\ddagger}\)Chen et al. (2017); Mathur et al. (2019) & — & 0.329 & 0.511 & — & 0.510 & 0.428 & 0.572 & 0.339 \\ inLEPORA\_baseline\({}^{\ddagger}\)Han et al. (2012, 2013) & — & — & — & 0.463 & 0.390 & — & — & — \\ sentBLEU\({}^{\dagger}\)Papineni et al. (2002) & 0.367 & 0.248 & 0.396 & 0.465 & 0.392 & 0.334 & 0.469 & 0.270 \\ YiS1-0\({}^{\ddagger}\)Lo (2019) & 0.406 & 0.304 & 0.483 & 0.539 & 0.494 & 0.402 & 0.535 & 0.266 \\ YiS1-1\_1 [Lo, 2019) & 0.475 & 0.351 & 0.537 & **0.551** & **0.546** & 0.470 & 0.585 & **0.355** \\ YiS1-1\_sRL\({}^{\dagger}\)Lo (2019) & — & 0.368 & — & — & — & — & **0.361** \\ \hline Prism-ref This Work) & **0.582** & **0.427** & **0.591** & 0.313 & **0.531** & **0.558** & 0.584 & **0.376** \\ LASER + LM (Contrastive) & 0.535 & 0.401 & 0.568 & 0.306 & 0.408 & 0.503 & **0.640** & **0.356** \\ LM & 0.439 & 0.329 & 0.477 & 0.181 & 0.284 & 0.430 & 0.586 & 0.279 \\ LASER & 0.408 & 0.334 & 0.509 & 0.340 & 0.363 & 0.396 & 0.511 & 0.284 \\ mBART (Contrastive) & 0.345 & 0.302 & 0.401 & 0.528 & 0.462 & 0.365 & 0.443 & 0.280 \\ \hline \hline \end{tabular}
\end{table}
Table 12: WMT19 Segment-level results, metrics (excludes QE as metric results), from English. n denotes number of pairwise judgments. **bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(\dagger\):WMT19 Baseline Ma et al. (2019) \(\ddagger\):WMT19 Metric Submission Ma et al. (2019)



\begin{table}
\begin{tabular}{l r r r r r r} \hline \hline  & **de-cs** & **de–fr** & **fr-de** \\ n & 35793 & 4862 & 1369 \\ \hline ibm1-morpheme* (Popovic et al., 2011) & 0.048 & -0.013 & -0.053 \\ ibm1-pos4gram* (Popović et al., 2011) & — & – & -0.074 & -0.097 \\ YSiI-2* (Lo, 2019) & 0.199 & 0.186 & 0.066 \\ \hline Prism-src (This work) & **0.444** & **0.371** & **0.316** \\ \hline \hline \end{tabular}
\end{table}
Table 16: WMT’19 Segment-level results, QE as a metric, non-English. n denotes number of pairwise judgments. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(*\):WMT’19 QE-as-Metric Submission (Fonseca et al., 2019)

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & **de-en** & **fi–en** & **gu-en** & **kk–en** & **lt-én** & **ru–en** & **zh-en** \\ n & 85365 & 38307 & 31139 & 27094 & 21862 & 46172 & 31070 \\ \hline ibm1-morphheme* (Popovic et al., 2011) & -0.074 & 0.009 & – & – & 0.069 & – & – \\ ibm1-pos4gram* (Popović et al., 2011) & -0.153 & – & – & – & – & – & – \\ LASIM* & -0.024 & – & – & – & – & 0.022 & – \\ LP* & -0.096 & – & – & – & – & -0.035 & – \\ UNI* (Yankovskaya et al., 2019) & 0.022 & 0.202 & – & – & – & 0.084 & – \\ UNI+* (Yankovskaya et al., 2019) & 0.015 & 0.211 & – & – & – & 0.089 & – \\ YiSi-2* (Lo, 2019) & 0.068 & 0.126 & -0.001 & 0.096 & 0.075 & 0.053 & 0.253 \\ YiSi-2_sRL* (Lo, 2019) & 0.068 & – & – & – & – & – & 0.246 \\ \hline Prism-src (This work) & **0.109** & **0.300** & **0.102** & **0.391** & **0.356** & **0.178** & **0.336** \\ \hline \hline \end{tabular}
\end{table}
Table 14: WMT’19 Segment-level results, QE as a metric, to English. n denotes number of pairwise judgments. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(*\):WMT’19 QE-as-Metric Submission (Fonseca et al., 2019)

\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline  & **en–cs** & **en–de** & **en–fi** & **en–gu** & **en-kk** & **en–lt** & **en–ru** & **en-zh** \\ n & 27178 & 99840 & 31820 & 11355 & 18172 & 17401 & 24334 & 18658 \\ \hline ibm1-morphheme* (Popovic et al., 2011) & -0.135 & -0.003 & -0.005 & – & – & -0.165 & – & – \\ ibm1-pos4gram* (Popovic et al., 2011) & \(-\) -0.123 & – & – & – & – & – & – \\ LASIM* & \(-\) 0.147 & – & – & – & – & -0.240 & – \\ LP* & \(-\) -0.119 & – & – & – & – & -0.158 & – \\ UNI* (Yankovskaya et al., 2019) & 0.060 & 0.129 & 0.351 & – & – & – & 0.226 & – \\ UNI+* (Yankovskaya et al., 2019) & \(-\) & – & – & – & – & 0.222 & – \\ USFD* (Iwe et al., 2018) & \(-\) & -0.029 & – & – & – & 0.136 & – \\ USFD-TL* (Ive et al., 2018) & \(-\) & -0.037 & – & – & – & 0.191 & – \\ YiSI-2* (Lo, 2019) & 0.069 & 0.212 & 0.239 & 0.147 & 0.187 & 0.003 & -0.155 & 0.044 \\ YiSi-2*L (Lo, 2019) & \(-\) & 0.236 & – & – & – & – & – & 0.034 \\ \hline Prism-src (This work) & **0.470** & **0.402** & **0.555** & **0.215** & **0.507** & **0.499** & **0.486** & **0.287** \\ \hline \hline \end{tabular}
\end{table}
Table 15: WMT’19 Segment-level results, QE as a metric, from English. n denotes number of pairwise judgments. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(*\):WMT’19 QE-as-Metric Submission (Fonseca et al., 2019)

WMT 2019 System-Level results for Top 4 Systems

Table 17 Table 18, and Table 19 show system-level results for just the top 4 systems, for language pairs into, out of, and not including English, for WMT 2019. We show statistical significance following the shared task but note it appears extremely noisy.

\begin{table}
\begin{tabular}{l c r r r r r r} \hline \hline  & **de-en** & **fl–en** & **gu-en** & **kk–en** & **lt–cn** & **ru–en** & **zh–en** \\ n & & 4 & 4 & 4 & 4 & 4 & 4 \\ \hline BEER\({}^{\ddagger}\)Stanojevic and Sima’an [35] & -0.760 & **0.065** & **0.981** & **0.957** & **0.423** & -0.122 & -0.625 \\ BERTr\({}^{\ddagger}\)Mathur et al. [19] & 0.251 & 0.430 & 0.966 & 0.864 & 0.518 & **0.505** & **0.402** \\ BERtscoreZhang et al. [2019], 2020 & 0.272 & **0.683** & **0.913** & **0.897** & **0.753** & **0.456** & -0.220 \\ BLEU\({}^{\dagger}\)Papineni et al. [2002] & **-0.822** & **-0.275** & **0.966** & **0.958** & **0.625** & **-0.356** & **-0.694** \\ BLEURTSellam et al. [2020] & **0.953** & **0.714** & **0.881** & 0.929 & **0.841** & **0.522** & 0.660 \\ CDER\({}^{\dagger}\)Leusch et al. [2006] & -0.740 & **-0.214** & **0.940** & 0.948 & 0.389 & **-0.108** & -0.611 \\ CharacterR\({}^{\dagger}\)Wang et al. [2016] & **-0.664** & **-0.079** & **0.980** & **0.924** & 0.386 & **0.052** & -0.092 \\ chrR\({}^{\dagger}\)Popovic [2015] & -0.610 & **0.170** & **0.986** & 0.893 & 0.377 & -0.043 & -0.147 \\ chrF+\({}^{\prime}\)Popović [2017] & -0.612 & **0.157** & 0.982 & **0.886** & 0.341 & -0.019 & -0.093 \\ EED\({}^{\ddagger}\)Stanchev et al. [2019] & -0.503 & **0.125** & **0.978** & **0.904** & 0.323 & **0.033** & –0.06 \\ ESIM\({}^{\ddagger}\)Chen et al. [2017]; Mathur et al. [2019] & 0.895 & 0.740 & **0.847** & **0.965** & 0.896 & **0.534** & **0.819** \\ hLEPORabaseline\({}^{\ddagger}\)Han et al. [2012, 2013] & \(--\) & \(--\) & \(0.816\) & \(--\) & \(--\) & **0.312** \\ hLEPORb baselines\({}^{\ddagger}\)Han et al. [2012, 2013] & \(--\) & \(--\) & \(--\) & \(0.816\) & \(0.257\) & \(--\) & **0.312** \\ METeor++.2.0(syntax)\({}^{\ddagger}\)Guo and Hu [2019] & -0.591 & 0.349 & **0.978** & **0.912** & 0.413 & **0.024** & -0.214 \\ Meteor++.2.0(syntax+copy)\({}^{\ddagger}\)Guo and Hu [2019] & -0.587 & 0.399 & **0.980** & 0.888 & 0.413 & **0.051** & -0.17 \\ NIST\({}^{\dagger}\)Doddington [2002] & **-0.82** & **0.111** & **0.963** & 0.913 & **0.746** & **-0.458** & **-0.906** \\ PER\({}^{\dagger}\)Predsonitz et al. [2019] & **-0.787** & **0.232** & 0.945 & 0.731 & 0.086 & **-0.081** & **0.730** \\ PREPI\({}^{\dagger}\)Yoshimura et al. [2019] & **-0.981** & 0.754 & 0.976 & **0.863** & 0.171 & **-0.357** & **-0.927** \\ saceBLEU.BLEU\({}^{\dagger}\)Post [2018] & -0.823 & **-0.333** & **0.966** & **0.958** & 0.426 & **-0.217** & **-0.694** \\ sacebBLEU.chrF\({}^{\dagger}\)Post [2018] & -0.633 & **0.113** & 0.954 & **0.875** & 0.311 & -0.094 & 0.347 \\ TER\({}^{\dagger}\)Snover et al. [2006] & -0.798 & **0.032** & 0.942 & **0.963** & **0.585** & **-0.137** & -0.845 \\ WER\({}^{\dagger}\) & -0.816 & **-0.125** & **0.940** & **0.958** & **0.621** & **-0.153** & **-0.859** \\ WMDO\({}^{\ddagger}\)Chow et al. [2019] & -0.711 & 0.344 & **0.943** & **0.921** & 0.290 & **0.114** & -0.352 \\ YISi-0\({}^{\ddagger}\)Lo [2019] & -0.714 & **0.074** & **0.991** & **0.946** & **0.540** & -0.079 & -0.663 \\ YISI-1\({}^{\ddagger}\)Lo [2019] & 0.045 & **0.610** & **0.962** & **0.887** & **0.552** & **0.365** & -0.067 \\ YiSi-1\({}^{\ddagger}\)Lo [2019] & 0.081 & **0.580** & **0.959** & **0.874** & **0.560** & 0.342 & -0.069 \\ \hline ibmi-morphheme\({}^{\circ}\)Popvić et al. [2011] & **-0.643** & **0.065** & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) \\ bMI-pos4gram\({}^{\dagger}\)Popović et al. [2011] & **-0.831** & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) \\ LASIM\({}^{\ddagger}\)LAPS\({}^{\ddagger}\)LP.1\({}^{*}\) & **0.777** & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) \\ UNIT\({}^{\star}\)Yankovskaya et al. [2019] & **0.703** & **0.830** & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) \\ UMIT\({}^{\star}\)Yankovskaya et al. [2019] & **0.796** & **0.791** & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) \\ YISi-2\({}^{\rightarrow}\)Lo [2019] & **-0.809** & **0.780** & **-0.125** & **0.834** & **-0.362** & **-0.325** & **-0.889** \\ YISi-2\({}^{\ast}\)Li [2019] & **-0.749** & \(--\) & \(--\) & \(--\) & \(--\) & \(--\) & \(-0.83\) \\ \hline Prism-ref (This Work) & 0.401 & **0.719** & **0.896** & 0.796 & **0.877** & **0.431** & 0.523 \\ Prism-ref w/ ParaBank 2 (Contrastive) & **0.957** & **0.788** & **0.871** & **0.759** & **0.939** & **0.625** & **0.899** \\ LASER + LM (Contrastive) & **0.957** & 0.768 & 0.867 & **0.870** 



\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline  & **en–cs** & **en–de** & **en–fi** & **en–gu** & **en-kk** & **cn–lt** & **en–ru** & **en–zh** \\ n & 4 & 4 & 4 & 4 & 4 & 4 & 4 & 4 \\ \hline BEER\({}^{\ddagger}\)Stanojevic and Sima’an [2015] & 0.872 & **-0.801** & **0.960** & 0.899 & 0.226 & **0.888** & **0.961** & **0.992** \\ BERTscoreZhang et al. [2019, 2020] & **0.868** & -0.722 & **0.859** & **0.922** & 0.288 & **0.955** & **0.953** & 0.982 \\ BLEU\({}^{\dagger}\)Papineni et al. [2002] & 0.930 & **-0.37** & 0.898 & **0.860** & 0.181 & **0.925** & 0.753 & 0.987 \\ CDER\({}^{\ddagger}\)Leusch et al. [2006] & **0.946** & **-0.975** & 0.837 & **0.900** & **-0.011** & **0.880** & **0.917** & **0.986** \\ CharactTER\({}^{\ddagger}\)Wang et al. [2016] & 0.828 & **-0.777** & **0.887** & **0.902** & 0.295 & **0.675** & **0.974** & **0.997** \\ chrF\({}^{\dagger}\)Popovic [2015] & 0.799 & -0.590 & **0.936** & **0.926** & 0.277 & **0.901** & **0.954** & 0.987 \\ chrF+\({}^{\dagger}\)Popovic [2017] & 0.816 & -0.605 & **0.921** & **0.923** & 0.283 & **0.858** & 0.940 & **0.996** \\ EED\({}^{\ddagger}\)Stanchev et al. [2019] & 0.825 & -0.552 & **0.939** & **0.913** & 0.267 & **0.921** & 0.961 & **0.997** \\ ESIM\({}^{\ddagger}\)Chen et al. [2017, 2019] & – & **-0.796** & **0.957** & – & 0.418 & **0.997** & **0.986** & 0.987 \\ HLEPORa\_baseline\({}^{\ddagger}\)Han et al. [2012, 2013] & – & – & – & 0.915 & **0.062** & — & — & — \\ NIST\({}^{\dagger}\)Doddington [2002] & **0.946** & **-0.233** & **0.971** & **0.893** & 0.082 & **0.988** & 0.724 & 0.979 \\ PER\({}^{\dagger}\) & 0.916 & **-0.995** & **0.850** & **0.887** & **-0.260** & 0.390 & **0.911** & **0.980** \\ sacreBLEU.BELEU\({}^{\dagger}\)Post [2018] & **0.970** & **-0.976** & 0.845 & 0.859 & 0.181 & 0.638 & **0.878** & 0.962 \\ sarceBLEU.chrF\({}^{\dagger}\)Post [2018] & 0.907 & **-0.816** & **0.921** & **0.902** & 0.239 & **0.980** & **0.970** & **0.963** \\ TER\({}^{\dagger}\)Snover et al. [2006] & **0.969** & **-0.989** & **0.889** & **0.874** & **-0.060** & **0.988** & **0.895** & 0.984 \\ WER\({}^{\dagger}\) & **0.973** & **-0.993** & 0.876 & 0.868 & **-0.058** & **0.973** & **0.894** & 0.987 \\ YiSt-0\({}^{\ddagger}\)Lo [2019] & 0.879 & -0.796 & **0.975** & **0.920** & 0.196 & 0.787 & **0.940** & 0.982 \\ YiSt-1\({}^{\ddagger}\)Lo [2019] & 0.847 & -0.220 & **0.976** & **0.917** & 0.342 & **0.838** & **0.963** & **0.990** \\ YiSi-1.srl\({}^{\ddagger}\)Lo [2019] & – & -0.378 & – & – & – & – & – & **0.994** \\ \hline ibmI-morpheme\({}^{*}\)Popović et al. [2011] & **-0.771** & **-0.425** & **0.430** & – & – & **0.969** & — & — \\ ibmI-pos4gram\({}^{*}\)Popović et al. [2011] & – & -0.502 & – & – & – & – & — \\ LASIM\({}^{\star}\) & – & -0.914 & – & – & – & 0.223 & — \\ LP.1\({}^{*}\) & – & 0.949 & – & – & – & -0.407 & — \\ UNIT\({}^{*}\)Yankovskaya et al. [2019] & 0.587 & **-0.96** & 0.637 & – & – & **0.655** & — \\ UNI\({}^{\star}\)Yankovskaya et al. [2019] & – & – & – & – & – & – & 0.644 & — \\ USFD\({}^{*}\)Ive et al. [2018] & – & **-0.729** & – & – & – & **0.985** & — \\ VSFD-TL\({}^{*}\)Ive et al. [2018] & – & -0.390 & – & – & – & 0.698 & — \\ YiSI-2\({}^{*}\)Lo [2019] & **0.793** & -0.933 & **-0.991** & -0.389 & **0.851** & -0.504 & 0.075 & 0.983 \\ YiSi-2\({}^{*}\)Lo [2019] & – & -0.915 & – & – & – & – & — & **0.991** \\ \hline Prism-ref (This Work) & **0.952** & **0.278** & 0.886 & **0.863** & 0.693 & 0.862 & **0.975** & **0.966** \\ LASER + LM (Contrastive) & **0.961** & **0.377** & 0.903 & **0.509** & 0.605 & **0.743** & 0.962 & **0.985** \\ Prism-src (This work) & **0.973** & -0.408 & 0.765 & **-0.703** & **0.833** & -0.003 & 0.708 & **0.863** \\ LM & **0.833** & **0.425** & 0.763 & **-0.712** & **0.953** & **0.633** & 0.916 & **0.846** \\ LASER & 0.851 & **0.246** & **0.983** & **0.568** & **0.328** & **0.263** & **0.995** & 0.988 \\ mBART (Contrastive) & 0.936 & -0.834 & **0.966** & 0.912 & **0.224** & **0.946** & **0.968** & **0.986** \\ \hline \hline \end{tabular}
\end{table}
Table 18: WMT19 System-level results, from English for the top 4 systems (as judged by humans) for each language pair. n denotes number of MT systems. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. †:WMT19 BaselineMa et al. [2019] ‡:WMT19Metric Submission Ma et al. [2019]



\begin{table}
\begin{tabular}{l r r r} \hline \hline  & **de–cs** & **de–fr** & **fr-de** \\ n & 4 & 4 & 4 \\ \hline BEER\({}^{\ddagger}\)Stanojevic and Sima’an, 2015 & **0.961** & **0.590** & 0.978 \\ BERTscoreZhang et al. (2019, 2020) & **0.976** & **0.707** & **0.973** \\ BLEU\({}^{\dagger}\)Papineni et al. (2002) & 0.812 & 0.495 & **0.983** \\ CDER\({}^{\dagger}\)Leusch et al. (2006) & **0.860** & 0.544 & 0.959 \\ CharacTER\({}^{\ddagger}\)Wang et al. (2016) & **0.871** & **0.626** & 0.963 \\ chrF\({}^{\dagger}\)Popovic (2015) & **0.920** & 0.531 & **0.952** \\ chrF+\({}^{\dagger}\)Popović (2017) & 0.909 & 0.522 & 0.946 \\ EED\({}^{\ddagger}\)Stanchev et al. (2019) & **0.873** & 0.582 & 0.945 \\ ESIM\({}^{\ddagger}\)Chen et al. (2017); Mathur et al. (2019) & **0.977** & **0.702** & **0.991** \\ hLEPORA_baseline\({}^{\ddagger}\)Han et al. (2012, 2013) & 0.771 & 0.314 & \\ hLEPORb_baseline\({}^{\ddagger}\)Han et al. (2012, 2013) & **0.754** & 0.314 & \\ NIST\({}^{\dagger}\)Doddington (2002) & 0.754 & 0.561 & **0.990** \\ PER\({}^{\dagger}\) & 0.913 & 0.401 & **0.990** \\ sacreBLEU.BLEU\({}^{\dagger}\)Post (2018) & **0.888** & 0.495 & 0.958 \\ sacreBLEEU.chrF\({}^{\dagger}\)Post (2018) & **0.964** & 0.575 & 0.920 \\ TER\({}^{\dagger}\)Snover et al. (2006) & **0.999** & 0.541 & **0.989** \\ WER\({}^{\dagger}\) & **0.997** & **0.566** & **0.991** \\ YiSi-O\({}^{\ddagger}\)Lo (2019) & 0.838 & **0.655** & 0.961 \\ YiSi-I\({}^{\ddagger}\)Lo (2019) & **0.967** & 0.677 & 0.967 \\ ViSi-I\({}^{\ddagger}\)Lo (2019) & – & – & **0.974** \\ \hline ibm1-morheme\({}^{*}\)Popvić et al. (2011) & 0.645 & **-0.885** & **-0.339** \\ ibm1-pos4gram\({}^{*}\)Popović et al. (2011) & – & **-0.106** & -0.33 \\ YiSi-2\({}^{*}\)Lo (2019) & 0.368 & 0.209 & **-0.687** \\ \hline Prism-refThisWork) & **0.968** & **0.648** & **0.998** \\ LASER + LM (Contrastive) & **0.947** & **0.774** & 0.975 \\ Prism-srcThiswork) & **0.903** & **0.600** & **0.181** \\ LM & 0.336 & **0.770** & -0.903 \\ LASER & **0.552** & **0.713** & 0.953 \\ mBART (Contrastive) & 0.806 & **0.615** & **0.972** \\ \hline \hline \end{tabular}
\end{table}
Table 19: WMT19 System-level results, non-English for the top 4 systems (as judged by humans) for each language pair. n denotes number of MT systems. **Gold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. †:WMT19 BaselineMa et al. (2019) ‡:WMT19 Metric Submission Ma et al. (2019) \(*\):WMT19 QE-as-Metric Submission Fonseca et al. (2019)

WMT 2019 Metric and QE as Metric System-Level Results

Table 20, Table 21, and Table 22, show system-level results, for metrics (excludes QE as metric) for language pairs into, out of, and not including English, for the WMT 2019 MT metrics shared task, along with all baselines and submitted systems.

Table 23, Table 24, and Table 25, show system-level results, for QE as metric, for language pairs into, out of, and not including English, for the WMT 2019 MT metrics shared task, along with all baselines and submitted systems.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **de–en** & **fi–en** & **gu–en** & **kk–en** & **lt–en** & **ru–en** & **zh–en** \\ n & 16 & 12 & 11 & 11 & 11 & 14 & 15 \\ \hline BEER\({}^{t}\)[Stanojevic and Sima’an, 2015] & 0.906 & **0.993** & 0.952 & 0.986 & 0.947 & 0.915 & 0.942 \\ BERT\({}^{t}\)[Mathur et al., 2019] & **0.926** & 0.984 & 0.938 & 0.990 & 0.948 & **0.971** & 0.974 \\ BERTScore[Zhang et al., 2019, 2020] & **0.949** & **0.987** & **0.981** & 0.980 & 0.962 & 0.921 & 0.983 \\ BLEU\({}^{t}\)[Papineni et al., 2002] & 0.849 & 0.982 & 0.834 & 0.946 & 0.961 & 0.879 & 0.899 \\ BLEURT[Sellam et al., 2020] & **0.940** & 0.978 & 0.878 & 0.993 & **0.991** & **0.977** & 0.984 \\ CDER\({}^{t}\)[Leusch et al., 2006] & 0.890 & **0.988** & 0.876 & 0.967 & 0.975 & 0.892 & 0.917 \\ CharaCTER\({}^{t}\)[Wang et al., 2016] & 0.898 & **0.990** & 0.922 & 0.953 & 0.955 & 0.923 & 0.943 \\ chrF[Popovic, 2015] & **0.917** & **0.992** & 0.955 & 0.978 & 0.940 & 0.945 & 0.956 \\ chrF+[Popovic, 2017] & **0.916** & **0.992** & 0.947 & 0.976 & 0.940 & 0.945 & 0.956 \\ EED\({}^{t}\)[Stanchev et al., 2019] & 0.903 & **0.994** & 0.976 & 0.980 & 0.929 & 0.950 & 0.949 \\ ESIM\({}^{t}\)[Chen et al., 2017; Mathur et al., 2019] & **0.941** & 0.971 & 0.885 & 0.986 & **0.989** & **0.968** & **0.988** \\ hLEPOR\_baseline\({}^{t}\)[Han et al., 2012, 2013] & – & – & – & 0.975 & – & – & 0.947 \\ HLEPOR\_baseline\({}^{t}\)[Han et al., 2012, 2013] & – & — & – & 0.975 & 0.906 & – & 0.947 \\ Meteor++.2.0(syntax)\({}^{t}\)[Guo and Hu, 2019] & 0.887 & **0.995** & 0.990 & 0.974 & 0.928 & **0.950** & 0.948 \\ Meteor++.2.0(syntax+copy)\({}^{t}\)[Guo and Hu, 2019] & 0.896 & **0.995** & 0.900 & 0.971 & 0.927 & **0.952** & 0.952 \\ NIST\({}^{t}\)[Doddington, 2002] & 0.813 & 0.986 & 0.930 & 0.942 & 0.944 & 0.925 & 0.921 \\ PER\({}^{t}\)[Joshimura et al., 2019] & 0.883 & **0.991** & 0.910 & 0.737 & 0.947 & 0.922 & 0.952 \\ PREP\({}^{t}\)[Yoshimura et al., 2019] & 0.575 & 0.614 & 0.773 & 0.776 & 0.494 & 0.782 & 0.592 \\ sacreBLEUEUE\({}^{t}\)[Post, 2018] & 0.813 & 0.985 & 0.834 & 0.946 & 0.955 & 0.873 & 0.903 \\ sacreBLEEU.chRF\({}^{t}\)[Post, 2018] & 0.910 & **0.990** & 0.952 & 0.969 & 0.935 & 0.919 & 0.955 \\ TER \({}^{t}\)[Snover et al., 2006] & 0.874 & **0.984** & 0.890 & 0.799 & 0.960 & 0.917 & 0.840 \\ WERT\({}^{t}\)[Ward et al., 2019] & 0.863 & 0.983 & 0.861 & 0.793 & 0.961 & 0.911 & 0.820 \\ WWDO\({}^{t}\)[Chow et al., 2019] & 0.872 & **0.987** & 0.983 & **0.998** & 0.900 & 0.942 & 0.943 \\ YiSi-0\({}^{t}\)[Lo, 2019] & 0.902 & **0.993** & **0.993** & 0.991 & 0.927 & **0.958** & 0.937 \\ YiSi-1\({}^{t}\)[Lo, 2019] & **0.949** & **0.989** & 0.924 & 0.994 & 0.981 & **0.979** & 0.979 \\ YiSi-1\({}_{s}\)srL\({}^{t}\)[Lo, 2019] & **0.950** & **0.989** & 0.918 & 0.994 & 0.983 & **0.978** & 0.977 \\ \hline Prism-ref (This Work) & **0.954** & 0.983 & 0.764 & **0.998** & **0.995** & 0.914 & **0.992** \\ Prism-ref w/ ParaBank 2 (Contrastive) & **0.949** & 0.979 & 0.925 & 0.993 & 0.981 & 0.948 & **0.994** \\ LASER + LM (Contrastive) & **0.938** & 0.974 & 0.974 & **0.997** & **0.996** & 0.940 & 0.988 \\ mBART (Contrastive) & 0.906 & **0.991** & 0.949 & 0.974 & 0.917 & 0.880 & 0.956 \\ \hline \hline \end{tabular}
\end{table}
Table 20: WMT19 System-level results, to English. n denotes number of MT systems. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. †:WMT19 Baseline [Ma et al., 2019] ‡:WMT19 Metric Submission [Ma et al., 2019 



\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline  & **en-cs** & **en-de** & **en–fi** & **en–gu** & **en–kk** & **en-lt** & **en–ru** & **en–zh** \\ n & 11 & 22 & 12 & 11 & 11 & 12 & 12 & 12 \\ \hline BEER\({}^{\dagger}\)(Stanojević and Sima’an, 2015) & **0.990** & 0.983 & **0.989** & 0.829 & 0.971 & **0.982** & 0.977 & 0.803 \\ BERTscore (Zhang et al., 2019, 2020) & 0.981 & **0.990** & 0.970 & **0.922** & 0.981 & 0.978 & **0.989** & 0.925 \\ BLEU\({}^{\dagger}\)(Papineni et al., 2002) & 0.897 & 0.921 & **0.969** & 0.737 & 0.852 & **0.989** & 0.986 & 0.901 \\ CDER\({}^{\ddagger}\)(Leusch et al., 2006) & 0.985 & 0.973 & **0.978** & 0.840 & 0.927 & **0.985** & **0.993** & 0.905 \\ CHARCTE\({}^{\ddagger}\)(Wang et al., 2016) & **0.994** & **0.986** & 0.968 & **0.910** & 0.936 & 0.954 & **0.985** & 0.862 \\ chrF\({}^{\dagger}\)(Popovic, 2015) & 0.990 & 0.979 & **0.986** & **0.841** & **0.972** & **0.981** & 0.943 & 0.880 \\ chrF+\({}^{\dagger}\)(Popović, 2017) & **0.991** & 0.981 & **0.986** & 0.848 & **0.974** & **0.982** & 0.950 & 0.879 \\ EED\({}^{\ddagger}\)(Stanchev et al., 2019) & **0.993** & **0.985** & **0.987** & 0.897 & **0.979** & 0.975 & 0.967 & 0.856 \\ ESIM\({}^{\ddagger}\)(Chen et al., 2017; Mathur et al., 2019) & – & **0.991** & 0.957 & – & **0.980** & **0.989** & **0.989** & 0.931 \\ hLEPORa_baseline\({}^{\ddagger}\)(Han et al., 2012, 2013) & – & – & — & 0.841 & 0.968 & – & – & – \\ NIST\({}^{\dagger}\)(Doddington, 2002) & 0.896 & 0.321 & 0.971 & 0.786 & 0.930 & **0.993** & **0.988** & 0.884 \\ PER\({}^{\dagger}\) & 0.976 & 0.970 & **0.982** & 0.839 & 0.921 & 0.985 & 0.981 & 0.895 \\ sacreBLEU.BLEU\({}^{\dagger}\)(Post, 2018) & **0.994** & 0.969 & 0.966 & 0.736 & 0.852 & **0.986** & 0.977 & 0.801 \\ sacREBLEU.chRF\({}^{\dagger}\)(Post, 2018) & 0.983 & 0.976 & 0.980 & 0.841 & **0.967** & 0.966 & **0.985** & 0.796 \\ TER\({}^{\dagger}\)(Snover et al., 2006) & 0.980 & 0.969 & **0.981** & **0.865** & 0.940 & **0.994** & **0.995** & 0.856 \\ WER\({}^{\dagger}\) & 0.982 & 0.966 & **0.980** & **0.861** & 0.939 & **0.991** & **0.994** & 0.875 \\ YiSi-\(0^{\ddagger}\)(Lo, 2019) & **0.992** & 0.985 & **0.987** & 0.863 & 0.974 & 0.974 & 0.953 & 0.861 \\ YiSi-\(1^{\ddagger}\)(Lo, 2019) & 0.962 & **0.991** & 0.971 & **0.909** & **0.985** & 0.963 & **0.992** & **0.951** \\ YiSi-\(1_{\ddagger}\)(Lo, 2019) & – & **0.991** & – & – & — & — & — & **0.948** \\ \hline Prism-ref (This Work) & 0.958 & **0.988** & 0.949 & 0.624 & **0.978** & 0.937 & 0.918 & 0.898 \\ LASER + LM (Contrastive) & 0.962 & **0.989** & 0.957 & 0.775 & 0.969 & 0.958 & **0.987** & **0.950** \\ mBART (Contrastive) & **0.987** & 0.988 & **0.982** & **0.917** & **0.981** & 0.965 & 0.978 & 0.866 \\ \hline \hline \end{tabular}
\end{table}
Table 21: WMT19 System-level results, from English. n denotes number of MT systems. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. \(~{}\dagger\):WMT19 Baseline [Ma et al., 2019]\(\ddagger\):WMT19 Metric Submission [Ma et al., 2019]

\begin{table}
\begin{tabular}{l c r r r} \hline \hline  & & **de-cs** & **de-fr** & **fr-de** \\ n & 11 & 11 & 10 \\ \hline BEER\({}^{\ddagger}\)(Stanojević and Sima'an, 2015) & **0.978** & **0.941** & 0.848 \\ BERTScore (Zhang et al., 2019, 2020) & 0.969 & **0.971** & **0.899** \\ BLEU\({}^{\dagger}\)(Papineni et al., 2002) & 0.941 & 0.891 & 0.864 \\ CDER\({}^{\dagger}\)(Leusch et al., 2006) & 0.864 & **0.949** & 0.852 \\ CharacTER\({}^{\ddagger}\)(Wang et al., 2016) & 0.965 & 0.928 & 0.849 \\ chrF\({}^{\dagger}\)(Popović, 2015) & **0.974** & 0.931 & 0.864 \\ chrF\({}^{\ddagger}\)(Popovic, 2017) & 0.972 & 0.936 & 0.848 \\ EEID\({}^{\ddagger}\)(Stanchev et al., 2019) & **0.982** & **0.940** & 0.851 \\ ESIM\({}^{\ddagger}\)(Chen et al., 2017; Mathur et al., 2019) & **0.980** & **0.950** & **0.942** \\ HLEPORA\_baseline\({}^{\ddagger}\)(Han et al., 2012, 2013) & 0.941 & 0.814 & – \\ HELEPOR\_baseline\({}^{\ddagger}\)(Han et al., 2012, 2013) & 0.959 & 0.814 & 0.862 \\ NIST\({}^{\dagger}\)(Doddington, 2002) & 0.954 & **0.916** & **0.899** \\ PER\({}^{\dagger}\) & 0.875 & 0.857 & 0.869 \\ sacreBLEU.BLEU\({}^{\dagger}\)(Post, 2018) & 0.869 & 0.891 & 0.882 \\ sacrBELU.chrF\({}^{\dagger}\)(Post, 2018) & **0.975** & **0.952** & **0.895** \\ TER\({}^{\dagger}\)(Snover et al., 2006) & 0.890 & **0.956** & **0.894** \\ WER\({}^{\dagger}\) & 0.872 & **0.956** & 0.820 \\ YiSI-\(0^{\ddagger}\)(Lo, 2019) & **0.978** & **0.952** & **0.908** \\ YiSi-\(1^{\ddagger}\)(Lo, 2019) & 0.973 & **0.969** & **0.912** \\ \hline Prism-ref (This Work) & 0.976 & 0.936 & **0.911** \\ LASER + LM (Contrastive) & **0.990** & **0.935** & **0.924** \\ mBART (Contrastive) & 0.96



\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline  & **e-es** & **en–de** & **en–fi** & **en–gu** & **eu–kk** & **en–lt** & **en–ru** & **enh–zh** \\ n & 11 & 22 & 12 & 11 & 11 & 12 & 12 & 12 \\ \hline ibM1-morpheme* (Popovic et al., 2011) & **-0.871** & 0.870 & 0.084 & – & – & **-0.81** & – & – \\ ibM1-pos4gram* (Popović et al., 2011) & \(-\) & 0.393 & – & – & – & – & – & – \\ LASIM* & — & 0.871 & – & – & – & – & – & **-0.823** & – \\ LP.1* & — & -0.569 & – & – & – & – & -0.661 & – \\ UNIT* (Yankovskaya et al., 2019) & 0.028 & 0.841 & **0.907** & – & – & – & **0.919** & – \\ UNI+* (Yankovskaya et al., 2019) & – & – & – & – & – & **0.918** & – \\ USFD* (Ive et al., 2018) & – & -0.224 & – & – & – & **0.857** & – \\ USFD-TL* (Ive et al., 2018) & — & -0.091 & – & – & – & 0.771 & – \\ Y1S1-2* (Lo, 2019) & 0.324 & 0.924 & 0.696 & **0.314** & 0.339 & 0.055 & -0.766 & -0.097 \\ YiS1-2* (Lo, 2019) & – & 0.936 & – & – & – & – & -0.118 \\ \hline Prism-src (This work) & **0.865** & **0.976** & **0.933** & **0.444** & **0.959** & **0.908** & **0.822** & **0.793** \\ \hline \hline \end{tabular}
\end{table}
Table 24: WMT19 System-level results, QE as a metric, from English. n denotes number of MT systems. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. *:WMT19 QE-as-Metric Submission (Fonseca et al., 2019)

\begin{table}
\begin{tabular}{l c r r r r r r r} \hline \hline n & \multicolumn{2}{c}{**de–cs**} & \multicolumn{2}{c}{**de–fr**} & \multicolumn{1}{r}{**fr–de**} \\ n & 11 & 11 & 10 \\ \hline ibm1-morpheme* (Popovic et al., 2011) & 0.355 & -0.509 & **-0.625** \\ ibM1-pos4gram* (Popovic et al., 2011) & \(-\) & 0.085 & **-0.478** & **-0.53** \\ YSiS1-2* (Lo, 2019) & 0.606 & **0.721** & **-0.53** \\ \hline Prism-src (This work) & **0.973** & **0.889** & **0.739** \\ \hline \hline \end{tabular}
\end{table}
Table 25: WMT19 System-level results, QE as a metric, non-English. n denotes number of MT systems. **Bold** denotes top scoring method and any other methods with whose 95% confidence interval overlaps with that of a top method. *:WMT19 QE-as-Metric Submission (Fonseca et al., 2019)