BERTScore: Evaluating Text Generation with BERT
Authors: Tianyi Zhang*
Varsha Kishore*
Felix Wu*
Kilian Q. Weinberger
Yoav Artzi
Venue:  (2020)
Keywords: Metric
Evaluation
Contextual Embedding
Text Generation

Abstract: We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task and show that BERTScore is more robust to challenging examples compared to existing metrics. 

Published conference paper at ICLR 2020
BERTSCORE: EVALUATING TEXT GENERATION WITH BERT
Tianyi Z Varsha Kishore's Felix Wu' Kilian Q. Weinberger and Yoav Artzi
DDpaarmeett of Computer Science and Cornell Tech, Cornell University
[vk352, fw245, kilian] kilian]@ccornel] (eteee/oooreelooeee {yoav]êcs corne 11 edu
ASAPP Inc. Ezhang@asapp.com
ABSTRACT
We propose BERTSCORE, an automatic evaluation metric for text generation. Analogously to common metrics, BERTSCORE computes similarity score for each token the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTSCORE correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTSCORE is more robust to challenging examples when compared to existing metrics.
INTRODUCTION
Automatic evaluation of natural language generation, for example in machine translation and caption generation, requires comparing candidate sentences to annotated references. The goal is to evaluate semantic equivalence. However, commonly used methods rely on surface-form similarity only. For example, BLEU (Papineni al. 20002), the most common machine translation metric, simply counts n-gram overlap between the candidate and the reference. While this provides a simple and general measure, it fails to account for meaning prriiiiiiggeeegggggggg. lexical and compositional diversity.
In this paper. we introduce BERTSCORE, a language generation evaluation metric based on pre- trained BERT contextual embeddings (Devlin et al. 2019). BERTSCORE computes the similarity of two sentences as a sum of cosine similarities between their tokens' embeddings.
common pitfalls in --gram-based metrics (Banerjee & Lavie 2005). First, such methods often fail to robustly match paraphrases. For example, given the reference peo- ple like foreign cars, BLEU and METEOR (Banerjee & Lavie, 2005) incorrectly give a higher score to people like visiting places abroad compared to consumers prefer imported cars. This leads to performance underestimation when semantically-coore correct phrases are penalized because they differ from the surface form of the reference. In contrast to string matching (e.g.. in BLEU) or matching heuristies (e.g., in METFOR), we compute similarity using contextualized token embeddings. which have been shown to be effective for paraphrase detection (Devlin et al.) al. (219)). Second, n-gram mod- ordering changes (Isozaki
BERTSCORE addresses
els fail to capture distant dependencies and penalize semantically-critce al., 2010). For example, given small window of size two, BLEU will only mildly penalize swapping of cause and effect clauses (e.g. A because instead of because A), especially when the arguments A and B are long phrases. In contrast, contextualized embeddings trained to effectively capture distant dependencies and ordering.
We experiment with BERTSCORE machine translation and image captioning tasks using the outputs of 363 systems by correlating BERTSCORE and related metrics to available human judg- ments. Our experiments demonstrate that BERTSCORE correlates highly with human evaluations. In machine translation, BERTSCORE shows stronger system-level and segment-level correlations with human judgments than existing metrics on multiple common benchmarks and demonstrates
Equul contribution. Work done Comell.

---

Published conference paper at ICLR 2020
strong model selection performance compared BLEU. We also show that BERTSCORE is well-correlated with human annotators for image captioning, surpassing SPICE, a popular task- specific metric (Anderson et al., (016). Finally, we test the robustness of BERTSCORE on the adversarial paraphrase dataset PAWS (Zhang et al. 2019). and show that it is more bust to adversarial examples than other metrics. The code for BERTSCORE is available at https://gi http:///gt..uu...o.....et/e. http:///////ggg///eett.ooo.
PROBLEM STATEMENT AND PRIOR METRICS
Natural language text generation is commonly evaluated using annotated reference sentences. Given reference sentence tokenized to k tokens (Zi...., zz.) and candidate å tokenized to tokens (f....., &t), generation evaluation metric a function f fzx, E R. Better metrics have a higher correlation with human judgments. Existing metrics can be broadly categorized into using n-gram matching, edit distance, embedding matching, or learned functions.
2.1 2-GRAM MATCHING APPROACHES
The most commonly used metrics for generation count the number of n-grams that oceur in the reference and candidate æ. The higher the n is, the more the metric is able to capture word order, but it also becomes more restrictive and constrained to the exact form of the reference.
Formally, let 5 and ST be the lists of token n-grams (n E 2 in the reference and candidate sentences. The number matched n n-grams is Ewesp I[w E S%). where I[|] is an indicator function. The exact match precision (Exact-P,,) and recall (Exact-R,,) scores are:
Exact-P,, Ewesp I/ S% and Exact-R,, Ewesp Il E ST]
(8%
SS
Several popular metrics build upon one or both of these exact matching scores.
BLEU The most widely used metric in machine translation is BLEU (Papineni et al. 2002). which includes three modifications to Exact-P,. First, each n-gram in the reference can be matched at most once. Second, the number of exact matches is accumulated for all reference-candidate pairs in the corpus and divided by the total number of n n-grams in all candidate sentences. Finally, very short candidates are discouraged using a brevity penalty. Typically, BLEU is computed for multiple values of n (e.g.
2. 4) and the scores are averaged geometrically. A smoothed variant, SENT- BLEU (Kochn et al. 2007) is computed at the sentence level. In contrast to BLEU, BERTSCORE is not restricted maximum r--ramm length, but instead relies on contextualized embeddings that are able to capture dependencies of potentially unbounded length.
METEOR METEOR (Banerjee Lavie 2005) computes Exact-P, and Exact-R, while allowing backing-oft off from exact unigram matching to matching word stems, synonyms, and paraphrases. For example, running may match run if no exact match is possible. Non-exact matching uses an external stemmer, synonym lexicon, and a paraphrase table. METFOR 1.5 (Denkowski & Lavie 2014) weighs content and function words differently, and also applies importance weighting to different matching types. The more recent METEOR+ 2.0 (Guo & Huj 2019) further incorporates learned external paraphrase resource. Because METEOR requires external resources, only five languages supported with the full feature set, and eleven partially supported. Similar to METEOR, BERTSCORE allows relaxed matches, but relies on BERT embeddings that are trained on large amounts of raw text and are currently available for 104 languages. BERTSCORE also supports importance weighting, which we estimate with simple corpus statistics.
Other Related Metrics NIST (Doddington 2002 a revised version of BLEU that weighs each n-gram differently and uses alternative brevity penalty. ABLEU (Galley et al. 2015) modifies multi-reference BLEU by including human annotated negative reference sentences. CHRF (Popović
2015) compares character n-grams in the reference and candidate sentences. CHRF++ (Popović
2017 extends CHRF to include word bigram matching. ROUGE (Lin 2004) is a commonly used metric for summarization evaluation. ROUGE-n (Lin) 2004) computes Exact-R,, (usually
1, while ROUGE-L is a variant Exact-R, with the numerator replaced by the length of the longest common subsequence, CIDER (Vedantam et al. 2015) is image captioning metric that computes
2

---

Published conference paper at ICLR 2020
cosine similarity between tf-idf weighted n-grams.. We adopt similar approach to weigh tokens differently. Finally, Chaganty et al. (2018) and Hashimoto et al. (2019) combine automatic metrics with human judgments for text generation evaluation.
2.2 EDIT-DISTANCE-AAN BASED METRICS
Several methods use word edit distance or word error rate (Levenshtein, 1966), which quantify similarity using the number of edit operations required to get from the candidate to the refer- ence. TER (Snover et al., 2006) normalizes edit distance by the number reference words, and ITER (Panja & Naskar 20188) adds stem matching and better normalization. PER (Tillmann et al. 1997) computes position independent error rate, CDER (Leusch et al. (006) models block reorder- ing edit operation. CHARACTER (Wang al., 2016) and EED (Stanchev et al. (019) operate on the character level and achieve higher correlation with human judgements on some languages.
2.3
EMBEDDING-BASED METRICS
Word embeddings (Mikolov et al, 2013; Pennington et al. 2014 Grave et al. 2018 Nguyen et al, Athiwaratkun al. 2018) are learned dense token representations. MEANT 2.0 (Lo) 2017)
2017
uses word embeddings and shallow semantic parses to compute lexical and structural similarity. YISI-1 (Lo al., 2018) is similar MEANT 2.0, but makes the use of semantic parses optional. Both methods use a relatively simple similarity computation, which inspires our approach, including using greedy matching (Corley & Mihalcea 2005) and experimenting with a similar importance weighting to YISI-1. However, we use contextual embeddings, which capture the specific use of token in sentence, and potentially capture sequence information. We do not use external tools to generate linguistic structures, which makes our approach relatively simple and portable to new languages. Instead of greedy matching, WMD (Kusner et al. 2015), WMD, (Chow et al.) 2019), and optimal matching based earth mover's 's distance (Rubner ct al. 1998). The tradeoff between greedy and optimal matching was studied by Rus & Lintcan (2012). Sharma al. (2018) compute similarity with sentence-level representations. In contrast, teken-level computation allows us to weigh tokens differently according to their importance.
SMS Clark et al., 2019) propose
2.4 LEARNED METRICS
Various metrics are trained to optimize correlation with human judgments. BEER (Stanojević & Sima' an 2014) uses a regression model based character n-grams and word bigrams. BLEND (Ma et al. 2017) uses regression to combine 29 existing metrics. RUSE (Shimanaka et al., 2018 com- bines three pre-trained sentence embedding models. All these methods require costly human judg- ments supervision for each dataset, and risk poor generalization to new domains, even within known language and task (Chaganty et al.) 2018). [ui al. (2018) and Lowe al. (2017) train neural model to predict if the input text is human-generater This approach also has the risk of being optimized existing data and generalizing poorly new data. In contrast, the model underlying BERTSCORE is not optimized for any specific evaluation task.
BERTSCORE
Given a reference sentence x = (II...., ......,,)) and candidate sentence å = (.,,.... 21). we use contextual embeddings represent the tokens, and compute matching using cosine similarity, op- tionally weighted with inverse document frequency scores. Figure illustrates the computation.
Token Representation We use contextual embeddings to represent the tokens in the input sen- tences and J. In contrast to prior word embeddings (Mikolov et al.) 2013] Pennington et al. 2014], contextual embeddings, such as BERT (Devlin et al. 2019) and ELMO (Peters et al. 2018). can generate different vector representations for the same word in different sentences depending on the surrounding words, which form the context of the target word. The models used to generate these embeddings are most commonly trained using various language modeling objectives, such as masked word prediction (Devlin et al. 2019).
We provide an ablation study of this design choice in Appendix
3

---

Published conference paper at ICLR 2020
Contextual Embedding
Pairwise Cosine Similarity
Maximum Similarity
Importance Weighting (Optional)
Reference weather cold today
weather AAnEOu
Mebereee
NUSKT
20 10.700 Di
cold aurtnasdio
Candidate &t it iss freezing today
Vestoing
Figure 1: Illustration of the computation of the recall metric RBERT- Given the reference and candidate è, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy matching in red, and include the optional idf importance weighting.
We experiment with different models (Section 4 using the tokenizer provided with each model. Given tokenized reference sentence quence of vectors (xi,..., Xz). Similarly, the tokenized candidate = . Fin) is mapped to &.)). The main model
(Ir ....,... rR),
the embedding model generates se-
is BERT, which tokenizes the input text into sequence of word pieces (Wu et al. 2016), where unknown words are split into several commonly observed sequences of characters. The representation for each word piece is computed with Transformer encoder (Vaswani et al. 2017) by repeatedly applying self-attention and nonlinear transformations in an alternating fashion. BERT embeddings have been shown to benefit various NLP tasks (Devlin)
et al.
2019 Liu 2019 Huang et al. 2019 Yang et al. 2019a).
Similarity Measure The vector representation allows for a soft measure of similarity instead of exact-string (Papineni al. 2002) or heuristic (Banerjee & Lavie 2005) matching. The cosine
similarity of reference token T2 and candidate token ăü is
[x, We use pre-normalized
x, &
vectors, which reduces this calculation to the inner product x! &j. While this measure considers tokens in isolation, the contextual embeddings contain information from the rest of the sentence.
The complete and each token in & to a token in to compute precision. We use greedy matching to maximize the matching similarity score where each token is matched the most similar token in the other sentence. We combine precision and recall to compute an Fl measure. For reference and candidate é, the recall, precision, and Fl scores are:
BERTSCORE
matches each token in to token in å to compute recall,
maxx x &g FBERT = PBERT RBERT
RBERT
Σ max x &,, 8yE2
PBERT
Σ EYER
Z,Er
PBERT + RBERT
2:E2
Importance Weighting Previous work on similarity measures demonstrated that rare words can be more indicative for sentence similarity than common words (Banerjee & Lavie (2055) Vedantam ct al. 2015). BERTSCORE enables us to easily incorporate importance weighting. We experiment with inverse document frequency (idf) scores computed from the test corpus. Given M reference sentences { Ji=1 the idl score of word-piece token is
idf(w) log
Σ
Mw 201
M
where [[:] is an indicator function. We do not use the full tf-idf measure because we process single sentences, where the term frequency (tf) likely 1. For example, recall with idf weighting
Ener idf(x;; max e2 x, x;
RBERT
Because we use reference sentences to compute idf, the idf scores remain the same for all systems evaluated on specific test set. We apply plus-one smoothing to handle unknown word pieces.
We compare greedy matching with optimal assignment in Appendix
4

---

Published conference paper at ICLR 2020
Baseline Rescaling Because we use pre-normalized vectors, our computed scores have the same numerical range of cosine similarity (between -1 and 1). However, in practice we observe scores in a more limited range, potentially because of the learned geometry of contextual embeddings. While this characteristic does not impact BERTSCORE'S capability to rank text generation systems, makes the actual score less readable. We address this by rescaling BERTSCORE with respect to its empirical lower bound b as baseline. We compute b using Common Crawl monolingual datasets I For each language and contextual embedding model, we create 1M candidate-referenn pairs by grouping two random sentences. Because of the random pairing and the corpus diversity, each pair has very low lexical and semantic overlapping fl We compute b by averaging BERTSCORE computed these sentence pairs. Equipped with baseline we rescale BERTSCORE linearly. For example, the rescaled value ABERT of ABERT is:
RBERT b
RBERT- 1
After this operation REERT is typically between 0 and 1. We apply the same rescaling procedure for PBERT and FBERT This method does not affect the ranking ability and human correlation of BERTSCORE, and is intended solely to increase the score readability.
EXPERIMENTAL SETUP
We evaluate our approach on machine translation and image captioning.
Contextual Embedding Models We evaluate twelve pre-trained contextual embedding models, including variants of BERT (Devlin et al. (019). RoBERTa (Liu al. 2019b), XLNet (Yang al.
2019b), and XLM (aaplle & Conneau 2019). We present the best- best-performing models in Section 5 We use the 24-layer ToBERTajarge mode for English tasks, 12-layer BERT chiness model for Chi-
chRTehinese nese tasks, and the 12-layer cased multilingual BERT multi model for other languages We show the
performance of all other models in Appendix F Contextual embedding models generate embedding representations at every layer in the encoder network. Past work has shown that intermediate layers produce more effective representations for semantic tasks (Liu et al., 2019a). We use the WMT16 dataset (Bojar et al. 2016) validation set to select the best layer of each model (Appendix (B)
Our main evaluation corpus is the WMT18 metric evaluation dataset (Mal ct al., 2018). which contains predictions of 149 translation systems across 14 language pairs, gold references, and two types of human judgment scores. Segment-level human judgments assign score to each reference-candidde
Machine Translation
pair. System-level human judgments associate each system with a single score based all pairs in the test set. WMT18 includes translation from English to Czech, German, Estonian, Finnish, Russian, and Turkish, and from the same set of languages to English. We follow the WMT18 standard practice and use absolute Pearson correlation |p| and Kendall rank correlation to evaluate metric quality, and compute significance with the Williams test (Williams) 1959) for [p] and bootstrap re-samp sampling for 7 as suggested by Grahamm Baldwin (0114). We compute system- level scores by averaging BERTSCORE for every reference-candidee hybrid systems by randomly sampling one candidate sentence from one of the available systems for each reference sentence (Graham & Liu 2016). This enables system-level experiments with a higher number of systems. Human judgments of each hybrid system are created by averaging the WMT18 segment-level human judgments for the corresponding sentences in the sampled data. We compare BERTSCORES to one canonical metric for each category introduced in Section 2 and include the comparison with all other participating metrics from WMT18 in Appendix [F]
pair. We also experiment with
In addition to the standard evaluation, we design model selection experiments. We use 10K hybrid systems super-sampled from WMT18. We randomly select 100 out of 10K hybrid systems, and rank them using the automatic metrics. We repeat this process 100K times. We report the percentage of the metric ranking agreeing with the human ranking on the best system (Hits@) 1). In Tables 23 28
https: pettps:/ccommncraste wttp///o/oocccacccccoccoooo /rg/
BLEUU computed on these pairs is around zero.
'We use the tokenizer provided with each model. For all Hugging Face models that use the GPT-2 tokenizer, at the time of our experiments, the tokenizer adds a space to the beginning of each sentence.
All the models used are from http:///iin./ooooggggtttaaaaaaaa
5

---

Published conference paper at ICLR 2020
Metric
ent+cs (5/5)
en<+de
ene>et
ent>fii
en
en
ene+2h
(16/16)
(14/14)
(9/12)
(8/9)
(5/8)
(14/14)
BLEU
.770/995
971/9981
_966/977
973/.962
979/9983
.657/8266
.78//947
ITER
975/915
990/1984
975/.981
.96//973
9371.975
8611 865
.80//
RUSE
981/
997/
.990/
.91//
988/
.853/
.81/
YiSi-1
950/.987
992/,985
979/9979
9731.940
991/.992
9581.976
951/.963
PBERT
980/,994
.988/988
.900/9881
.995/957
982/.990
.791/,935
.981/9544
RBERT
.98//997
9971.990
986/1980
.977,980
.991.989
.054/.799
.90//976
FBERT
990//997
999/.989
990//982
998/1972
990/,990
,991,908
.988/967
FRERT (idf)
985/,995
.99//990
.922/981
992/9722
991//991
.226/941
.989/9973
Table I: Absolute Pearson correlations with system-level human judgments WMT18. For each language pair, the left number is the co-English correlation, and the right is the From-Englishe We bold correlations of metrics not significantly outperformed by any other metric under Williams Test for that language pair and direction. The numbers in parenthesis are the number of systems used for each language pair and direction.
Metric
ent+cs
ent>de
ent+et
ent*ff
en<+ru
ent>>r
en++zh
BLEU
956/.993
9697,977
9811.971
.622/958
972/9977
588/.796
968/.941
ITER
966/.865
990/.978
975/.982
989//966
943/.965
.742/.872
.988/
RUSE
974/
.996/
.88/
.93//
982/
.780/
973/
YiSi-1
942//985
991/.983
976/.976
964/.938
985/.989
881/.942
943/.957
PRERT
9965/989
995/.983
.90//970
976/.951
976/.988
846/.936
975/.950
RAERT
9899/995
9971,991
.982/9979
.899/977
.88/,989
540/.872
.981/9800
FBERT
.978/.993
.99119888
9891.978
983/.969
9851.989
.600/.10
.981/969
FREKT (idf)
.922/995
.98//988
.988/9799
989/.969
983/.987
.531.877
.980/ 963
Table 2: Absolute Pearson correlations with system-level human judgments on WMT18. We use 10K hybrid super-sampled systems for each language pair and direction. For each language pair, the left number is the to-English correlation, and the right is the from- -ong-shgiih. Bolding criteria is the same as Table
we include two additional measures to the model selection study: (a) the mean reciprocal rank the top metric-rated system according to the human ranking, and (b) the difference between the human score of the top human-rated system and that of the top metric-rated system.
Additionally, we report the same study on the WMT17 (Bojar et al. 2017) and the WMT16 (Bojar al. 2016) datasests in Appendix This adds 202 systems to our evaluation.
Image Captioning We use the human judgments of twelve submission entries from the COCO 2015 Captioning Challenge. Each participating system generates a caption for each image in the COCO validation set (Lin et al. (2014), and each image has approximately five reference cap- tions. Following Cui et al. (2018). we compute the Pearson correlation with two system-level metrics: the percentage of captions that are evaluated better or equal human captions (M1) and the percentage of captions that are indistinguishable from human captions (M2). We compute BERTSCORE with multiple references by scoring the candidate with each available reference and returning the highest score. We compare with eight task- aagk-aanostic metrics: BLEU (Papineni et al.) 2002). METFOR (Banerjee & L.avie 2005), ROUGE-L (Lin) 2004). CIDER (Vedantam al 2015), BEER (Stanojević & Sima an 2014). EED (Stanchev et al. 2019). CHRF++ (Popović) 2017). and CHARACTER (Wang et al.] 2016). We also compare with two task task-specific metrics: SPICE (Ander- son et al. 2016) and LEIC (Cui et al. 2018). SPICE is computed using the similarity of scene graphs parsed from the reference and candidate captions. LEIC is trained to predict a caption is written by human given the image.
For WMT16, only conduct segment-level experiments on to-English pairs due in the dataset.

---

Published conference paper at ICLR 2020
Metric
ent-cs
en<+dee
ent>et
ent>fi
en
ent->tr
ene+zh
BLEU
.134/151
803/.610
.756/.618
.461/088
.228/5199
.095/.029
638/.515
ITER
114/.000
814/,692
.742/.733
.475/1111
.234/5322
.102/0000
.673/
RUSE
.214
823/
.785/
.48//
.248/
.19//
.60//
YiSi-1
.159/.788
809//671
.749/.671
.677.2300
.248/544
.108/.398
613//594
PRERT
.173/.880
.706/.663
764/.771
.498/0788
.255/.545
.140/.722
.661/.511
RBERT
1663/114
804//730
.770/.722
.494/1488
.260/5542
.005/.030
.677/.557
FBERT .175/.884
.22//703
.76917633
.501/082
2621/544
.142/.031
.673/.629
FBERT (idf)
.179/.188
.824/722
.760/7764
.503//082
.655/539
.004/.300
.678/.595
Table 3: Model selection accuracies @Hits@ 1) on WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10-3, We bold the highest numbers for each language pair and direction.
Metric
ené+cs (Sk/5k)
en<+>e (78k/ 20k)
cn<+et (57k/32k)
ené+fi (16k/I0k)
en+-ru (10k/22k)
en++tr (9k/Ik)
en+ (33k/29k)
BLEU
.233/389
415/.620
.285/.414
.154/355
.228/3300
.15//261
.178/3111
ITER
.198/333
396/.610
235/.392
128/.311
.139/2291
-029//236
.14/
RUSE
.34/
.498/
368/
.233/
.311/
.25/
.218/
YiSi-1
.319/496
488/.691
.351/546
.311/5044
.300/400
.234/4188
.211/3323
PBERT
387//541
541/.715
.39//549
.883/486
.351,414
.280/328
.28//337
RIERT
.388/5570
.54/.728
.391/.594
.344.565
.3431,420
.290/4411
.255/3367
FBERT
404/.562
.550/7288
.3971586
.266/.466
.353/4423
.992/399
.264/.364
FBERT (idf)
408/5553
.550/7211
.395/585
.231/5377
.346/425
.296/4406
.260/3366
Table Kendall correlations with segment-level human judgments on WMT18. For each language pair, the left number is the to-English correlation, and the right is the from-English. We bold corre- lations of metrics not significantly outperformed by any other metric under bootstrap sampling for that language pair and direction. The numbers in parenthesis are the number of candidate-reference sentence pairs for each language pair and direction.
RESULTS
Machine Translation Tables 33 show system-level correlation human judgements, correla- tions on hybrid systems, and model selection performance. We observe that BERTSCORE is con- sistently a top performer. In to-English results, RUSE (Shimanaka et al.) (008)) shows competitive performance. However, RUSE is supervised method trained WMTI6 and WMTI5 human judgment data. In cases where RUSE models were not made available, such as for our From-English experiments, it is not possible use RUSE without additional data and training. Table shows segment-leve correlations. We see that BERTSCORE exhibits significantly higher performance compared to the other metries. The large improvement particularly suitable to analyze specific examples, where SENTBLEU is less reliable. In Appendix A provide qualitative examples illustrate the segment-level performance difference between SENTBLEU and BERTSCORE. At the segment-level, B BERTSCORE even significantly outperforms RUSE. Overall, we find that applying importance weighting using idf at times provides small bene- fit, but in other cases does not help. Understanding better when such importance weighting is likely help an important direction for future work, and likely depends on the domain of the text and the available test data. We continue without idf weighting for the rest of our experiments. While recall AAERT, precision PBERT, and Fl FBERT alternate as the best measure different setting, FBER performs reliably well across all the different settings. Our overall recommendation is there- fore to use F1. We present additional results using the full set of 351 systems and evaluation metrics in Tables 12 28 in the appendix, including for experiments with idf importance weighting, different contextual embedding models, and model selection.
BLEU stands out, making BERTSCORE
Image Captioning Table 5 shows correlation results for the COCO Captioning Challenge. BERTSCORE outperforms all task-agnosti baselines by large margins. Image captioning presents challenging evaluation scenario, and metrics based on strict n rr-gram matching, including BLEU and ROUGE, show weak correlations with human judgments. idf importance weighting shows signifi-
7

---

Published conference paper at ICLR 2020
Metric
M1
M2
Type
Method
QQP
PAWSego
BLEU
-0.19*
-0,005*
Trained on QQP
DecAtt
0.939'
0.263
METEOR
0.606
0.594"
(supervised)
DIIN
0.952'
0.324
BERT
0.963
0.351
ROUGE-L
0.090
0.096
CIDER
0.438
0.440*
Trained on QQP
DecAtt
0.511
SPICE
0.759
0.750
-PAWSSOP (supervised)
DIN
0.778
LEIC
0.939
0.949*
BERT
0.831
BEER
0.491
0.562
BLEU
0.707
0,527
EED
0,545
0.599
METEOR
0.755
0.532
CHRF++
0.702
0.729
ROUGE-L
0.740
0.536
CHARACTER
0.800
0.801
Metric
CHRF++
0.577
0.608
(Not trained QQP or
BEER
0.741
0,564
PREKT
-0.105
-0,041
EED
0.743
0,611
RBERT 0.888
0.863
CHARACTER
0.698
0.650
FBERT 0.322
0,350
POIKT
0.757
0.687
RREST (idf)
0.917
0.889
ROCRT
0.744
0.685
FEERT 0.761
0.685
Table 5: Pearson correlation on the 2015 COCO Captioning Challenge. The MI and M2 measures are described Section LEIC uses images as addi- tional inputs. Numbers with are cited from Cui et al.) (2018). We bold the highest correlations of task-specific and task-agnostic metrics.
FRERT (idl)
0.777
0.693
Table 6: Area under ROC curve (AUC) on QQP and PAWSoop datasets. The scores trained De- cATT (Parikh et al., (0166), DIIN (Gong et al. (018)). and fine-tuned BERT are reported by [Zhang et al. (2019). Numbers with are scores the held-out test set of QQP. We bold the highest correlations of task- specific and task-agnostic metrics.
cant benefit for this task, suggesting people attribute higher importance content words. Finally, LEIC (Cui et al., 2018), a trained metric that takes images as additional inputs and is optimized specifically for the COCO data and this set of systems, outperforms all other methods.
Speed Despite the use of a large pre-trained model, computing BERTSCORE is relatively fast. We are able to process 192.5 candidate-referent pairs/second using a GTX- 1080Ti GPU. The complete WMT18 en-de test set, which includes 2,998 sentences, takes 15.6sec process, compared 5.4sec with SacreBLEU (Post 2018), a common BLEU implementation. Given the sizes of commonly used test and validation sets, the increase in processing time relatively marginal, and BERTSCORE is a good fit for using during validation (e.g., for stopping) and testing, especially when compared to the time costs of other development stages.
6 ROBUSTNESS ANALYSIS
We test the robustness of BERTSCORE using adversarial paraphrase classification. We use the Quora Question Pair corpus (QQP: [yer al. 2017) and the adversarial paraphrases from the Para- phrase Adversaries from Word Scrambling dataset (PAWS: ZZang et al.) (019).. Both datasets con- tain pairs of sentences labeled to indicate whether they are paraphrases or not. Positive examples in QQP are real duplicate questions, while negative examples are related, but different questions. Sentence pairs in PAWS are generated through word swapping. For example, in PAWS, Flights from New York to Florida may be changed to Flights from Florida to New York and good classifier should identify that these two sentences are not paraphrases. PAWS includes two parts: PAWS which is based on the QQP data, and PAWSwai- We use the PAWSogP development set which contains 667 sentences. For the automatic metrics, we use no paraphrase detection training data. We expect that pairs with higher scores more likely to be paraphrases. To evaluate the automatic metrics on QQA, we use the first 5,000 sentences in the training set instead the the test set because the test labels are not available. We treat the first sentence the reference and the second sentence as the candidate.
Table 6 reports the area under ROC curve (AUC) for existing models and automatic metrics. We observe that supervised classifiers trained on QQP perform worse than random guess on PAWS which shows these models predict the adversarial examples are more likely to be paraphrases. When
8

---

Published conference paper at ICLR 2020
adversarial examples are provided in training, state-of-the-art models like DIIN (Gong et al. 2018) and fine-tuned BERT are able to identify the adversarial examples but their performance still de- creases significantly from their performance on QQP. Most metries have decent performance on QQP, but show significant performance drop on PAWSoqp- almost down to chance performance. This suggests these metrics fail to to distinguish the harder adversarial examples. In contrast, the performance of BERTSCORE drops only slightly, showing more robustness than the other metrics.
7 DISCUSSION
We propose BERTSCORE, new metric for evaluating generated text against gold standard refer- ences. BERTSCORE is purposely designed to be simple, task agnostic, and easy to use. Our analysis illustrates how BERTSCORE resolves some of the limitations of commonly used metries, especially on challenging adversarial examples. We conduct extensive experiments with various configuration choices for BERTSCORE, including the contextual embedding model used and the use of impor- tance weighting. Overall, extensive experiments, including the ones in the appendix, show that BERTSCORE achieves better correlation than common metrics, and is effective for model selec- configuration of BERTSCORF that clearly outperforms all others.
tion. However, there is While the differences between the top configurations are often small, it is important for the user to be aware of the different trade-off's, and consider the domain and languages when selecting the exact configuration to use. In general, for machine translation evaluation, we suggest using L'BERT, which we find the most reliable. For evaluating text generation English, we recommend using the 24- layer RoBERTatarge model to compute BERTSCORE. For non- non-Englist language, the multilingual BERT multi a suitable choice although BERTSCORE computed with this model has less stable performance on low-resource languages. We report the optimal hyperparameter for all models we experimented with Appendix B
Briefly following our initial preprint publication, Zhho et al. (2199) published a concurrently devel- oped method related to ours, but with a focus on integrating contextual word embeddings with earth mover's distance (EMD; Rubner et al., 1998) rather than our simple matching process. They also propose various improvements compared to our use of contextualized embeddings. We study these improvements in Appendix Cldnd show that integrating them into BERTSCORE makes it equivalent or better than the EMDD-baed approach. Largely though, the effect of the different improvements on BERTSCORE, is
modest compared to their method. Shortly after initial publication, YiSi-1 was updated to use BERT embeddings, showing improved performance (Lo 2019). This further corroborates our findings. Other recent related work includes training a model on top of BERT to maximize the correlation with human judgments (Mathur et al.) 2019 and evaluating gen- eration with BERT model fine-tuned on paraphrasing (Yoshimura et al. (019). More recent work shows the potential of using SERTS for training a summarization system (Li et al. 2019) and for domain-specific evaluation using SciBERT (Beltagy et al. 2019) to evaluate abstractive text summarization (Gabriel et al., 2019).
In future work, we look forward to designing new task-specific metrics that use BERTSCORE as a subroutine and accommodate task-specific needs, similar to how Wieting et al (2019) suggests to use semantic similarity for machine translation training. Because BERTSCORE is fully differentiable, it also can be incorporated into training procedure to compute learning loss that reduces the mismatch between optimization and evaluation objectives.
ACKNOWLEDGEMENT
This research is supported in part by grants from the National Science Foundation (II-1161114, III- 1526012, IIS 149882, IIS-1724282, TRIPODS- HAPODS--740822..
PARRER-17504900 the Office of Naval Research DOD (N00014-17-1-2175), and the Bill and Melinda Gates Foundation, SAP, Zillow, Workday, and Facebook Research. We thank Graham Neubig and David Grangier for for their insightful comments. We thank the Coreell NLP community including but not limited to Claire Cardie, Tianze Shi, Alexandra Schofield, Gregory Yauney, and Rishi Bommasani. We thank Yin Cui and Guandao Yang for their help with the COCO 2015 dataset.

---

Published conference paper at ICLR 2020
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic proposi- tional image caption evaluation. In ECCV, 2016.
Ben Athiwaratkun, Andrew Wilson, and Anima Anandkumar. Probabilistic fasttext for multi-sense word embeddings. In ACL, 2018.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with im- proved correlation with human judgments. In EEEvaluation@AAAL, 2005.
Iz Beltagy, Kyle Lo,, and Arman Cohan. SciBERT: A pretrained language model for scientific text. ArXiv, 2019.
Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. Results of the WMTI6 metrics shared task. WMT, 2016.
Ondfej Bojar, Yvette Graham, and Amir Kamran. Results of the WMT17 metrics shared task. In WMT, 2017.
Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In ACL, 2018.
Julian Chow, Lucia Specia, and Pranava Madhyastha. WMDO: Fluency-based word mover's dis- tance for machine translation evaluation. In 2019.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In ACI., 2019.
Courtney Corley and Rada Mihalcea. Measuring the semantic similarity of texts. In ACL Workshop, EMSEE '05, 2005.
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. Learning to evaluate image captioning. CVPR, 2018.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In WMT@ACL, 2014.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In HLT, 2002.
William Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In /WP, 2005.
Saadia Gabriel, Antoine Bosselut, Ari Holtzman, Kyle Lo, Asli elikyilmaz, and Yejin Choi. Cooper- ative generator-discrminnato networks for abstractive summarization with narrative flow. ArXiv, 2019.
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Mar- garet Mitchell, Jianfeng Gao, and William B. Dolan. deltaBLEU; A discriminative metric for generation tasks with intrinsically diverse targets. In ACL, 2015.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In /CML, 2017.
Yichen Gong, Heng Luo, and Jian Zhang. Natural language inference over interaction space. In ICLR, 2018.
Yvette Graham and Timothy Baldwin. Testing for significance of increased correlation with human judgment. In EMNLP, 2014.
10

---

Published conference paper at ICLR 2020
Yvette Graham and Qun Liu. Achieving accurate conclusions in evaluation of automatic machine translation metrics. In NAACI, 2016.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. arXiv preprint /rXiv:/802.06m 880200893, 2018.
Yinuo Guo and Junfeng Hu. Meteor++ 2.0: Adopt syntactic level paraphrase knowledge into ma- chine translation evaluation. In WMT; 2019.
Tatsu Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In NAACL-IILT, 2019.
Chenyang Huang, Amine Trabelsi, and Osmar R Zaiane. ANA at semeval-2019 task 3: Contex- tual emotion detection in conversations through hierarchical LSTMs and BERT. arXiv preprint arXiv:1904.00132, 2019.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic evaluation of translation quality for distant language pairs. In EMNLP, 2010.
Shankar Iyer. Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs.
https://tinyurll www.co/.../e... 2017.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison- Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In ACL, 2007.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In /CML, 2015.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv, 2019.
Gregor Leusch, Nicola Ueffing, and Hermann Ney. CDER: Efficient MT evaluation using block movements. In EACL, 2006.
Vladimir losifovich Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions and Rever sals. Soviet Physics Doklady, 10, 1966.
Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. Deep reinforcement learning with distributional semantic rewards for abstractive summarization. EMNLP-IJCNL2? 2019.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In ACI., 2004.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv: 1r03vv/903.0880 1903008855, 2019a.
Yang Liu. Fine-tune BERT for extractive summarization. arXiv preprint arXiv: 2019
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretrain- ing approach. arXiv,, abs/1907.1 1692, 2019b.
Chi-kiu Lo. MEANT 2.0: Accurate semantic mt evaluation for any output language. In WMT, 2017.
Chi-kiu Lo. YiSi a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In WMT, 2019.
11

---

Published conference paper at ICLR 2020
Chi-kiu Lo, Michel Simard, Darlene Stewart, Samuel Larkin, Cyril Goutte, and Patrick Littell. Ac- curate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metrie: The NRC supervised submissions the parallel corpus filtering task. In WMT, 2018.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. Towards automatic Turing test: Learning evaluate dialogue responses. In ACL, 2017.
Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. Blend: a novel combined mt metric based direct assessment casict-deu submission to WMT17 metries task. In WMT, 2017.
Qingsong Ma, Ondrej Bojar, and Yvette Graham. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In WMT, 2018.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Putting evaluation in context: Contextual em- beddings improve machine translation evaluation. ACL, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffirey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
Dai Quoc Nguyen, Dat Quoc Nguyen, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. A mixture model for learning multi-sense word embeddings. In ACL, 2017.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In WMT, 2018
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint (rXi::9904.01038 2019.
Joybrata Panja and Sudip Kumar Naskar. Iter: Improving translation edit rate through optimizable edit costs. WMT, 2018
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. decomposable attention model for natural language inference. In EMNLP, 2016.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. EMNLP, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. Deep contextualized word representations. NAACL-HLT, 2018.
Maja Popović. chrf: character n-gram f-score for automatic evaluation. WMT@ACL, 2015.
Maja Popović. chrf++: words helping character n-grams. In WMT, 2017.
Matt Post. A call for clarity in reporting BLEU scores. In WMT, 2018.
Nils Reimers and Iryna Gurevych. Alternative weighting schemes for elmo embeddings. arXiv preprint arXiv:1904.0295: 2019.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to image databases. In ICCV. IEEE, 1998.
Vasile Rus and Mihai Lintean. A comparison of greedy and optimal assessment of natural language student input using word-to-wont word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP. ACI,, 2012.
12

---

Published conference paper at ICLR 2020
Andreas Rckl, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. Concatenated power mean word embeddings universal cross-lingual sentence representations. arXiv, 2018.
Shikhar Sharma, Layla EI Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised metries in task-oriented dialogue for evaluating natural language generation. arXiv preprint (eXiv:/706009795 2018.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In WMT, 2018.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In AMTA, 2006.
Peter Stanchev, Weiyue Wang. and Hermann Ney. EED: Extended edit distance measure for machine translation. In WMT, 2019.
Miloš Stanojević and Khalil Sima'an, Beer: Better evaluation as ranking. In WMT, 2014.
Christoph Tillmann, Stephan Vogel, Hermann Ney, Arkaitz Zubiaga, and Hassan Sawaf. Accelerated dp based search for statistical translation. EUROSPEECH, 1997.
Kristina Toutanova, Chris Brockett, Ke M Tran, and Saleema Amershi. A dataset and evaluation metrics for abstractive compression of sentences and short paragraphs. In EMNLP, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image
description evaluation. In CVPR, 2015.
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. Character: Translation edit rate on character level. In WMT, 2016
John Wieting, Taylor Kinn-Kikkaatrick, Kevin Gimpel, and Graham Neubig. Beyond BLEU.training neural machine translation with semantic similarity. In ACL, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In ACL, 2018.
Evan James Williams. Regression analysis. wiley, 1959.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 2016.
Wei Yang, Haotian Zhang, and Jimmy Lin. Simple applications of BERT for ad hoc document retrieval. arXiv preprint 190. 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov. and Quoe V. Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv, 2019b.
Ryoma Yoshimura, Hiroki Shimanaka, Yukio Matsumura, Hayahide Yamagishi, and Mamoru Ko- machi. Filtering pseudo-referencee by paraphrasing for automatic evaluation of machine transla- tion. In WMT, 2019.
13

---

Published conference paper at ICLR 2020
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scram- bling. arXiv preprint 19Xivv/90400/ 130, 2019.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In EMNLP, 2019.
14

---

Published conference paper at ICLR 2020
Case
No.
Refercuce and Candidate Pairs
Human
Faeax
BLEU
st At same time Kingfisher closing B&Q outlets across the country
2: At the same time. Kingfisher will close 60 B & Q stores narionwide
38
125
530
Foear > -Lau
2: Hewleet- Packard will reduce jobs up to 30,000
Kit-Packard
up 30,000 jobs
119
441
According opinion Hungury, Serbiu safe third country"
2: Accordinn to Hungarian view, Serbia "sale third country"
2: Experts believe November's v Black Fiiday could be hokding back spending.
2: Experts believe
Black Friday November put brakes spending 22: And it's from this perspective that will watch him doc.
492
from this perspective, will see dic.
37
111
414
2: their view the human dignity of the man had been violated. man injured. 2: For esample when he steeted a shot from [leye over the crossbur the 56th minute.
500
470
115
Look human dignity
BLiu > Fanx
2: So, for example. when he
lilyye the latte (56th).
516
524
185
2: good prunk is tunny, but takes moments to reverse.
495
424
152
8: good prank is (unny, but it takes only
before be becomes boomerang.
2: 1 will exert the pressure on it and her urge to make decisson.
2: will pot pressure on them and omus on thom make docision.
507
471
220
Byposting 2: Transport for London is the Plaka animal "vandalism" is funny,
Transport London
527
527
246
big obstac access to johs market
knowledge German language. long bonder
11.
2: major hurdle for access the labour market are lack of knowledge of English.
558
131
313
Finer > Human
2: Hungary had the night of Tuesday closed its 175 km long bonder with Serbia.
Monday might Hungary
its
413
They got nothing. but they
allowed to keep clothes.
2: You got nothing. but could keep elothes.
majority of Republicans don't see Tromp's temperament problem.
14.
mujority Republicans see Trump's temperament not problem.
290
134
2: His car was still running in the driveway.
15.
8: His
still in the driveway.
299
71
2: Currently the mujority of staff are men.
moment men predorninate among the staff.
525
553
Human > Finerr
17.
2: There are, indoed, mulipplc variables play.
fact,
variables
role.
30
446
552
2: One was a man about 1ii tall.
18.
2: One of the men was about 1,80 metres size.
124
551
528
2:
sure take
454
547
2: All of this certainly exacts its toll.
Wage gains shown signs of picking up.
2: Increnses of wages showed signs of recovery.
140
514
Table 7: Examples sentences where similarity ranks assigned by Human, FBERTS and BLEU differ significantly WMT16 German-to-Ell to-English evaluation task. 2: gold reference, à: candidate outputs of MT systems. Rankings assigned by Human, FRERT and BLEU are shown in the right three columns. The sentences are ranked by the similarity, iee. rank is the most similar pair assigned by a score. An ideal metric should rank similar to humans.
A QUALITATIVE ANALYSIS
We study BERTSCORE and SENTBLEU using WMT16 Coiman-to-Engllsee (Bojar al. 2016). We rank all 560 sandidate-referene pairs by human score, BERTSCORE, or SENTBLEU from most similar least similar. Ideally, the ranking assigned by BERTSCORF and SENTBL.EU should be similar to the ranking assigned by the human score.
Table first shows examples where BERTSCORE and SENTBLEU scores disagree about the ranking for a candidate-reference pair by a large number. We observe that BERTSCORE is effectively able capture synonyms and changes in word order. For example, the reference and candidate sentences in pair 3 are almost identical except that the candidate replaces opinion in Hungary with Hungarian view and switches the order of the quotation mark (") and a. While BERTSCORE ranks the pair relatively high, SENTBLEU judges the pair as dissimilar, because it cannot match synonyms and is sensitive to the small word order changes. Pair shows a set changes that preserve the semantic meaning: replacing to cut with will reduce and swapping the order of 30,000 and jobs. BERTSCORE ranks the candidate translation similar to the human judgment, whereas SENTBLEU ranks much lower. We also see that SENTBLEU potentially over-rewards 7--rrm overlap, even when phrases are used very differently. In pair 6, both the candidate and the reference contain the human dignity of the man. Yet the two sentences convey very different meaning. BERTSCORE agrees with the human judgment and ranks the pair low. In contrast, SENTBLEU considers the pair as relatively similar because of the significant word overlap.
15

---

Published conference paper at ICLR 2020
Refereme::
people
enjoy
driving
foreign
Reference::
where
did
live before
moved to prageu
Candidate:
imporieed
CLT'
aBc
prefermdd
hy
people
Candidate:
before
whene
were
stay Vnn
Moxt Steslar
Leas Siaiilr
Figure 2: BERTSCORE visualization. The cosine similarity of each word matching in PBERT are color-coded.
The bottom half of Table 7 shows examples where BERTSCORE and human judgments disagree about the ranking. We observe that BERTSCORE finds it difficult to detect factual errors. For example, BERTSCORE assigns high similarity to pair 11 when the translation replaces German language with English and pair 12 where the translation incorrectly outputs Tuesday when it is supposed generate Monday. BERTSCORE also fails to identify that Sft in is equivalent with 1.80 metres in pair 18. As a result, BERTSCORE assigns low similarity to the eighth pair in Table SENTBLEU also suffers from these limitations.
Figure 2 visualizes the BERTSCORE matching of two pairs of candidate and reference sentences. The figure illustrates how FBERT matches synonymous phrases, such as imported cars and foreign cars. We also see that FBERT effectively matches words even given a high ordering distortion, for example the token people in the figure.
16

---

Published conference paper at ICLR 2020
B REPRESENTATION CHOICE
As suggested by previous works (Peters et al.) 2018 Reimers & Gurevych 2019). selecting good layer good combination of layers from the BERT model is important. In designing BERTSCORE, we use WMT16 segment-level human judgment data as a development set to fa- cilitate representation choice. For Chinese models, tune with the WMT17 "en-zh" data because the language pair "en-zh" is not available in WMT16. In Figure 3 we plot the change of human correlation of FBERT over different layers of BERT, RoBERTa, XLNet and XLM models. Based on results from different models, we identify common trend that FBERT computed with the intermediate representations tends work better. We tune the number of layer use for range of publicly available models Table 8 shows the results of our hyperparameter search.
Model
Total Number of Layers
Best Layer
bert-base-uncased
12
bert-large- uncased
24
18
bert-base-er base--e-eeeed finetuned mipe
12
bert- base
12
9
hert-base-chinese
12
8
roberta-base
12
10
roherta-large roberta-large-mall
24
17
19
xlnet-base-cased
12
xlnet-large-cased
24
7
xlm-mim-en- en---------088
12
7
xlm-mlm- 10elmm--0--1220 100-1280
16
Table 8: Recommended layer of representation for BERTSCORE. The layers chosen
based on a held-out validation set (WMT16).
http:://ggiiiiiticcoooooomomoomm models.html
17

---

Published conference paper at ICLR 2020
BERT models: Pearson Correlation
RoBERTa Models: Pearson Correlation
of BERT-F1 with human assessment WMT-16 to-en
of BERT-FI with human assessment WMT-16 to-cn
0.70
0.7
Correlation
Correlation
0.65
0.6
0.5
SUM BERT
RoBERG
BERT
RoBERDbup
0.56
RaHE RoHERD
0
2
8
10
12
11
16
18
20 22 24
2
4
6
10
12
14
16
18
20
22
21
Layer
Layer
XLNet models: Pearson Correlation
XLM models: Pearson Correlation
of BERT-FI with human assessment WMT- 16 to-en
of BERT-F1 with human ussessment on WMT-16 to-en
0.5
0.6
Correlation
Correlation
0.5
0.4
0.4
0.3
0.3
XLNNS
XLMa
XLNetaee
112
XLM
0.2
10
12 14 16 18 20
22 24
2
4
6
10
12
14
16
Layer
Layer
BERT-Chinese: Pearson Correlation
of BERT-FI with human assessment on WMT- 1- en-zh
0.78
Correlation
0.76
0.72
4
10
12
Layer
Figure 3: Pearson correlation of FBERT computed with different models, across different layers, with segment-level human judgments on the WMT16 to-English machine translation task. The WMT17 English-Chinen Chiness data is used for the BERT Chinese model. Layer 0 corresponds to using BPE embeddings. Consistently, correlation drops significantly in the final layers.
18

---

Published conference paper at ICLR 2020
C ABLATION STUDY OF MOVERSCORE
Word Mover's Distance (WMD; Kusner et al., 2013) semantic similarity metric that relies on word embeddings and optimal transport. MOVERSCORE (Zhao et al., 2019) combines contextual embeddings and WMD for text generation evaluation. In contrast, BERTSCORE adopts a greedy approach to aggregate token-level information. In addition to using WMD for generation evalu- ation, Zhao et al. 2019) also introduce various other improvements. We do a detailed ablation study understand the benefit of each improvement, and to investigate whether it can be applied to BERTSCORE. We use a 12-layer uncased BERT model on the WMT17 to-English segment-level data, the same setting as Zhao et (2019).
We identify several differences between MOVERSCORE and BERTSCORE by analyzing the released source code. We isolate each difference, and mark it with bracketed tag for our ablation study:
1. [MNLI] Use a BERT model fine-tuned on MNLI (Williams et al. 2018).
2. [PMEANS] Apply power means (Rekl et al. 2018 to aggregate the information of different
layers 9
3. [IDF-I. L.] For reference sentences, instead of computing the idf scores the 560 sen- tences in the segment-level data (TDF-S] $)), compute the idf scores on the 3,005 sentences in the system-level data.
4. [SEP] For candidate sentences, recompute the idf scores on the candidate sentences. The weighting of reference tokens are kept the same as in [IDF-S]
5. [RM] Exclude punctuation marks and sub-word tokens except the first sub-word in each word from the matching.
We follow the setup of Zhao et al. 2019)) and use their released fine-tuned BERT model to conduct the experiments. Table shows the results of our ablation study. We report corre- lations for the two variants of WMD Zhao et al. (219) study: unigrams (WMD1) and bi- grams (WMD2). Our FBERT corresponds to the vanilla setting and the importance weighted vari- ant corresponds to the [IDF-S] setting. The complete MOVERSCORE metric corresponds to [IDF-S]+ [SEP] + [PMEANS] [MNLI] [RM]. We make several observations. First, for all lan- guage pairs except fi-en and Iv-en, we can replicate the reported performance. For these two lan- guage pairs, Zhao et al. (0199) did not release their implementations at the time of publication To Second, confirm the effectiveness of [PMEANS] and [MNLI]. In Appendix (F] we study more pre-trained models and further corroborate this conclusion. However, the contribution
other tech- niques, including [RM] and [SEP], seems less stable. Third, replacing greedy matching with WMD does not lead to consistent improvement. In fact, oftentimes BERTSCORE is the better met- ric when given the same setup. In general, for any given language pair, BERTSCORE is always
among the best performing ones. Given the current results, it is not clear tht WMD is better than greedy matching for text generation evaluation.
Zhao et al. 2019) the embeddings from the last five layers from BERT and I.2-normalizes the embed- ding vectors each layer hefore computing the P-MEANS and 1.2-normalizing the concatenated P-MEANS. A public comment the project page indicates that some of the techniques are not applied for these two language pairs http://hhhaammmt//////eemem/meee
19

---

Published conference paper at ICLR 2020
Ablation
Metric
de-en
fi-en
Iv-en
zh-en
WMD1
0.628
0.655
0.795
0.692
0.701
0.715
0.699
Vanilla
WMD2
0.638
0.661
0.797
0.695
0.700
0.728
0.714
FBERT
0.659
0.680
0.817
0.702
0.719
0.727
0.717
WMD1
0.636
0.662
0.824
0.709
0.716
0.728
0.713
IDF-
WMD2
0.643
0.662
0.821
0.708
0.712
0.732
0.715
FBERT
0.657
0.681
0.823
0.713
0.725
0.718
0.711
WMD1
0,633
0.659
0.825
0.708
0.716
0.727
0.715
IDF-
WMD2
0.641
0.661
0.822
0.708
0.713
0.730
0.716
FBERT
0.655
0.682
0.823
0.713
0.726
0.718
0.712
WMD1
0.651
0.660
0.819
0.703
0.714
0.724
0.715
IDF-L + SEP
WMD2
0.659
0.662
0.816
0.702
0.712
0.729
0.715
FBERT
0.664
0.681
0.818
0.709
0.724
0.716
0.710
WMD1
0.651
0.686
0.803
0.681
0.730
0.730
0.720
IDF++ SEP RM
WMD2
0.664
0.687
0.797
0.679
0.728
0.735
0.718
FBERT
0.659
0.695
0.800
0.683
0.734
0.722
0.712
WMD1
0.658
0.663
0.820
0.707
0.717
0.725
0.712
IDE-L + SEP + PMEANS
WMD2
0.667
0.665
0.817
0.707
0.717
0.727
0.712
FBERT
0.671
0.682
0.819
0.708
0.725
0.715
0.704
WMD1
0.659
0.679
0.822
0.732
0.718
0.746
0.725
IDF-L+ SEP
WMD2
0.664
0.682
0.819
0.731
0.715
0.748
0.722
MNLI
FBERT
0.668
0.701
0.825
0.737
0.727
0.744
0.725
WMD1
0.672
0.686
0.831
0.738
0.725
0.753
0.737
IDFL + SEP PMEANS MNLI
WMD2
0.677
0.690
0.828
0.736
0.722
0.755
0.735
FBRRT
0.682
0.707
0.836
0.741
0.732
0.751
0.736
IDE-L + SEP
WMD1
0.670
0.708
0.821
0.717
0.738
0.762
0.744
PMENNS + MNLI RM
WMD2
0.679
0.709
0.814
0.716
0.736
0.762
0.738
FBERT
0.676
0.717
0.824
0.719
0.740
0.757
0.738
Table 9: Ablation Study of SCORE and BERTSCORE using Pearson correlations on the WMT17 to-English segment-level data. Correlations that are not outperformed by others for that language pair under Williams Test are bolded. We observe that using WMD does not consistently improve BERTSCORE
20

---

Published conference paper at ICLR 2020
Type
Metric
Meaning
Grammar
Combined
PBERT
0.36
0.47
0.46
BERTSCORE
RAERT
0.64
0.29
0.52
FEERT
0.58
0.41
0,56
BLEU
0,46
0.13
0.33
METEOR ROUGE-L
0.53
0.11
0.36
Common metrics
0.51
0.16
0.38
SARI
0.50
0.15
0.37
Best metrics according to Toutanova (2166)
SKIP---KECALL-MUUT-RRE BAASE-2-RECALL-MMULLMAY PARSE-2-RECALL--UULZEE
0.59
N/A
0.51
N/A
0.35
0.52
0.57
0.35
0.52
Table 10: Pearson correlations with human judgments on MSR Abstractive Text Compression Dataset.
D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION
We the human judgments provided from the MSR Abstractive Text Compression Dataset (Toutanova et al.) 2016) to illustrate the applicability BERTSCORE to abstractive text compression evaluation. The data includes three types of human scores: (a) meaning: how well compressed text preserve the meaning of the original text; (b) grammar: how grammatically correct a compressed text is; and (c) combined: the average of the meaning and the grammar scores. We follow the experimental setup of "Toutanova et al. (2016) and report Pearson correlation between BERTSCORE and the three types of human scores. Table shows that RBERT has the highest cor- relation with human meaning judgments, and PвEK correlates highly with human grammar judg- ments. FBER provides balance between the two aspects.
21

---

Published conference paper at ICLR 2020
Task
Model
BLEU
PRERT
RAERT
FRERT
PBERT
RBERT
FBERT
CoavS2S (Gehring et al.
2017)
0.266
0.6099
0.6055
0.6075
0.8499
0.8482
0.8488
WMT14
Transformer-Dig big OOtt et 2018
0.298
0.6587
0.6528
0.6558
0.8687
0.8664
0.8674
En-De
DynamicCony"*
2019
0.297
0.6526
0.6464
0.6495
0.8664
0.8640
0.8650
ConvS2S (Gehring et al.
2017)
0.408
0.6998
0.6821
0.6908
0.8876
0.8810
0.8841
WMT14
Transformer-byy big (Ott al. 2018
0.432
0.7148
0.6978
0.7061
0.8932
0.8869
0.8899
En-Fr
DynamicConv [Wu ct al.
2019
0.432
0.7156
0.6989
0.7071
0.8936
0.8873
0.8902
Transformer-iwstt"
OOt et al.
2019
0.350
0.6749
0.6590
0.6672
0.9452
0.9425 0.9438
IWSLT14
LightConv (Wu et al. 2019)
0.348
0.6737
0.6542 0.6642
0.9450
0.9417 0.9433
De-En Em
DynamicConv (Wu 2019
0.352
0.6770
0.6586 0.6681 0.9456
0.9425 0.9440
BLEU scores and BERTSCORES of publicly available pre-trained MT models in fairseq (Ou et al., 2019). We show both rescaled scores marked with and raw BERTSCORE scores. ": trained unconfirmed WMT data version, **: trained on WMT16 + ParaCrawl, ***; trained on WMT16, + trained by us using fairseq.
Table 11:
E BERTSCORE OF RECENT MT MODELS
Table 1I shows the BLEU scores and the BERTSCORES of pre-trained machine translation models on WMT14 Beglish-to-Germare WMT14 Eaglish-to-Frena IWSLT14 German-tor Ho-mann-t--Engihh task. We used publicly available pre-trained models from fairseq (Our et al. 2019 Because pre- trained Transformer model on IWSLT is not released, we trained our own using the fairseq library. We use multilingual cased BERT basq for English-to-Gerran and Eaglish-to-Frenchs pairs, and English uncased BERT basc
13 for German-to-English pairs. Interestingly, the gap between Dy-
namicConv (Wu et al. 2019) trained on only WMT16 and a Transformer (Ott et al.) 2018 trained on WMT16 and ParaCraw (about 30x more training data) becomes larger when evaluated with BERTScORE rather than BLEU.
Code and pre-trained model available at https://giinbbbommmmccc/a/aaee
Haash code: seeet--beee-mllsiingaa--------ee
version=0.2.0
13 Hash code: roserta-l arge 1.7 ve 2e-sion=0.2.0
22

---

Published conference paper at ICLR 2020
F ADDITIONAL RESULTS
In this section, we present additional experimental results:
1. Segment-level and system-level correlation studies on three years of WMT metric evalua- tion task (WMT16-15
2. Model selection study on WMT18 10K hybrid systems
3. System-level correlation study 2015 COCO captioning challenge
4. Robustness study on PAWS-QQP.
Following BERT (Devlin et al. 2019). a variety of Transformer-based (Vaswani et al. 2017) pre- trained contextual embeddings have been proposed and released. We conduct additional experiments with four types of pre-trained embeddings: BERT, XLM (Lample & Conneau 2019). XLNet (Yang) al. 2019b), and RoBERTa (Liu et al. 2019b). XLM (Cross-lingual Language Model) is Trans- former pre-trained on the translation language modeling of predicting masked tokens from pair of sentence in two different languages and masked language modeling tasks using multi-lingual train- ing data. Yang et al. (2019b) modify the Transformer architecture and pre-train it on permutation language modeling task resulting some improvement on top of the original BERT when fine-tuned on several downstream tasks. Liu et al. (2019h) introduce RoBERTa (Robustly optimized BERT ap- proach) and demonstrate that an optimized BERT model is comparable to or sometimes outperforms an XLNet downstream tasks.
We perform comprehensive study with the following pre-trained contextual embedding models
BERT models:
bert-based-chinng bert-base-uncased.
bert-large-uncased, and
bert-bbase-cased-mpp bert- bert-bbas-multiiingual----
RoBERTa models: roberta-base roberta- rro and roberta-large
XLNet models: base- cased and xlnet -ase--baaealaee
XLM models: xlm-mlm-e en 2048 and xlm-mlm 110n 100-1280
F.1 WMT CORRELATION STUDY
Experimental setup Because missing data in the released WMT16 dataset (Bojar al. 2016). we are only able to experiment with to-English segment-level data, which contains the outputs of 50 different systems on language pairs. We use this data as the validation set for hyperparam- eter tuning (Appendix B). Table [2 shows the Pearson correlations of all participating metrics and BERTSCORES computed with different pre-trained models. Significance testing for this dataset does not include the baseline metrics because the released dataset does not contain the original outputs from the baseline metrics. We conduct significance testing between BERTSCORE results only.
The WMT17 dataset (Bojar et al. 2017) contains outputs of 152 different translations 14 lan- guage pairs. We experiment on the segment-level and system-level data on both to-English and English language pairs. We exclude fi-en data from the segment-level experiment due to error in the released data. We compare our results to all participating metrics and perform standard significance testing done by [Bojar al. (207)). Tables 16 show the results.
The WMT18 dataset (a et al. 2018) contains outputs of 159 translation systems on 14 lan- guage pairs. In addition to the results in Tables 4 complement the study with the correla- tions of all participating metrics in WMT18 and results from using different contextual models for BERTSCORE.
Results Table 12 collectively showcase the effectiveness of BERTSCORE in correlating with human judgments. The improvement of BERTSCORE is more pronounced on the segment-level than on the sevel. We also see that more optimized or larger BERT models can produce better contextual representations (e.g., comparing FROBERTA Large and FIBETTLLagE). In contrast, the smaller XLNet performs better than large one. Based on the evidence in Figure & and Tables 12
15 Denoted by names specified at hhun:h/wwggnnnaaacoooooooommmmon models.htm
23

---

Published conference paper at ICLR 2020
hypothesize that the permutation language task, though leading to good of model weights for fine-tuning on downstream tasks, does not necessarily produce informative pre-trained embeddings for generation evaluation. We also observe that fine-tuning pre-trained models on a related task, such natural language inference (Williams al. 2018), can lead to better human correlation in evaluating text generation. Therefore, for evaluating English sentences, we recommend computing BERTSCORE with 24-layer RoBERTa model fine-tuned the MNLI dataset. For evaluating Non-English sentences, both the multilingual BERT model and the XLM model trained on 100 languages are suitable candidates. We also recommend using domain- language-specinn contex- tual embeddings when possible, such as using BERT Chinese models for evaluating Chinese tasks. In general, advise users consider the target domain and languages when selecting the exact configuration to use.
F.2 MODEL SELECTION STUDY
Experimental setup Similar to Section 4 we use the 10K hybrid systems super-sampled from WMT18. We randomly select 100 out of TOK hybrid systems, rank them using automatic metries, and repeat this process 100K times. We add to the results in the main paper (Table performance of all participating metrics in WMT18 and results from using different contextual embedding models for BERTSCORE. We reuse the hybrid configuration and metric outputs released in WMT18. In addition to the Hits@1 measure, we evaluate the metries using (a) mean reciprocal rank (MRR) of the top metric-rated system in human rankings, and (b) the absolute human score difference (Diff) between the top metrie- and human-rated systems. Hits @ captures metric's s ability to select best system. The other two measures quantify the amount of error metric makes in the selection process. Tables Tables[3] show the results from these experiments.
Results The additional results further support our conclusion from Table BERTSCORE demon- strates better model selection performance. We also observe that the supervised metric RUSE dis- plays strong model selection ability.
F.3 IMAGE CAPTIONING ON COCO
We follow the experimental setup described in Section 4 Table 29 shows the correlations of several pre-trained contextual embeddings. We observe that precision precision-based methods such as BLEU and PBERT are weakly correlated with human judgments on image captioning tasks. We hypothesize that this is because human judges prefer captions that capture the main objects in picture for image captioning. In general, RBERT has a high correlation, even surpassing the task task-specific metric SPICE Anderson et al.) (2016). While the fine-tuned RoBERTa-Large model does not result in the highest correlation, it is one the best metrics.
F.4 ROBUSTNESS ANALYSIS ON PAWS-QQP
We present the full results of the robustness study described Section Table 30 In general, we observe that BERTSCORE is more robust than other commonly used metrics. BERTSCORE computed with the 24-layer RoBERTa model performs the best. Fine-tuning RoBERTa-Large on MNLI (Williams et al. 2018)) can significantly improve the robustness against adversarial sentences. However, a fine-tuned BERT on MRPC (Microsoft Research Paraphrasing Corpus) (Dolan Brock- ett 2005) performs worse than its counterpart.
24

---

Published conference paper at ICLR 2020
Setting
Metric
ca-en
de-en
li-en
10-c0
ru-en
tr-en
560
560
560
560
560
560
DPMFCOMB
0.713
0.584
0.598
0.627
0.615
0.663
METRICS-F
0.696
0.601
0.557
0.662
0.618
0.649
COBALT-F.
0.671
0.591
0.554
0.639
0.618
0.627
UPF-COBA.
0.652
0.550
0.490
0.616
0.556
0.626
MPEDA
0.644
0.538
0.513
0.587
0.545
0.616
CHRF2
0.658
0.457
0,469
0.581
0.534
0.556
Unsupervised
CHRF3
0.660
0.455
0.472
0.582
0.535
0.555
0.454
0.452
0.570
0.522
UoW REVAL
0.577
0.528
0.471
0.547
0.528
0.531
WORDF3
0.599
0..477
0.473
0.525
0.504
0.536
WORDF2
0.596
0.445
0.471
0.522
0.503
0.537
WORDFI
0.464
0.508
0.497
SENTBLEU
0.557
0.448
0.484
0.499
0.502
0.532
DTED
0.794
0.254
0.361
0.329
0.375
0.267
Supervised
BEER
0.661
0.462
0.471
0,551
0.533
0.545
PaERI-Bax
0.729
0.617
0.719
0.651
0.684
0.678
flaxR Base
0.741
0.639
0.616
0.693
0.660
0.660
FREKI-Ba:
0.747
0.640
0.061
0.723
0.672
0.688
PRERT Bue 100 idf)
0.723
0.638
0.662
0.700
0.633
0.696
RIEETT Bos (no idf)
0.745
0.656
0,638
0.697
0.653
0.674
FUERI-Bas idf)
0.747
0.663
0.666
0.714
0.662
0.703
PEERT Bas MRFC
0.697
0.618
0.614
0.676
0.62
0.695
HOERT Bus MRPC
0.723
0.587
0.648
0.725
0.644
0.617
0.691
0.654
0.702
PBERT
(idF)
0.713
0.613
0.630
0.693
0.635
0.691
Rarky Boer MRPC (idl)
0.727
0.631
0.573
0,666
0.642
0.662
F FRuEr MRFC (idf)
0.735
0.637
0.620
0.700
0.658
0.697
Pamxi-Larg
0.756
0.671
0.701
0.723
0.678
0.706
0.768
0.684
0.677
0.720
0.686
0.699
F Famge-tar
0.774
0.693
0,705
0,736
0.701
0.717
Parxt Lage (idF)
0.758
0.653
0.704
0.734
0.685
0.705
Hgg (dl)
0.771
0.680
0.718
0.687
0.692
Fianze (idF)
0.774
0.678
0.700
0.740
0.701
0.711
D.738
0.642
0.671
0.712
0.669
0.671
Haannen-Bos
0.745
01669
0.645
0,698
0.682
0.653
0.761
0.674
0.686
0.752
0.697
0.689
Trained
Peauttn mae (idr)
0.751
0.626
0.723
0.685
0.668
Heaamnn-oo (dll
0.744
0.652
0.638
0.699
0.685
0.637
Fuamirs- (id
0.767
0.653
0.688
0.737
0.705
0.685
0.757
0.702
0,709
0.735
0.721
0.676
HosstKi-Las
0.765
0.713
0.686
0.718
0.714
0.676
Phannsn-tary
0.780
0.724
0.728
0.753
0.738
0.709
PlottMts-aap (idf)
0.771
0.682
0,705
0.727
0.714
0.681
Raanaas-iary (idl)
0.762
0.695
0.683
0,711
0.708
0.678
FhotiKis-aap (idf)
0.786
0.704
0.727
0.747
0.732
0.711
0.777
0.718
0.733
0.744
0.729
0.747
HRUSERT Laus MNLL
0.731
0.702
0.741
0.727
0.732
Flamiich
0.795
0.736
0.733
0.757
0.744
0.756
PRANERn Lang MNLI (idf)
0.794
0.695
0.731
0.752
0.752
0.747
RRuBERT Laze MNLI (idl)
0.792
0.706
0.6944
0.757
0.724
0,733
Fluntrn-taag--ae MaL1 (idF)
0.804
0.710
0.729
0.760
0.742
0.754
PaLNesBaae
0.708
0.612
0.639
0.650
0. 0.606
0.690
RyLNN Busc
0.728
0.630
0.645
0.621
0.675
FxLNs-Mae
0.727
0.631
0.640
0.639
0.626
0.695
PxLNa-Sas (idf)
0.726
0.618
0.655
0.678
0.629
0.700
RyaNN Base (idf)
0.618
0.66
0.635
0.682
Fae (idf)
0.739
0.635
0.649
0.681
0.643
0.702
PALNET-LARGE
0.710
0.577
0.643
0.647
0.616
0.684
RyL. NET-LARGE
0.732
0.600
0.610
0.636
0.627
0.663
FXL-NET-LARGE
0.733
0.600
0.643
0.655
0.637
0.691
PALNET-LARGE (idf)
0.728
0.574
0.652
0.669
0.633
RELLNET-LAEE (idf)
0.735
0.592
0.597
0.642
0.629
0.662
FXLNEE-LARGE (idf)
0.742
0.592
0.643
0.670
0.645
0.685
PaMM-sn
0.688
0. 0.569
0.613
0.645
0.583
0.659
RXLM
0.715
0.603
0.577
0.645
0.609
0.644
I'XMM-D
0.713
0.597
0.610
0.657
0.610
0.668
PXLM En (idf)
0.728
0.576
0.649
0.681
0.604
0.683
PXLM Ea (idf)
0.730
0.597
0.591
0.659
0.622
0.669
FXLM-En (idf)
0.739
0.594
0.636
0.682
0.626
0.691
Table 12: Pearson correlations with segment-level human judgments on WMT16 to-English trans- lations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.
25

---

Published conference paper at ICLR 2020
Setting
Metric
cs-en
de-en
fi-en
Iy.en
nu-en
tr-en
zh-en
560
560
560
560
560
560
560
CHRF
0,514
0.531
0.671
0.525
0.599
0.607
0.591
CHRF
0.523
0.534
0.678
0.520
0.588
0.614
0.593
MEANT 2.0
0.578
0.565
0.687
0,586
0.607
0.596
0.639
Unsupervised
MEANT 2. N.0-NOSRL
0.566
0.564
0.682
0.573
0.591
0.582
0.630
SENTBLEU
0.435
0.432
0.571
0.393
0.484
0.538
0,512
TREEADOREG
0.486
0.526
0.638
0.571
0,535
UHH_TSKM
0.507
0.479
0.600
0.394
0.465
0.478
0.477
AUTODA
0.499
0.543
0.673
0.533
0.584
0.625
0.583
BEER
0,511
0.530
0.681
0.515
0.577
0.600
0,582
Supervised
BLEND
0.594
0.571
0.733
0.577
0.622
0.67]
0,661
BLEU2VEC
0.439
0.429
0,489
0.529
0.526
NGRAM2VEC
0.436
0.435
0.582
0.383
0.490
0.538
0.520
PaIRT-Rue
0.625
0.659
0.808
0.68%
0.698
0.713
0.675
Ramner-Baac
0.653
0.645
0.782
0.662
0.678
0.716
0.715
FUERI-Bac
0.654
0.671
0.811
0.692
0.707
0.731
0.714
PanKr-too lidl)
0.626
0.702
0,667
Racxi-Bas (idf)
0.652
0.658
0,789
0.678
0.696
0.703
0.712
FunKT-Boss [idl)
0.657
0.680
0.823
0,712
0.725
0.718
0.711
Purrr-
0.599
0.630
0.788
0.657
0.659
0.710
0.681
0.613
0.620
0.754
0.616
0.650
0.685
0,705
Feasc Buse SBaC
0.717
0.712
Phrar-Booe Baat-----Mrre (dd))
0.609
0.630
0.801
0,680
0.676
0.712
0.682
Ratar Sue (id0
0.611
0.628
0.759
0.633
0.665
0.687
0.703
FRERT- Mas-MMrC (idl)
0.633
0.649
0.803
0.678
0.690
0.719
0.713
Lagg
0.638
0.685
0.816
0.717
0.719
0.746
0.693
Recat-Laxs:
0.661
0.676
0.782
0.693
0.705
0.744
0.730
FAEKI- Lasge
0.666
0.701
0.723
0.730
0.760
0.731
Panxi-Largs (df
0.644
0.692
0.827
0.728
0.729
0.734
0.689
Reess-LLage (idf)
0.665
0.686
0.796
0.712
0.729
0.733
0,730
Laux Laue (idf)
0.671
0.707
0.829
0.738
0.745
0.746
0.729
PR.BERT Base
0.639
0.663
0.801
0.689
0.688
0.700
0.704
RauaEKI Buse
0.648
0.652
0.768
0.651
0.669
0.684
0.734
FR.BERT
0.675
0.683
0.818
0.693
0.707
0.718
0.740
PoWBERT (idF)
0.629
0.655
0.804
0,702
0.711
0.707
0,700
RRuBEKI Buc (idF)
0.652
0.646
0,773
0.667
0.676
0.689
0,734
FR.BERT Bac (idr)
0.673
0.673
0.823
0.708
0.719
0.721
0.739
Pro-Trained
PRuBERT
0.744
RRGaERT-
0.685
0.714
0.778
0.711
0.718
0.713
0.759
FRUBERT Large
0.710
0.745
0,833
0.756
0.746
0.751
0.775
PREBERT Large udd))
0.644
0.721
0.815
0,740
0.734
0.736
0.734
RyseEK Lage (idf)
0.683
0.783
0.718
0.720
0.726
0.751
F FROBERT Lage (idf)
0.703
0.737
0.838
0.761
0.752
0.764
0.767
PELBERTS-oeg Larg MNLI
0.694
0.736
0.822
0.764
0.741
0.754
0.737
0.706
0.725
0.785
0.732
0.741
0.750
0.760
F PREBERT Lage MNLI
0.723
0.747
0.822
0,764
0.758
0.767
0.765
RRoEERT Large MNLL (idF)
0.686
0.738
Rewamen- aarg--MM (idf)
0.697
0.717
0.796
0.741
0.753
0.757
0.762
FROBERTO Large MNU (idf)
0.714
0.740
0.835
0.774
0.773
0.776
0.767
PsINET-Bue
0.595
0.579
0.779
0.632
0.626
0.688
0.6446
0.603
0.560
0.746
0.617
0.624
0.689
0.677
0.610
0.700
0.675
Pa (dd)
0.616
0.603
0.795
0.665
0.659
0.693
0.649
Hx Ro (dd
0.614
0.583
0.765
0.640
0.648
0.697
0.688
FXLNET-Bus Buse (ddf)
0.627
0.603
0.795
0.663
0.665
0.707
0.684
PAINET-Lay
0.620
0.622
0.796
0.648
0.648
0.694
0.660
0.622
0.701
FXLNEr-Lege
0.635
0.627
0.794
0.654
0.664
0.705
0.698
PSINET-aag (idf)
0.635
0.633
0.808
0.673
0.672
0.688
0.649
Exa Rai (id
0.626
0.611
0.770
0.646
0.661
0.682
0.700
PAINE--Lage (idl)
0.682
0.700
0.695
PaLt-E
0.649
0.672
0.643
RxLM-tn
0.592
0.586
0.734
0.618
0.647
0.673
0.686
FXLM--i
0.595
0.605
0.768
0.641
0.664
0.686
0.683
PSiM-nn (idl)
0.599
0.618
0.795
0.670
0.686
0.690
0.657
Rxuas-ta (idF)
0.624
0.684
0.698
FaIM-in (idl)
0.630
0.624
0.798
0.676
0.698
0.698
0.694
Table 13: Absolute Pearson correlations with segment-level human judgments on WMT17 to-
English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.
26

---

Published conference paper at ICLR 2020
Setting
Metric
en-cs 32K
en-de
en-fi
en-lv
en-ru
en-tr
en-zh
3K
3K
3K
560
247
560
r]
AUTODA
0.041
0.099
0.204
0.130
0.511
0.409
0.609
AUTODA-TECTO
0.336
CHRF
0.376
0.336
0.503
0.420
0.605
0.466
0.608
CHRF+
0.377
0.325
0.514
0.421
0.609
0,474
Unsupervised
CHRF++
0.368
0.328
0.484
0.417
0.604
0.466
0.602
MEANT
0.350
0.727
MEANT NO0-L
0.395
0.324
0.565
0.425
0.636
0.482
0.705
SENTBLEU
0.274
0.269
0.446
0.259
0.468
0.377
0.642
TREEAGGREG
0.361
0.305
0.509
0.383
0.535
0.441
0.566
BEER
0.398
0.336
0.557
0.420
0.569
0.490
0.622
BLEND
0.578
Supervised
BLEU2VEC
0.305
0,313
0.503
0.315
0.472
0.425
NGRAM2VEC
0.486
0.317
PolxI-Mali
0,412
0.364
0,561
0.435
0.606
0.579
0.759
Ryakt-Muhi
0,443
0.430
0.587
0.480
0.663
0.571
0.804
FRERT-Malli
0.440
0.404
0.587
0.466
0.653
0.587
0.806
PbERT-Muhi (idf)
0.411
0.328
0.568
0.444
0.616
0.555
0.741
RBERT Muli (idf)
0.449
0.416
0.591
0.479
0.665
0.579
0.796
FBERT Molii (idf)
0.447
0.379
0.588
0.470
0.657
0.571
0.793
Pre-Trained
PALM-100
0.406
0.383
0.553
0.423
0.562
0.611
0.722
RxIM-100
0,446
0.436
0.587
0.458
0.626
0.652
0.779
FXLM 100
0.444
0.424
0.577
0.456
0.613
0.628
0.778
PXLM 100 (idf)
0.419
0.367
0.557
0.427
0.571
0.595
0.719
RXLM 100 (idf)
0.450
0.424
0.592
0.464
0.632
0.644
0.770
FoLM-100 (idf)
0.448
0.419
0.580
0.459
0.617
0.644
0.771
Table 14: Absolute Pearson correlation (rrl) and Kendall correlation (7) with segment-level human
judgments WMT17 from-English translations. Correlations of metrics not significantly outper- formed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.
27

---

Published as a conference paper at ICLR 2020
Serting
Metric
cS-en
de-en 11
fi-en
Iv-en
ru-en
tr-en
zh-en
BLEU
0.971
0.923
0.903
0.979
0.912
0.976
0,864
CDER
0,989
0,985
0.922
0.973
0.904
CHARACTER
0,972
0.974
0.946
0.932
0.958
0.949
0.799
CHRF
0.939
0.968
0.938
0.968
0.952
0.944
0,859
CHRF
0.940
0.965
0.927
0.973
0.945
0.960
0,880
MEANT 2.0
0.926
0.950
0,941
0,970
0,962
0.932
0,838
Unsupervised
MEANT 2. 2.0-NOSRL
0.902
0.936
0.933
0.963
0.960
0.896
0,800
NIST
1,000
0.931
0.931
0,960
0.912
0.971
0,849
PER
0.968
0.8%6
0.962
0.911
0.932
0,877
TER
0,989
0.906
0.952
0,971
0.912
0.954
0.847
TREEAGOREG
0.983
0.920
0.977
0.986
0.918
0.987
0.861
UHH.TSKM
0.996
0.937
0.921
0,990
0.914
0.987
0,902
WER
0.987
0.896
0.948
0.969
0.907
0.925
0.839
AUTODA
0,438
0.959
0.925
0,973
0.907
0.916
0.734
BEER
0.972
0.960
0.955
0,978
0.936
0.972
0,902
Supervised
BLEND
0.968
0.976
0.958
0,979
0.964
0.984
0.894
BLEU2VEC
0.989
0.936
0,888
0,966
0.907
0.961
0.886
NGRAM2VEC
0.984
0.935
0.890
0.963
0.907
0.955
0,880
PaERT-Nue
0.918
0.981
0.892
Rausci-trc
0.995
0.975
0.944
0
0,953
0.991
0.975
F FnERT-Rae
0.987
0.961
0.979
0,991
0.937
0.991
0.953
PunRt-Sse: (idF)
0.983
0.937
0.998
0.992
0.939
0.985
0,878
Resait-Bse Beec Gdff
0.985
0.949
Fmngt-Soes (idl)
0.992
0.967
0.995
0.992
0.960
0.996
0,951
0.982
0.926
0.990
0,987
0.916
0.970
0,899
Ragat-ase har MBFC
0.999
0.979
0.950
0.982
0.957
0.977
0,985
FRERT Buse
0.994
0.957
0,986
0,994
0.938
0.980
0,960
Purat-ne Pura--na-Man Mire (id)
0.976
0
Rauat Bus MIFC. (idF)
0.999
0.987
0.962
0,980
0.975
0.979
0,973
Furar-noo--ae (idf)
0.997
0.968
0.995
0.997
0.956
0.989
0.963
Pansi-LLans
0
0.937
0.991
0.996
0.921
0.987
0.905
Racxi-Lass
0.996
0.975
0.953
0.985
0.954
0.992
0.977
FBEKI Lans
0.990
0.960
0.981
0.995
0.938
0.992
0.957
Panxi-Layg (idr)
0.986
0.938
0,998
0.995
0.939
0.994
0,897
Reaat-Lape (idf)
0.997
0.982
0.967
0.979
0.974
0.992
0,966
PaExt-Luge (dI)
0.994
0.965
0.993
0,995
0.958
0.998
0.959
PouBERT
0.987
0.930
0.984
0.966
0.916
0.963
0.955
RAuaEKI Blat
0,999
0.982
0.947
0.979
0.956
0.986
0,984
FROBERTO Baie
0.996
0.961
0.993
0,993
0.937
0.983
0
PAIBERT. Buc (idf)
0.990
0.938
0,980
0.956
0.929
0.967
0.962
RRuBEKI- Bus (idf)
0.998
0.987
0.963
0.979
0.971
0.986
0.974
FRUBERT. Buc (idF)
0.996
0.970
0.999
0.994
0.952
0.989
0.982
Pre-Trained
PRoBERT-Lagg
0.989
0.948
0,984
0,949
0.927
0.960
0.967
RRUSEKT Lans
0.998
0.988
0.957
0
0.969
0.982
0,984
FROBEKT. Lage
0.995
0.973
0.997
0.991
0.949
0.984
0.987
PlanERT--tage (idf)
0.989
0.959
0.975
0.935
0.944
0.968
0.974
Lauz (idf)
0.995
0.991
0.962
0.979
0.981
0.981
0.970
(ROBERTE-LAA Luge (idf)
0.996
0.982
0.998
0
0.965
0.991
0.984
Ps..ERTo- MNLI
0.994
0.981
0.974
Ruonnen- MN11
0.995
0.991
0.962
0
0.973
0.985
0.984
PREBERTO Lage
0.999
0.982
0.992
0.996
0.961
0.988
0.989
P' -aog MNLI (id[)
Large--
0.995
0.970
0.997
0,985
0.955
0.988
0,979
RAUBEKI Lans MNLI (idf)
0.994
0.992
0.967
0.977
0.983
0.988
0.972
FROBERTO MNU (idf)
0.999
0.989
0.996
0.997
0.972
0.994
0.987
Pa Rase
0.988
0.938
0.993
0.993
0.914
0.974
0,960
0.999
0.978
0.956
0.977
0.946
0.981
0,980
PXLNET Bise
0.996
0.963
0.986
0.991
0.932
0.981
0.978
PaNET--Be Bow (dd
0.992
0.939
Rii (idf)
0.999
0.986
0.968
0.973
0.964
0.987
0.955
FXL Bose (idf)
0.998
0.974
0.996
0.994
0.950
0.990
0.970
Ps
0.991
0.944
0.996
0.995
0.924
0.982
0.943
Rxaxurt-tapp
0.996
0
0.945
0,971
0.961
0.986
0.958
FXLNET-LLay:
0.992
0.961
PSINETatage (dd))
0.995
0.955
0.999
0.996
0.941
0.985
0.937
Rx Bxt (idf)
0.993
0.985
0.951
0.960
0.975
0.974
0.910
PaINET-Lage (ddl
1,000
0.978
0.994
0.993
0.962
0.994
0.954
PaAM-Ia
0.983
0.933
0.994
0,989
0.918
0.973
0.928
RyiM-tn
0.998
0.985
0.972
FSLM--1
0.994
0.960
0.985
0.995
0.938
0.984
0.964
PaIM-n (idl)
0.986
0.940
0.997
0.992
0.939
0.979
0.916
RxLM-t (idF)
0.999
0.983
0.966
0.980
0.975
0.991
0.952
Pkast-a (idl)
0.995
0.993
0.958
Table 15: Absolute Pearson correlations with system- human judgments on WMT17 to- toglish translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.
28

---

Published conference paper at ICLR 2020
Setting
Metric
en-de 16
en-lv 17
en-ru
en-tr
en-zh
8
BLEU
0,956
0.804
0.866
0.898
0.924
CDER
0.968
0.813
0.930
0.924
0.957
CHARACTER
0.981
0.938
0.897
0.939
0.975
0.933
CHRF
0.976
0.863
0.955
0.950
0.991
0.976
CHRF++
0.974
0.852
0.956
0.945
0,986
0,976
MEANT 2.0
0.858
0.956
Unsupervised
MEANT 2.0-NOSRL
0.976
0.770
0.959
0.957
0,991
0.943
NIST
0.962
0.769
0.935
0.920
0.986
PER
0.954
0.687
0.851
0.887
0.963
TER
0,955
0.796
0,909
0.933
0.967
TREEAGGREG
0.947
0.773
0.927
0.921
0.983
0.938
UKH.TSKM
WER
0.954
0.802
0.906
0.934
0.956
AUTODA
0.975
0.603
0.729
0.850
0.601
0.976
BEER
0.970
0.842
0.930
0.944
0.980
0.914
Supervised
BLEND
0,953
BLEU2VEC
0.963
0.810
0,859
0.903
0.911
NGRAM2VEC
0.862
PoERT-Muhi
0.959
0.798
0.960
0.946
0.981
0.970
RutkI-Muti
0.982
0.909
0.957
0.980
0.979
0.994
FBuRT-Mani
0.976
0.859
0.959
0.966
0.980
0.992
PBERT-Mall (idf)
0.963
0.760
0.960
0.947
0.984
0.971
HAERT Multi (idf)
0.985
0.907
0.955
0.981
0.984
0.982
FBERT (idf)
0.979
0.841
0.958
0.968
0.984
0.991
Pre-Trained
PXLM-100
0.967
0.825
0.965
0.953
0.974
0.977
RXUM-100
0,980
0.902
0.965
0.982
0.977
0.979
FXLM-100
0,979
0.868
0.969
0.971
0,976
0.986
PAL 100 (idl)
0,968
0.809
0.965
0.955
0,980
0.975
RXLM (idf)
0.981
0.894
0.964
0.984
0.983
0.968
FXLM (idf)
0.979
0.856
0.966
0.973
0.982
0.979
Table 16: Absolute Pearson correlations with system-level human judgments on WMT17 from-
English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.
29

---

Published conference paper at ICLR 2020
Setting
Metric
LS-en
de
el-en 57K
li-en
tr-en
zb-en
5K
78K
16K
10K
9K
33K
CHARACTER
0.256
0,450
0,286
0.185
0.244
0.172
0.202
0.396
0.139
METEOR++
0.270
0.457
0.329
0.207
0.253
0.204
0.179
Unsupervised
UHH TSKM
SENTBLEU
0.274
0.235
0,436
0.415
0,300
0.285
0.168
0.154
0.228
0.235
0.154
0.145
0.151
0.178
0,330
0.225
YISI
0.319
0.488
0.351
0.231
0,300
0.234
0.211
YISI-1 SRL
0.317
0.483
0.345
0.237
0,306
0.233
0.209
BEER
0.295
0.481
0.341
0.232
0.288
0.229
0.214
Supervised
BLEND RUSE
0.322
0.492
0.354
0.226
0.290
0.232
0.217
0.347
0.498
0.368
0.273
0.31
0.259
PBERT Bese
0.264
0.325
0.264
RRERT-Baa
0.370
0.528
0.378
0.291
0.333
0.257
0.244
PRERT Bac
0.373
0,531
0,385
0.287
0.341
0.266
0.243
(BERT-Bea Base (ddf)
0.352
0,524
0,382
0.27
0.326
0.277
0.235
RREKI-Saz (idf)
0.368
0.536
0,388
0.300
0.340
0.284
0.244
FEERT (idf)
0.375
0.535
0.393
0.294
0.339
0.289
0.243
PEERT Bos MRPC
0.343
0.520
D.365
0.247
0.333
0.25
0.227
0.370
0,524
0.373
0.277
0.34
0.261
0.244
FBERT Buse MRPC
0.366
0,529
0.377
0.271
0.342
0.263
0.242
TAEKT Bose MRIC
0.371
0.25
0.318
RacaT: Roo-MarC: (idf)
0.379
0.531
0.383
0.285
0.339
0.266
0.242
FREKT BRc-MRRC (idf)
0.373
0,534
0,383
0.274
0,342
0.275
0.242
J'HERT-LARGE
0.361
0,529
0,380
0.276
0,340
0.266
0.241
RanKE-
0.386
0,532
0.386
0.297
0.347
0.268
0.247
FURRI-LARG
0.537
0,390
0.296
PEERT-LARGE (idl)
0.377
0.532
0.390
0.287
0.342
0.292
0.246
RamRt-1 ARCE: (idl)
0.386
0,544
0.396
0.308
0,356
0.287
0.251
FARKT-LARGE (idf)
0.388
0.545
0.399
0.309
0.358
0.300
0.257
RRannn
0.368
0.53
0.371
0.274
0,318
0.265
0.235
Fluitttt-BBa
0.391
0.540
0,383
0:273
0.339
0.270
0.249
P PRanKR (idl)
0.379
0,528
0.372
0.261
0,314
0.265
0.232
Ras (dd
0.389
0,539
0.384
0.288
0.332
0.267
0.245
Foomiii--taa (idf)
0.400
0.540
0.385
0.274
0.337
Pre- Pre-Trained
PReBaRD-LARGE
0.387
0.541
0.345
RoamKbb- ARGB
0.388
0.546
0,391
0.304
0.343
0.290
0.255
F FiseEICULAACEE
0.404
0.550
0.397
0.296
0.353
0.292
0.264
PRoBERD- LARGE (idl)
0.391
0,540
0.387
0.280
0.334
0.284
0.252
RRGMERD-LARII (idf)
0.548
0.305
0.338
0.295
F FAGHERD-LARGE (idr)
0.408
0.550
0.395
0.293
0.346
0.296
0,260
PRUBERIE Lans MNLI
0.397
0.549
0.396
0.299
0.351
0.295
0.404
0.553
0.393
0.313
0.351
0.279
0.253
"LusttKis-Lmm-- MNLL)
0.418
0.557
0.402
0.312
0.362
0.290
0.258
PROSERT Luur- MNLI
0.301
Resarnats Large-------- MNU (idE)
0.412
0,555
0,400
0.316
0.357
0.289
0.258
FRoistaci--are (idf)
0.417
0.559
0.403
0.309
0.357
0.307
0.258
PaUNas Buse
0.335
0.514
0.359
0.243
0.308
0.247
0.232
FXLNa
0.35
0.515
0.362
0.261
0.311
0.227
0 0.232
FXLNa-Busz Buac
0.351
0.517
0.365
0.257
0.315
0.25
0:7
PaISa Base (idf)
0.339
0,366
0.258
0.307
0.261
RxiNe-Maa (idf)
0.364
0.521
0.371
0.268
0.317
0.242
0.238
FXLNA- (idf)
0.355
0.524
0.374
0.265
0.320
0.261
0.241
PAL.EE-
0.344
0,522
0.371
0.252
0.316
0.264
0.233
REL-NET-LASS
0.358
0.275
0.332
FXL-NE-LE NET-LARGE
0.357
0.530
0,380
0.265
0.334
0.263
0.238
PEL LARGE (idf)
0.348
0.520
0.373
0.260
0.319
0.265
0.235
LARGE (idf)
0.366
0.529
0.378
0.278
0.331
0.266
0.241
FXL-NET-LE LARGE (idf)
0.375
0.530
0.382
0.274
0.332
0.274
0.240
PXLM Es.
0.349
0,516
0.366
0.244
0.310
0.259
0.233
MXxLM-En
Fki M-Hr-
0.358
0.525
0.373
0.259
0.322
0.258
0.238
PALs (idr)
0.355
0.527
0.374
0.254
0.311
0.28
0.238
RNLM- (idl)
0.362
0.528
0.376
0.274
0.333
0.26
0.24
FXLM-Es (idf)
0.367
0.531
0.382
0.273
0.330
0.275
0.246
Table 17: Kendall correlations with segment-level human judgments on WMT18 to-English transla- tions. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.
30

---

Published conference paper at ICLR 2020
Setting
Metric
en-es 5K
en-de
en-et
en-fi
en-ru
en-tr 1K
en-zh
20K
32K
10K
22K
29K
CHARACTER
0.414
0.604
0.464
0.403
0.352
0.404
0,313
ITER
0.333
0.610
0.392
0.311
0.291
0.236
SENTBLEU
0.389
0.620
0.414
0.355
0.330
0.261
0.311
Unsupervised
YISI-0
0.471
0.661
0.531
0.464
0.394
0.376
0.318
YISI-
0.496
0.691
0.546
0.504
0.407
0.418
0.323
YISI-1 SRL
0.696
0,310
BEER
0.518
0.686
0.558
0.511
0.403
0.374
0,302
Supervised
BLEND
0.394
POERT
0.541
0.715
0.549
0.486
0.414
0.328
0.337
ROERT Matn
0.570
0.728
0.594
0.565
0.420
0.411
0.367
FBERT Muli
0.562
0.728
0.586
0.546
0.423
0.399
0.364
PRERT-Moli (idf)
0.525
0.7
0.54
0.495
0.423
0.352
0.338
RBERT-Mali (idl)
0.569
0.727
0.601
0.561
0.423
0.420
0.374
FBERT-Moli (idf)
0.553
0.721
0.585
0.537
0.425
0.406
0.366
Pre-Trained
PXLM-100
0.496
0.711
0.561
0.527
0,417
0.364
0.340
RXLM-100
0.564
0.724
0.612
0.584
0.418
0.432
0.363
FXLM-100
0.533
0.727
0.599
0.573
0.421
0.408
0.362
PXLM-100 100 (idf)
0.520
0.710
0.572
0.546
0.421
0.370
0.328
RXLM (idf)
0.567
0.722
0.609
0.587
0,420
0.439
0,365
FXLM-100 (idf)
0.554
0.724
0.601
0.584
0.422
0.389
0.355
Table 18: Kendall correlations with segment-level human judgments on WMT18 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted bold. For each language pair, we specify the number examples.
31

---

Published as conference paper at ICLR 2020
Setting
Metric
de
filen
zh
16
14
BLEU
0,970
0.986
0,973
0.657
0.978
CDER
0.972
0.980
0.990
0.984
0.980
0.664
0.982
CHARACTER
0.970
0.993
0.979
0.989
0.991
0.782
0,950
ITER
0.975
0.990
0.975
0.996
0.937
0.861
0,980
METEOR+
0.945
0.991
0.978
0,971
0.995
0.864
0,962
NIST
0.954
0.984
0.983
0.975
0.973
0.970
0.968
Unsupervised
PER
0.970
0.985
0.983
0,993
0.967
0.159
0,931
TER
0.950
0.990
0.970
0.533
0.975
UHH_TSKM
0.952
0.980
0,989
0,982
0,980
0.547
0,981
WER
0.951
0.961
0.991
0,961
0.968
0.041
0.975
YISI-D
0.956
0.904
0.975
0,978
0.988
0.954
0.957
YISI-I
0.950
0.992
0.979
0,973
0.991
0.958
0.951
YIS:-1 SRL
0.965
0.995
0.981
0.977
0.992
0.869
0,962
BEER BLEND RUSE
0.958
0.994
0,985
0,991
0,982
0.870
0.976
Supervised
0.973
0,991
0.985
0,994
0.993
0.801
0.976
0
0.997
0.990
0.991
0,988
0.853
0
PaERI-Bac
0.965
0.995
0,986
0,973
0.976
0.941
0.974
Ramer-toss
0.067
0.988
F FuEni-Ree
0.982
0.994
0.983
0
0,985
0.949
0,984
PurRt-Rows lidl)
0
0.993
0.987
0,988
0.976
0.984
0.973
Reaan- (idf)
0.996
0.994
0.977
0,995
0.995
0.874
0,983
Fuaai-sso (idl)
0.995
0.988
0.981
Peast- MBRC
0,953
0.976
0.79%
0.977
0.992
0.994
0.983
0,988
0.993
0.707
0.990
Fiasee Besccr--sw-Bie MBac
0.975
0.995
0.987
0.975
0.986
0.526
0
PbESt-
(dd
0.957
0.997
0.989
0,967
0.975
0.894
0,980
Rouee-aae MarC (ide)
0.052
0.987
P PREst-Ue Bae Mire (idf)
0.975
0.998
0.987
0.985
0.987
0.784
0.987
PaRkI-Lane
0.978
0.992
0.987
0.971
0.920
0.978
0.997
0.990
0.985
0.990
0.992
0.098
0,990
FanKI-Lang
0.989
0.992
0.987
0.983
0.985
0.784
0,986
Laue (idf)
0.977
0.992
0
0.986
0.976
0.980
0.977
Ruzzn-Lage (jdf)
0.998
0.993
0,983
0
0.995
0,800
0,986
Fauxi-Lame (idf)
0.989
0.993
0.986
0.993
0.987
0.976
0.984
PRuBERT
0.970
0.995
0.991
0.998
0.976
0.796
0.980
RRoBEKT.
0.996
0.996
0.982
0.998
0.994
0.477
0.991
FRaBERnn Bas
0.984
0.997
0.989
0.999
0.987
0.280
0.989
PRABERT Bac (idF)
0.966
0.991
0.850
0.984
RROBERT, Base (idf)
0.995
0.998
0.981
0.998
0.995
0.230
0,989
FROBERT Bac (idf)
0
0.998
0.989
0.997
0.988
0.741
0.990
Pre-Trained
PRABERT Lage
0.980
0.998
0.990
0.995
0.982
0.791
0.981
ARABERT, Lage
0.998
0.997
0,986
0.997
0.995
0.054
0.990
FRABEKT Lang
0.990
0.990
0.998
0.990
0.499
0,988
PROBERT Lage (ddf)
0.972
0.997
0.993
0.985
0.982
0.920
0.983
RR.BER (idf)
0.996
0.997
0.984
0.997
0.995
0,578
0.989
FROBERT-LLon Lang (idf)
0.985
0.999
0.992
0.992
0.991
0.826
0,989
PR.BERTI- Lage MNLI
0.989
0.998
0.994
0.998
0.985
0.908
0,982
MounERTe-Lagge MNLI
1,000
0.996
0.991
0.996
0.998
0.992
0.998
0.992
0.665
0.989
PRSBERTO Large- MNLI (idf)
0.986
0.998
0.994
0.993
0.986
0.989
0.985
Aaoneem Large MNL1 (df)
0.999
0.997
0.986
0,997
0.993
0.633
0.990
FROBERT Lage MINL (idr)
0.995
0.998
0.996
0.993
0.963
0.990
0.970
0.979
0.982
RAINET-
0.994
0.997
0.979
0.995
0.994
0.795
0.990
0.983
0.997
0.983
0.993
0.987
0.505
0.988
PaNET- Buse (ddl)
0.968
0.998
0.986
0.990
0.978
0.923
0.982
RaINET- (idi)
0.993
0.439
FxL (idf)
0.981
0.999
0.984
0.995
0.989
0.722
0.988
PALNEr-Leg:
0.969
0.998
0.986
0.995
0.979
0.850
0
0.995
0.997
0.977
0.997
0.995
0.430
0.988
FSLNET-LLeg
0.983
0.998
0.983
0.997
0.988
0.713
0.988
PaLEE-LLage (idf)
0.963
Hxa arp- (dd
0.942
0.997
0.975
0.993
0.996
0.531
0.982
FxiNe--seg (idf)
0.978
0.997
0.983
0.996
0.990
0,886
0.984
PaLx-Ba
0.965
0.996
0.990
0.978
0.980
0.946
0.981
RxrM-t
0.990
0.995
0.984
0.996
0.996
0.286
0.987
FxiM-In
0.576
0.987
PaaM-in (ddl)
0.960
0.996
0.990
0.987
0.980
0.989
0.981
HyrM- (idf)
0.991
0.997
0.983
0.996
0.998
0.612
0.985
FaIM-an (idf)
0.976
0.998
0.988
0.994
0.992
0.943
0.985
Table 19: Absolute Pearson correlations with system-level human judgments on WMT18 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.
32

---

Published conference paper at ICLR 2020
Setting
Metric
en-cs 5
en-de 16
en-ct
en-fi
en-ru
en-tr
en-zh 14
14
12
9
BLEU
0.995
0.981
0.975
0.962
0.983
0.826
0,947
CDER
0.997
0.986
0.984
0.964
0.984
0.861
0.961
CHARACTER
0.993
0.989
0.956
0.974
0.983
0.833
0.983
ITER
0.915
0.984
0.981
0.973
0.975
0.865
METEOR+
NIST
0.999
0.986
0.983
0.949
0.990
0,902
0,950
Unsupervised
PER
0.991
0.981
0.958
0.906
0.988
0.859
0.964
TER
0.997
0.988
0.981
0.942
0.987
0.867
0.963
UHH TSKM
WER
0.997
0.986
0.981
0.945
0.985
0.853
0,957
YISI-0
0.973
0.985
0.968
0.944
0.990
0.990
0,957
YISI-1
0.987
0.985
0.979
0.940
0.992
0.976
0,963
YISI-1 SRL
0.990
0.952
BEER
0.992
0.991
0.980
0.961
0.988
0.965
0.928
Supervised
BLEND
0.988
RUSE
PRERT-Muli
0.994
0.988
0.981
0.957
0.990
0.935
0,954
RatRT-Mali
0.997
0.990
0.980
0.980
0.989
0.879
0.976
0.997
0.989
0.982
0.972
0.990
0.908
0.967
PRERT-Mali (idf)
0.992
0.986
0.974
0.954
0.991
0.969
0.954
RBERT-Mah (idf)
0.997
0.993
0.982
0.982
0,992
0.901
0,984
FBURT Moli (idf)
0.995
0.990
0.981
0.972
0.991
0.941
0.973
Pre-Trained
PXLM-100
0.984
0.992
0.993
0.972
0.993
0.962
0,965
RXIM-100
0.991
0.992
0.992
0.989
0.992
0.895
0.983
FXLM-100
0.988
0.993
0.993
0.986
0.993
0.935
0,976
PaM-1100 (idf)
0.982
0.992
0.994
0.975
0.993
0,968
0.964
RXLM 100 (idf)
0.993
0.993
0.991
0.989
0.993
0.911
0,986
FXLM (idf)
0.989
0.993
0.994
0.985
0.993
0.945
0.979
Table 20: Absolute Pearson correlations with sestem-level human judgments on WMT18 from- English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.
33

---

Published as conference paper at ICLR 2020
Setting
Metric
cx-en
et-en
nu-en
tr-en 10K
zh
10K
10K
10K
10K
10K
10K
BLEU
0
0.969
0
0
0.972
0.586
0.968
CDER
0.964
0.980
0.988
0.976
0.974
0.577
0.973
CHARACTER
0.960
0.992
0.975
0.979
0.984
0,680
0,942
ITER
0.966
0.990
0,989
0.943
0.742
0.978
METEOR++
0.937
0.990
0.975
0.922
0.989
0.787
0.954
NIST
0.942
0.982
0.980
0.965
0.965
0.862
0.959
Unsupervised
PER
0.937
0.982
0.978
0
0.956
0.043
0,923
TER
0.942
0.970
0,988
0,960
0,963
0,450
0,967
UHK_TSKM
0.943
0.979
0.987
0.974
0.973
0.443
0.972
WER
0.942
0.961
0.989
0,953
0.962
0.072
0.967
YISI-D
0.947
0.982
0.863
0,950
YISI-I
0.942
0.991
0.976
0,964
0.985
0.881
0,943
YISI-1 SRL
0.957
0.994
0.978
0,968
0.986
0.785
0,954
BEER
0,950
0.993
0.983
0.982
0.976
0.723
0.968
Supervised
BLEND RUSE
0.965
0.990
0.982
0,985
0.986
0.724
0,969
0.974
0.988
0.983
0.982
0.780
0.973
PoERT-Bac
0.954
0.992
0.984
0,980
0.970
0.917
0.965
Hamer-toss
0.988
0.994
0.744
0.987
0,988
0.801
0,975
F FutRi-Nae
0.973
0.994
0.981
0.987
0.982
0.924
0.973
PacKI-Ba: (idl)
0.957
0.994
0,983
0,966
0.970
0.875
0,966
Roune--oo (idf)
0.019
0.980
F Fuirt-Kos lidf)
0.974
0.993
0.980
0.978
0.978
0.853
0.976
PeExT Buse MBCC
0.949
0.995
0.986
0,960
0.969
0.832
0.972
RIERT MRFC
0.983
0.997
0.979
0.986
0.986
0.099
0.980
FeEset Besdi-Ba--Mac
0.967
0.997
0.984
0.978
0.981
0.722
0.979
PeEKT Bas MRzC (df)
0.949
0.994
0,986
0.946
0.969
0.743
0,969
(idI)
0.984
0.994
0,980
0,980
0,986
0.541
0,982
F FREat- Mase (idf)
0.967
0.995
0.984
0.968
0.979
0.464
0.978
PackI
0.969
0.991
0.985
0.979
0.970
0.915
0,969
RacxT-Lage
0.990
0.993
0,980
0,988
0.988
0.745
0,978
Fasari-Lang
0.982
0.993
0.984
0.986
0.981
0.909
0,976
PaExI-Laxe (idf)
0.970
0.991
0.984
0
0.971
0.858
0,970
RIERT Lage (idf)
0.989
0.990
0.982
0.982
0.985
0.047
0.982
Faaxx-Lame (idF)
0.981
0.991
0.984
0.976
0.978
0.722
0.978
Po.BERD
0.959
0.992
0.988
0.986
0.971
0.809
0.976
RROBEKI. Bist
0.987
0.997
0.978
0.989
0.988
0.238
0
FRInESTT Base
0.973
0.997
0.987
0,989
0.982
0.674
0,982
PRABERT. Buc (idF)
0.963
0.994
0.988
0.989
0.970
0.711
0.972
TRABERT Bos (idf)
0
0.996
0.979
0.989
0.987
0.353
0
FROMEMIL Bac (idf)
0.976
0.997
0.986
0.990
0.980
0.277
0.980
Pro-Trained
PRoBERTo- Lage
0.965
0.995
0.990
0.976
0.976
0.846
0.975
RAUBERT Larg
0.988
0.981
0.978
0.998
0.989
0.983
0.985
0.760
0.981
PRaBERT Luge (idF)
0.972
0.997
0.988
0.986
0.976
0.686
0.973
RR.sskn--agg Lage (idF)
0.990
0.996
0,983
0,989
0.989
0.096
0,982
Phanesn-tarp (idf)
0.982
0.998
0.988
0.989
0.983
0.453
0.980
0.984
0.977
AIoAEERT- MNL1
0.991
0.996
0.984
0,989
0.987
0.566
0,982
F "aaurats-tagg-1
0.987
0.998
0.989
0.988
0.986
0.873
0.982
PROBERT Lage MNLI (idF)
0.982
0.998
0.992
0.990
0.978
0.812
0.974
RAUBERT Larg- MNLI (idf)
0.992
0,988
0.983
FRaumnn aang (idf)
0.989
0.998
0.990
0.990
0.985
0.583
0.980
0.960
0.997
0.984
0
0.972
0.849
0.974
0.985
0.997
0.975
0.988
0.988
0.303
0.980
Fl NirT-Baa
0.974
0.998
0,981
0,986
0.9822
0.628
0,980
PALEET-Bs (dd)
0.657
0.974
RxINET-Bax (ddD)
0.985
0.996
0.976
0.987
0.987
0.666
0,982
Fx FxI (idf)
0.975
0.996
0.980
0.985
0.981
0.259
0.980
PAINET-Leg:
0.955
0.995
0.983
0.986
0.972
0.875
0.970
HoLNET-Lane
0.984
0.996
0.972
0.984
0.989
0.491
0.975
FXINET-L.es
0.976
PAINEE-Lage (idf)
0.961
0.997
0.983
0.987
0.973
0.816
0.973
Rxa (dd
0.987
0.996
0.975
0.989
0.988
0.320
0.981
PaInii-aag (idf)
0.976
0.997
0.980
0.989
0.982
0.623
0,980
PXLM-En
0.953
0.995
0.988
0.979
0.974
0.918
0.972
RyLM-Ee E-t
FXLM-In
0.969
0.997
0.986
0
0.985
0.869
0.977
PsuM--n (idf)
0.957
0.996
0.987
0.970
0.974
0.862
0.973
RXLM-EL (dd
0.982
0.995
0.981
0.988
0.989
0.213
0,980
PRIM-in (idf)
0.970
0.985
0.519
0.978
Table 21: Absolute Pearson correlations with human judgments on WMT18 to tonEnglish language pairs for 10K hybrid systems. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.
34

---

Published conference paper at ICLR 2020
Setting
Metric
en-cs
en-de
en-et
en-fi
en-ru
en-ir
en-zh
10K
10K
10K
10K
10K
10K
10K
BLEU
0,993
0.977
0.971
0.958
0.977
0.796
0,941
CDER
0.995
0.984
0.981
0.961
0.982
0.832
0.956
CHARACTER
0.990
0.986
0,950
0.963
0.981
0.775
0.978
ITER 0.865
0.978
0.982
0.966
0.965
0.872
METEOR+ 0.997
NIST
0.984
0.980
0.944
0.988
0.870
0,944
Unsupervised
PER
0.987
0.979
0.954
0.904
0.986
0.829
0.950
TER
0.995
0.986
0.977
0.939
0.985
0.837
0.959
UHH TSKM
WER
0.994
0.984
0.977
0.942
0.983
0.824
0.954
YISI-O
0.971
0.983
0.965
0.942
0.988
0.953
0,951
YISI-1
0.985
0.983
0.976
0.938
0.989
0.942
0.957
Y SRL
0.988
0.948
BEER
0.990
0.989
0.978
0.959
0.986
0.933
0.925
Supervised
BLEND RUSE
0.986
PRERT-Moli
0.989
0.983
0.970
0.951
0.988
0.936
0,950
RaERT-Mali
0.995
0.991
0.979
0.977
0.989
0.872
0.980
FBERI-Mati
0.993
0.988
0.978
0.969
0.989
0.910
0.969
PRERI-Mali (idf)
0.992
0.986
0.978
0.954
0.988
0.903
0,950
RBERT-Mati (idf)
0.995
0.988
0.977
0.976
0.987
0.850
0,972
FBERT Muli (idf)
0.995
0.988
0.979
0.969
0.987
0.877
0.963
Pre-Train re-Trained
PXLM-100
0.980
0.990
0.991
0.972
0.991
0.936
0,959
RXIM-100
0.991
0.990
0.989
0.985
0.991
0.882
0.981
FX
0.987
0.990
0.991
0.981
0.991
0.915
0,974
Pa (idl)
0.982
0.990
0.990
0.968
0,991
0.931
0,960
RXLM 100 (idf)
0.989
0.990
0.990
0.985
0.990
0.867
0.978
FXLM 100 (idf)
0.986
0.991
0.991
0.982
0.991
0.905
0.972
Table 22: Absolute Pearson correlations with human judgments on WMT18 from-English language pairs for 10K hybrid systems. Correlations of metries not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.
35

---

Published conference paper at ICLR 2020
Setting
Metric
cs-en
et-en
ru-cB
tr-en
zh-en
BLEU
0.096
0.661
CDER
0.162
0.795
0.764
0,493
0.234
0.087
0.660
CHARACTER
0.146
0.737
0.696
0.496
0.201
0.082
0.584
ITER
0.152
0.814
0.746
0.474
0.234
0.100
0.673
METEOR++
0.052
0.597
NIST
0.136
0.802
0.739
0,469
0.228
0.135
0.665
Unsupervised
PER
0.121
0.764
0.602
0.455
0.218
0.000
0.602
TER
0.139
0.789
0.768
0.470
0.232
0.001
0.652
TSKM
0.642
WER
0.149
0.776
0.760
0,471
0.227
0.000
0.654
YISI-D
0.148
0.780
0.703
0.483
0.229
0.106
0.629
YISI-1
0.157
0.808
0.752
0.456
0.250
0.110
0.613
Y
0.243
0.008
0.620
BEER BLEND RUSE
0.675
Supervised
0.184
0.820
0.779
0,484
0.254
0.003
0.611
0.213
0.823
0.788
0.487
0.250
0.109
0.672
PbERT-Base
0.190
0.815
0.778
0.468
0.261
0.130
0.655
RaEKI Bose
0.189
0.813
0.775
0.481
0.266
0.014
0,663
FEERI- Bas
0.194
0.778
0,474
0.265
0.144
0.670
PaEkI Sac lidf)
0.189
0.817
0.775
0.477
0.255
0.131
0.650
NaExT Base (ddf)
0.192
0,808
0.771
0.484
0.248
0.005
0.674
Baa (idf)
0.193
0.817
0.774
0.483
0.262
0.081
0.669
PeasT Buse MRPC
0.190
0.701
0.766
0.487
0.254
0.126
0.653
HIERT BRP-MRPC
0.199
0.671
Fraar- Row Mirr.
0.197
0.824
0.767
0.491
0.260
0.147
0.668
PEERT.
(idf)
0.186
0.806
0.765
0.492
0.247
0.125
0.661
HAEKT Bose MRFC (idF)
0,200
0.823
0.760
0.495
0.258
0.000
0.680
Pirat- Thrat-Roo-Mar (idf)
0.196
0.821
0.763
0.497
0.254
0.031
0.676
Panxi-Lanp
0.200
0.261
0.661
RacxT-Lagg
0.194
0.809
0.779
0,493
0.270
0.006
0.672
Funkt-
0.199
0.810
0.782
0.484
0.266
0.142
0.672
PSERT Lang (dd
0.200
0.813
0.772
0.485
0.256
0.136
0.657
HaszT-Lagg Lang (ddl)
0.197
0.005
0.675
Fiinr- (idf)
0.199
0.811
0.772
0.494
0.262
0.006
0.673
0.173
0.675
0.757
0.502
0.258
0.126
0.654
0.165
0.816
0.764
0.483
0.266
0.000
0.674
0.173
0.820
0.764
0.498
0.262
0.090
0.669
0.172
0,503
0.661
HRunERn Rooe (ill)
0.172
0.809
0.758
0,490
0.268
0.000
0.678
Faoura (dd
0.178
0,820
0.758
0,501
0.260
0.001
0.674
Pre-Trained
Planeat-Lagge
0.174
0.704
0.765
0.497
0.255
0.140
0.663
0.163
0,805
0.770
0.491
0.263
0.005
0.679
0.675
Plattit--aag (idf) (idf)
0.181
0.821
0.758
0,500
0.256
0.089
0,669
Rgsamen- mg (idD)
0.165
0.787
0.763
0.495
0.270
0.000
0.684
Flannsn-tany (id))
0.179
0.824
0,761
0,502
0.265
0.004
0.679
PReBERR--Les
0.185
0.828
0.780
0,504
0.263
0.133
0.654
HROBERI Lans- MNLI
0.004
0.670
F FRanEnti- MNLI
0.186
0.827
0.778
0.502
0.267
0.113
0.669
PhanEnt Lang (idf)
0.190
0.820
0.771
0.504
0.261
0.102
0.661
RAUBEKI Luns (id))
0.181
0.769
0.766
0.494
0.266
0.004
0.674
Beonskn-Lang-MM (idf)
0.265
0.004
0.671
PALNEV-BUSc Buse
0.186
0.771
0.762
0.496
0.247
0.153
0,658
RxLNET-Baa Buc
0.496
0.671
FkLNET-Ihae
0.186
0.824
0.765
0.499
0.253
0.049
0.673
PALNET-Bus (idf)
0.178
0.819
0,756
0,506
0.241
0.130
0.656
RKLNET Buc (idi)
0.000
0.673
FALNEE-BSE (idf)
0.182
0.821
0.755
0.505
0.250
0.000
0.670
PALNET-Lage
0.195
0.767
0.493
0.152
0.144
0.661
RXLNET-Lage
0.192
0.821
0.766
0.494
0.260
0.001
0.659
FXLNET-Lage
0.196
0.824
0.773
0.496
0.261
0.155
0.675
PALNET-Lage (idf)
0.191
0.811
0.765
0.500
0.167
0,444
0.657
BxLNET-Lage (idf)
0.196
0.495
0.000
0.673
FXINEE-Lagge (idf)
0.195
0,822
0,764
0.499
0.256
0.046
0.674
PaM--n
0.192
0.796
0.779
0.486
0.255
0.131
0.665
RxLM
0.202
0.818
0.772
0.495
0.261
0.005
0.662
FXLM-
0.199
0.827
0.778
0.491
0.262
0.086
0.674
PXLM in (idf)
0.189
0.770
0.485
0.116
0.662
RXLM Ea (idf)
0.202
0.812
0.761
0.490
0.250
0.003
0.668
PaMM-s (idf)
0.196
0.821
0.766
0,490
0.263
0.003
0.672
Table 23: Model selection accuracies (Hits@) 1) on to-English WMT18 hybrid systems. We report
the average of 100K samples and the 0.95 confidence intervals are below 10-3, 3, We bold the highest numbers for each language pair and direction.
36

---

Published as conference paper at ICLR 2020
Setting
Metric
cs-en
et-cn
ru-en
tr-en
zh-en
BLEU
0.265
0.799
CDER
0.362
0.890
0,870
0.689
0,451
0.256
0.799
CHARACTER
0.349
0.854
0.814
0.690
0.429
0.254
0.739
ITER
0.356
0.901
0.856
0.676
0.454
0.278
0,811
METEOR++
0.174
0.757
NIST
0.338
0.894
0.857
0.672
0,446
0.323
0,803
Unsupervised
PER
0.325
0.866
0.771
0.663
0.435
0.021
0.754
TER
0.342
0.885
0,873
0.673
0.447
0.063
0.792
TSKM
0,671
0,063
0.788
WER
0.353
0.876
0.868
0.674
0.443
0.034
0.790
YISI-D
0.344
0.881
0.834
0.681
0.452
0.275
0.776
YISI-1
0.352
0.896
0.864
0.671
0.470
0.285
0.765
Y
0.682
0.464
0.086
0.770
BEER
0.684
0.811
Supervised
BLEND RUSE
0.382
0.904
0.880
0.681
0.473
0.077
0.767
0.417
0.906
0.885
0,686
0.468
0.273
0.809
PbERT-Bae
0.386
0.901
0.880
0.674
0.481
0.318
0.793
RaEKI Bose
0.383
0.899
0.877
0.683
0.486
0.100
0,804
FEERI- Bas
0.388
0.879
0.678
0,484
0.331
0.806
PaekI Sas lidf)
0.390
0.9m2
0.877
0.681
0.475
0.318
0.786
NaExT Base (ddf)
0.390
0.896
0.874
0.686
0.475
0.077
0.811
FuE (idf)
0.393
0.902
0.876
0.685
0.483
0.225
0,806
PeaST Base MRPC
0.392
0.832
0.872
0.686
0.475
0.319
0.791
HIERT BRP-MRPC
0.397
0.811
Fraar- Row Mirr
0.398
0.907
0.872
0.690
0.481
0.335
0.806
PEERT.
(idf)
0.392
0.896
0,870
0.689
0.467
0.316
0.797
HAERT Bose MRFC (idF)
0.400
0.906
0.867
0.691
0.479
0.018
0,817
Pirat- Mare (idf)
0.400
0.905
0.869
0.693
0.475
0.097
0.812
Panxi-Lanp
0.327
0.799
RacxT-Lagg
0.391
0.897
0.879
0.690
0.490
0.085
0.810
Funkt- ang
0.397
0.898
0,882
0.684
0.486
0.328
0.810
PSERT Lang
0.398
0.900
0.875
0.685
0.475
0.323
0.794
Lasz Lang (ddl)
0,692
0,488
0.813
Fimre-- (idf)
0.398
0.899
0.875
0.691
0.482
0.086
0,810
0.372
0.814
0.866
0.697
0.475
0.313
0.795
0.366
0.902
0.870
0.683
0.483
0.026
0.813
0,374
0.904
0,870
0.694
0.480
0.224
0,808
0.802
HRunERn Rooe (dll)
0.374
0.898
0.866
0.688
0,486
0.028
0.816
Faoura (dd
0.380
0.904
0.866
0.696
0.479
0.037
0.812
Pre-Trained
Paanaat-Lagge
0.375
0.833
0.871
0.693
0.474
0.327
0.800
0.366
0.895
0.874
0.689
0,480
0.039
0.816
0.378
0.324
0.811
Plnttit--aags (idf)
(idf)
0,384
0.905
0.866
0.694
0,475
0,230
0,806
Rguames-1 mg (idD)
0.368
0.885
0.869
0.692
0.487
0.030
0,819
Flaunsn-taay (idf)
0,382
0.907
0.868
0.696
0.484
0.048
0.815
PReBERn--Lege
0.383
0.909
0.880
0.698
0.480
0.323
0.795
HROBERI Lanse MNLI
0,481
0.811
F FRanEnts- MNLI
0.385
0.900
0.879
0.697
0,484
0.286
0,809
PhanEnt- Lang Lagg-MNLI (idf)
0.389
0.905
0.874
0.698
0.478
0.268
0.803
RAUBEKI Luns (dd)
0.380
0.874
0.870
0.691
0.483
0.079
0.814
Beonsrn-langg-Me (idf)
0.387
0.906
0.872
0,482
0.082
0.811
PALNET-BIee Buse
0.385
0.875
0,869
0.692
0.469
0.342
0.796
RyLNET-Baa Buc
0.477
0.026
0,809
FRLNET-Ihae
0.385
0.907
0.871
0.694
0.476
0.128
0.810
PALNET-Bus (idf)
0.381
0.904
0.864
0.699
0.464
0.289
0.794
RALNET Buc (idE)
FALNEE-Bee (idf)
0.384
0.905
0.864
0.699
0.472
0.032
0.809
PALNET-Lage
0.392
0.844
0.873
0.689
0.367
0.338
0.799
RXLNET-Lage
0.389
0.905
0.871
0.690
0.482
0.031
0.800
FXLNET-Lag:
0.393
0.907
0.876
0.691
0.483
0.348
0.812
PALNET-Lage (idf)
0.393
0.899
0.870
0.694
0.387
0.333
0,794
LxLNEr-Lage (idf)
0.395
0.868
0
0.483
0.810
FXLNEE-Lagg (idf)
0,396
0.906
0,870
0.693
0.478
0.128
0.811
PaiM-tn
0.394
0.891
0.880
0.685
0.476
0.322
0.802
RxLM
0.401
0.903
0.875
0.692
0.483
0.082
0,803
FXLM-
0.400
0.909
0.878
0.689
0.483
0.234
0.811
PXUM in (idf)
0.391
0.903
0.480
0.293
0.797
RyLM Ea (idf)
0.402
0.900
0.868
0.688
0.477
0.068
0.806
PaIM-n (idf)
0.398
0.905
0.871
0.688
0.487
0.079
0,809
Table 24: Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18
hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10-3, We bold the highest numbers for each language pair and direction.
37

---

Published conference paper at ICLR 2020
Setting
Metric
cS-en
de-en
ct-en
li-en
ru-en
Ir-en
zh-en
BLEU
0.45
1.01
2.17
2.34
CDER
3.88
0.43
0.87
1.33
2.30
4.58
3.43
CHARACTER
3.77
0.49
0.94
2.07
2.25
4.07
3.37
ITER
METEOR+
3.70
0.41
0.69
1.13
2.28
1.40
3.50
NIST
3.93
0.49
1.10
1.19
2.36
1.42
3.92
Unsupervised
PER
2,02
0.46
1.71
1.49
2.25
4.22
3.20
TER
UHH TSKM
3.98
0.40
1.27
1.10
2.23
4.26
3.47
WER
3.85
0.44
1.48
1.18
4.87
5.96
3.72
YISI-D
3.81
0.48
0.72
1.20
1.75
1.40
3.44
YiSI-1
YISI- SRL
3.67
0.41
0.64
1.20
2.15
1.31
3.55
BEER BLEND RUSE
3.82
0.41
0.79
1.08
1.92
1.96
3.43
Supervised
3.77
0.41
0.66
1.09
2.21
1.28
3.46
3.13
0.32
0.64
1.03
1.51
1.94
3.15
PBERT Sas
3.97
0.36
0.72
1.16
2.20
1.25
RagRt Bese
FRERT-Bor
3.70
0.36
0.59
1.08
1.92
1.27
3 3.38
PSEKI (idf)
3.94
0.36
0.64
1.18
2.06
2.55
3.54
REERT Base (ddf)
1.54
0.43
0.63
1.87
1.12
5.96
3.38
FREKT-Boo Rooe (idf)
0.60
Puant-ts-MMur:
TIERR MRPC
2.66
0.43
0.62
1.75
1.10
3.89
0.36
0.60
1.09
1.08
3.82
3.23
PEART Bae MRBC (idf)
4.02
0.35
0.67
1.18
1.48
3.30
3.49
HBEKT Bus MRPC (idi)
T'eerr-uu Row--oo-e MBPC (idl)
3.86
0.38
0.61
1.11
1.14
4.24
3.28
Panxi-Lary
0.34
0.66
1.12
2.10
1.31
3.60
Ragg
1.49
0.40
0.59
1.56
1.17
1.35
3.61
Farzs- ang
1.71
0.35
0,58
1.08
1.65
1.29
3.60
Paunes-Larg
HarRT-Lagg (idl)
1.51
0.42
0.62
1.86
1.10
5,84
3.21
Panne- Gdd
1.49
0.38
0.60
1.17
1.24
1.96
3.53
3.89
0.37
0.75
1.18
1.07
3.45
2.62
1.92
0.39
0.64
1.57
1.11
5.75
3.13
Planiitt Rac (id))
0.38
0.67
1.20
3.27
Reaarkn-Boo (dll)
1.61
0.42
0.67
1.65
1.14
6.55
2.95
Fuammrtnee (dI)
3.18
0.38
0.60
1.11
1.13
6.54
3.11
Pre Ree-Trinee
3.64
0.36
0.71
1.10
1.03
2.69
2.57
RussiihsLLau
Flomnit-tary
2.38
0.35
0.58
1.06
3.57
2.95
Pannnis-Lan (idf)
2.70
0.36
0.69
1.13
1.08
3.18
2.89
ResEEn-Lase (idl)
1.55
0.39
0.66
1.59
1.10
6.66
3.18
0.37
Paonaan--aa XNLI
0.35
0.61
1.07
1.09
RAuBEKI Lams MNLI
FRABERT MNLI
1.42
0.35
0.59
1.07
1.07
1.27
3.41
PacHEnn- Lare-------- (idf)
1.55
0.35
0.60
1.08
1.12
1.54
3.87
RRUSEKI Laue MNLI
(idf)
1.45
0.39
0.64
1.65
1.09
3.32
Flammmn- (idf)
1.42
0.36
0.60
1.10
1.08
3.45
PALNET-Bse Base
1.07
1.16
RXLNET-
1.71
0.45
0.72
1.58
1.07
6.29
3.36
FXLNET-
3.78
0.39
0.62
L.05
1.07
3.60
3.20
PALNET-BaE
3,90
0.46
0.65
3.30
RXLNET-Baz (idf)
FXINET (idf)
3.67
0.42
0.66
1.11
1.22
7.13
3.23
PRINET-Lagge
3.94
0.37
0.71
I.10
21.10
1.85
2.90
RyLNET-Lage
2.23
0.41
0.69
1.34
1.07
4.46
3.40
PaINET-Lag
3.84
0.36
0.60
1,03
1.07
3.38
3.22
PXLNET (idf)
RXLNET- (idf)
1.60
0.43
0.78
1.70
1.09
6.13
3.20
FXLNET-Lage (idf)
3.80
0.38
0.63
1.06
1.09
3.72
3.25
PaMM-t
3.88
0.33
0.75
1.16
2.16
1.28
3.29
RyLM Es.
1.98
0.41
0.60
1.41
1.21
3.30
1.47
FXIM-
ParM-n (idf)
3.84
0.36
0.69
1.17
1.86
1.35
3.47
Rx (idF)
1.70
0.42
0.63
1.55
1.11
5,87
3. 1.36
Fx (idl)
3.72
0.40
0.62
1.14
1.32
4.15
3.43
Table 25: Absolute Difference (x100) of the top metric-rated and the top human-rated system
English WMT18 hybrid systems. Smaller difference signify higher agreement with human scores. We report the average of 100K samples and the 0.95 confidence intervals below 10-3 We bold the lowest numbers for each language pair and direction.
38

---

Published conference paper at ICLR 2020
Setting
Metric
en-es
en-dc
en-et
en-fi
en-tr
en-zh
BLEU 0.151
0.611
0.617
0.087
0.519
0.029
0.515
CDER
0.163
0.663
0.731
0.081
0.541
0.032
0.552
CHARACTER
0.135
0.737
0.639
0.492
0,543
0.027
0.667
ITER
0.000
0.691
0.734
0.112
0.534
0.031
METEOR+
NIST
0.182
0.662
0.549
0.083
0.537
0.033
0.553
Unsupervised
PER
0.179
0.555
0,454
0.062
0.535
0.032
0.539
TER
0.175
0.657
0,550
0.065
0,545
0.029
0,551
UHHLTSKM
WER
0.155
0.643
0.552
0.067
0.538
0.029
0.546
YISI-O
0.154
0.674
0.622
0.356
0.523
0.383
0.600
YISI-
0.178
0.670
0.674
0.230
0,548
0.396
0.595
YISI-1 SRL
0.708
0,537
BEER
0.174
0.670
0.662
0.113
0,555
0.296
0.531
Supervised
BLEND
0.559
RUSE
PblxI-Mali
0.181
0.665
0.771
0.077
0.550
0.373
0.550
Matkt-Mahi
0.184
0.728
0.722
0.146
0.544
0.031
0.657
FBtxI-Muli
0.185
0.703
0.764
0.081
0.548
0.032
0.629
PBERT-Moli (idf)
0.175
0.713
0.769
0,080
0.542
0.031
0.549
RBERT Mulli (idl)
0.177
0.725
0.752
0.178
0.538
0.031
0.628
FBERT Muli (idf)
0.178
0.721
0.766
0.081
0.543
0.030
0.594
Pre-Trace TeaaTrined
PRLM-100
0.175
0.669
0.748
0.079
0.550
0.314
0,582
RXIM-110
0.195
0.671
0.770
0.222
0.555
0.034
0.658
FXLM-100
0.187
0.670
0.775
0.099
0,552
0.034
0.615
PXLM 100 (idf)
0.163
0.664
0.750
0.091
0.550
0.288
0.578
RXLM (idf)
0.191
0.681
0.770
0.231
0.548
0.033
0.645
FoIM-100 (idf)
0.180
0.672
0.774
0.127
0.550
0.033
0.616
Table 26: Model selection accuracies (Hits@ 1) on to-English WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10-3, numbers for each language pair and direction.
We bold the highest
39

---

Published conference paper at ICLR 2020
Setting
Metric
en-es
en-dc
en-et
en-fi
en-tr
en-zh
BLEU
0.363
0.764
0.766
0.323
0.714
0.205
0.666
CDER
0.371
0.803
0.851
0.319
0.729
0.210
0.700
CHARACTER
0,346
0.853
0.781
0.667
0.732
0.205
0.809
ITER
0.044
0.825
0.853
0.365
0.717
0.210
METEOR+
NIST
0.393
0.803
0.710
0.326
0.726
0.211
0.698
Unsupervised
PER
0.387
0.719
0.624
0.301
0.725
0,211
0.678
TER 0.384
0.798
0.708
0.305
0.733
0,209
0.695
UHHLTSKM
WER
0.367
0.787
0.710
0.308
0.728
0.209
0.696
YISI-O
0.370
0.811
0.775
0.553
0.715
0.602
0.753
YISI-
0.390
0.808
0.811
0.439
0.735
0.612
0.750
YISI-1 SRL
0.835
0.691
BEER
0.388
0.808
0.804
0.353
0.739
0.507
0.683
Supervised
BLEND
0.742
RUSE
PblxI-Mali
0.395
0.805
0.876
0.314
0.736
0.586
0.694
Matxt-Mahi
0.401
0.849
0.844
0.368
0.732
0.212
0.802
FBExI-Muli
0,400
0.832
0.872
0.317
0.735
0.214
0.775
PBERT-Moli (idf)
0.390
0.839
0.875
0.320
0.730
0.213
0.691
RBERT Mull (idl)
0.395
0.847
0.864
0.398
0.727
0.212
0.776
FBERT Muli (idf)
0.395
0.844
0.873
0.319
0.730
0.212
0.739
Pre-Trase TeaaTrined
PRLM-100
0.391
0.808
0.862
0.316
0.735
0.522
0,733
RXIM-110
0.413
0.809
0.876
0.435
0.738
0.216
0.803
FXLM-100
0.404
0.809
0.878
0.333
0.737
0.216
0,767
PXLM 100 (idf)
0.377
0.805
0.863
0.326
0.735
0.497
0,729
RXLM (idf)
0.409
0.816
0.876
0.444
0.733
0.214
0.793
FXIM-100 (idf)
0.396
0.810
0.878
0.355
0.735
0.214
0.767
Table 27: Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18
hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10-3, 3 We bold the highest numbers for each language pair and direction.
40

---

Published conference paper at ICLR 2020
Sotting
Metric
en-es
en-de
en-et
en-fi
en-ru
en-zh
BLEU
1.26
6.36
2.59
0.92
0.76
9.40
3.01
CDER
1.25
6.70
1.90
1.41
0.87
9.37
1.75
CHARACTER
1.23
6.90
2.19
4.35
0.93
5.22
1.64
ITER
1.25
9.14
2.52
1.52
1.35
7.33
METEOR++
NIST
1.24
5.28
2.55
1.02
0.75
8.82
3.34
Unsupervised
PER
1,25
6.62
4.92
7.43
0.68
9.76
2.31
TER
1.21
6.02
4.34
2.17
0.73
8.80
1.43
UHH_TSKM
WER
1.22
6.15
4.19
2.43
0.72
9.28
1.49
YISI-O
1.25
6.62
1.53
1.46
0.75
3.47
2.87
YISI-1
1.22
6.27
1.21
1.13
0.71
3.51
3.33
YISI-1 SRL
6.57
3.71
BEER BLEND
1.21
5.96
1.84
0.77
0.74
3.36
1.96
Supervised
0.71
RUSE
PaERI-Mahi
1.17
3.27
1.38
1.24
0.75
4.14
2.08
RaLKE-Muni
1.16
6.68
0.77
0.94
0.68
3.22
1.31
FBERT-Moli
1.15
5.17
0.90
0.98
0.71
3.26
1.62
PBERT-Moli (idf)
1.14
3.82
1.66
1.27
0.76
4.57
2.04
RBERT Muli (idf)
1.15
6.97
0.83
3.65
0.68
3.32
1.37
FRERT-Mali (idf)
1.14
5.63
1.13
1.19
3.38
1.58
Pre-Trained
PXLM-100
1.22
6.30
1.14
0.79
0.74
3.73
2.21
RXLM-100 100
1.18
6.89
0.76
0.77
0.66
3.26
1.68
FXLM
1.19
6.44
0.82
0.76
0.69
3.21
1.57
PXLM-10 (idf)
1.21
6.61
1.07
0.78
0.72
5.59
2.02
M-1 (idf)
1.19
7.07
0.77
0.77
0.66
3.33
1.60
FX (idf)
1.20
6.57
0.86
0.76
3.28
1.56
Table 28: Absolute Difference (x100) of the top metric-rated and the top human-rated system to-
English WMT18 hybrid systems. Smaller difference indicate higher agreement with human scores. We report the average of 100K samples and the 0.95 confidence intervals are below 10-3, We bold the lowest numbers for each language pair and direction.
41

---

Published conference paper at ICLR 2020
Metric
M1
M2
BLEU-1
0.124
0.135
BLEU-2
0.037
0,048
BLEU-3
0.004
0.016"
BLEU-4
-0,019"
-0.005"
ROUGE-L
0.090
0.096
CIDER
0.438
0,440
SPICE
0.759
0.750
LEIC
0.939
0.949
BEER
0.562
EED
0.545
0.599
CHR
0.702
0.729
CHARACTER
0.800
0.801
Passe-
0.313
0.344
RauRT-noa
0.679
0.531
0.519
PRERU-Bae (df
0.243
0.286
RatRi-Sue (dd
0.834
0.783
Pacat-Bos (dd))
0.579
0.581
Rogxi- Secti-Bee-Miun Mine
0.644
0.641
0.470
0.512
Parer
(idl)
0.264
0.300
Roam--na- Min--iun (idf)
0.767
FaERY Rose MRPC (idl)
0.575
0.583
Pannt-targe
0.454
0.486
Ranxi-Lar
0.756
0.697
0.649
0.634
Pmurt-tagg (dd
0.372
BatKT (id()
0.873
0.821
Fong Gill)
0.645
0.647
PRUBERT
-0,223
.0.179
RRABERT- Base
0.827
0.800
PRIBERT Base (idf)
.0.256
0.267
RRoBERTO (dd
0.901
0.869
0.188
0.157
RREEER Lage
00.05
0.041
ReaNERT-
0.888
FRUGEXT Lags
0.322
0.350
PR.BERTO- (idf)
0.063
0.011
Rownern--Lagg (idf)
0.917
0,889
FR.BERT- Lage (idf)
PRaBERT Lage MNLI
0.129
0.208
RRLBERTI MNLI
0.820
0.823
F FRUBEXT Lage
0.546
0.592
P PAsSENTT Lane- MNLL (idf)
RRUBERT Lang SNNI
F FRUBERT
(idf)
0.605
0.596
PaiSa-Bas:
-0.046
0.080
RyLNes Basc
0.409
0.506
FxINe-Base
0.265
PauNa- Buse (idf)
0.006
0.145
HyLNs-Ba (id()
0.655
0.720
FxiNa--tax: (idf)
0.270
0.391
PaLNa-Lage
-0.188
00.155
RxiNa-tang
FxLNs-Lay:
-0.014
0.036
Pano--Lase (idl)
-0.186
.0072
Rxa Na-pary (id
0.554
0.555
FaLNe-Lage (idf)
0.151
0.234
Pa
RxIM-t
0.333
0.263
FaIM-
0.297
0.243
Px (id
0.266
0.275
Raax-n
0.700
0.640
FSLM-E (id[)
0.499
0.470
Table 29: Pearson correlation on the 2015 COCO Captioning Challenge. The M1 and M2 measures described in Section 4 We bold the best correlating task-specific and task-agnostic metrics in each setting LEIC uses images as additional inputs. Numbers with are cited from Cui et al. (018))
42

---

Published conference paper at ICLR 2020
Type
Method
QQP
PAWSoor
DecAIL
0.939
0.263
Trained QQP (supervised)
DIIN
0.952
0.324
BERT
0.963
0.351
DecAn
Trained on QQP PAW (supervised)
DINN
0.778
BERT
0.831
BLEU-1
0.737
0.402
BLEU-2
0.720
0.548
BLEU-3
BLEU-4
0.707
0.527
METEOR
0.755
0.532
ROUGE-L
0,740
0.536
BEER
0.741
0.564
EED
0.743
0.611
CHARACTER
0.698
0,650
PasKi-Saa
0,750
0.654
Hauxe-Bus Bowe
F FunKi-Ite:
0.755
0.654
ParRT-Bow (idl)
0.766
0.665
RISRT-Ba (dd
0.752
0.665
Fmuxi-tose (idf)
0.770
0.664
PaExT-BBuse Mur--MrCC
0.742
0.615
HarRi-Rae-Maee MRYC
0.729
0.617
0.746
0.614
PhEst
(idl)
0.752
0.618
Ramer-reas- Sannr--ee--MRI MRRC
0.619
F Fuean-nu Mer- MuPC. (idf)
0.756
0.617
Panki-Laxe
0,752
0,706
RBERT-Laxe
0.740
0.710
Fanxi-Lang
0.756
0.707
Peast Lage (idf)
0.766
0.713
Rmmm-saage (idi)
0.751
0.718
Finsx--Lag (idf)
0.769
0,714
Metric (Not trained on QQP or PAWSqge)
PaLBERI
0.746
0.657
REuBEKI
0.736
0.656
0.751
0.654
PhaBERI Sac (idf)
RRoBEKI Bue (idf)
0.745
0.666
FRouERn Sac (idf)
0.765
0,664
PROBEKI
0.757
0.687
RRUBEKT Lage
0.744
0.685
Phonekt-
0,761
0.685
PROBEKI Lags (idf]
0.773
0.691
RR.BEXT (idf)
0.757
0.697
0.777
0.693
PRoBERTo Lage MNLI
0.763
0.767
HRuBERTe Lage MNLI
Fhomet MNI
0.766
0.770
PEABERT
(idF)
0.783
0.756
TROSERS Lage MN2I (dff)
0.767
0.764
FhomERT-t -aagg---- (idf)
0.737
RxLNa-Baa Bose
0.731
0.607
Fe
0.739
0.605
PAUNe Bac (idf)
0.751
0.625
ToaNa Base
Fat No-Nas (idf)
0.751
0.626
PaiNa-Larp
0.742
0.593
Range-tage
0.734
0.598
Fia Na-t ang
0.744
0.596
Paisa-Lar
Hx-Nest (ddD)
0.749
0.610
Fa (id
0.760
0.606
PsAM-in
0.734
0.600
RXLaS-Ea
0.725
0.604
FxIM-im
0.602
РaLM-i (df))
0.757
0,596
RaaM- (idl)
0.745
0.603
FaIM-nn (idf)
0.759
0.600
Table 30: Area under ROC curve (AUC) QQP and PAWS datasets. The scores of trained DecATT (Parikh et al. al.] 2016). DIIN (Gong et al. 2018). and fine-tuned BERT are reported by Zhang al. (2019). We bold the best task-specific and task-agnostic metrics. Numbers with scores on the held-out test set of QQP.
43


Reviews:
[{"review_id": "S1lNbqHjKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n*** Update ***\nI'd like to thank the authors for answering my questions, and I am satisfied with their response. I have read the other reviews for this paper as well, and I am keeping my score.\n\n\nThis paper proposes BERTScore, a method for automatic evaluation of text. Their method uses BERT to produce contextualized word representations for the words in the reference and hypothesis. Then they compute the precision, recall, and F1 by greedily matching up words between the hypothesis and reference. To be more specific, for say recall they take each word in the reference and compute the cosine with all words in the hypothesis. Then they add up the largest cosine similarity for each word and average them together. Precision is defined similarly but with the roles of hypothesis and reference switched. F1 is then the harmonic mean of these two scores. They also experiment with using idf to weight importance.\n\nTheir method is simple, but achieves very strong results and there are a ton of experiments in this paper (it is 41 pages). The focus is largely on metrics for MT, but they also evaluate on image captioning. The paper is also very thorough and many of the questions I had when reading it are answered (like effect of optimal matching, running time, etc.). The latter (running time) being one of the downsides of the method if it was to be used for fine-tuning MT systems. 40 times slower than BLEU, but I think this increased cost would be worth it and could be engineered around.\n\nOverall, I like the paper - it is simple and effective on its goal task of automatic evaluation for text generation. I think we are moving that way as a field and this paper proposes a useful method and is additionally a good study on the subject.\n\nA question I have is why the method doesn't perform well in certain cases. For instance, in Table 2 and 3 - some of the evaluations with tr and fi fall well below relative performance for other language pairs. Does this have to do with the quality of the representations in multilingual BERT? What is YiSi-1 doing, for instance for model selection of en-fi and en-tr that makes it have so much better performance?\n\nEdit: I also wonder if incorporating idf would be better if the values were computed by a larger corpus. I think it would make the most sense to compute these from the training data for the underlying BERT models. Since BERT itself is a function of this training data, it seems appropriate that these values would be as well (or perhaps at least a subset of this data).\n\nMissing citations:\nA citation to \"Beyond BLEU:Training Neural Machine Translation with Semantic Similarity\" from ACL 2019 should be incorporated into the related work. They use semantic similarity to fine-tune NMT systems with their own embedding-based (semantic similarity) metric and they found some nice properties from training in this way. Have you tried BERTScore on sentence similarity tasks? It's possible BERTScore could have strong performance and some readers may wonder this. There are evaluations on PAWS for paraphrase detection which I appreciated, but that is a little different.\n\nA citation to \"Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization\" from EMNLP 2019 should also be incorporated. This paper is a big boon to BERT score showing that it is a very helpful metric for fine-tuning summarization systems. They don't even need a cross-entropy term since BERTScore captures fluency so well. I'd like to see it for MT as well, but perhaps that is the next paper.\n\nTypos:\nThe word \"language\" is misspelled twice in Appendix E.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgQNGqJ9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Contributions\n\nThis paper introduces a new text generation scoring approach using BERT, called BERTScore. Using BERT embeddings and optionally idf scores, a greedy matching is performed between all reference and candidate words, with cosine similarity between vector representations as the scoring. From this, a precision, recall and F1 score can be derived. This notably outperforms BLEU, as well as other metrics, most but not all of the time. The paper offers a broad range of comparisons and analysis.\n\nDecision\n\nI'm leaning towards accepting the paper on the basis of the following.\n\nStrong points taken in consideration:\n- Simple, well-motivated metric that uses powerful BERT-style models, without being slow to compute either.\n- Good performance empirically on WMT. I'm less convinced on COCO since using the image is fair game there.\n- Code is provided, and it is simple and adaptable for future work.\n- Experimentation is detailed and reproducible.\n\nWeaker points taken in consideration:\n- Work conducted in parallel matches or exceeds the performance of BERTScore. This shouldn't necessarily be a reason to choose not to publish this work in my opinion, but it should be taken into consideration. I like that the authors were open and clear regarding this in their discussion.\n- The authors haven't come up with a recommendation for a single configuration of their approach. In one place they recommend F-BERT without idf, in another they argue for picking and choosing based on context, with little help about how to choose. I think practitioners are only going to be willing to switch away from BLEU, for example, if a single one-size-fits-all metric is proposed instead. I identify this ambiguity between BERTScore versions as an important weakness of the paper.\n- It's unclear throughout whether words or wordpieces are the main token being considered. Most discussion and definitions use \"words\", but in section 3, subsection Token Representation, it appears to be clearly stated that BERTScore uses a BERT model based on word pieces. I recommend adjust the language to be more consistent throughout. Also, scoring examples with word pieces would be more consistent with this as well, imo. Notably, I'm actually unsure whether you compute IDF over words or word pieces, and how this is applied. \n- Finally, I found some weaknesses in the Importance Weighting section (though this isn't too important since IDF isn't part of the recommended BERTScore I believe). The IDF scores would be stronger if they were computed on a bigger in-domain corpus than the gold test set. This would add extra steps to using BERTScore though and make things more complicated in practice, but this should nevertheless probably be tried, or at least discussed in the paper. Also, the plus-one smoothing handles unknown words (or word piece?) and I'm not sure why. If we're using the test set to compute IDF, and the sentences we're looking *are* in the test set, then there shouldn't be unknown words and no smoothing is required.\n\nSo overall, I still think this deserves publication because it's valuable information for researchers, and the metric itself could be immediately useful to some as well. However, the weaknesses mentioned make me hesitate to fully endorse the work.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlIpZuCKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a simple application of BERT-based contextual embeddings for evaluation of text generation models such as machine translation and image caption generation. An extensive set of experiments have been carried out to show that the proposed BERTScore metric achieves better correlation with human judgments than the existing metrics. Overall, the paper is well-written and the motivations are clear. However, I am not sure about the technical novelty of the paper as the proposed approach is a natural application of BERT along with traditional cosine similarity measures and precision, recall, F1-based computations, and simple IDF-based importance weighting. \n\nOther comments:\n\n- It would be interesting to see how the proposed metric performs to evaluate paraphrase generation and text simplification models as the models need to follow specific constraints such as semantic equivalence, novelty, simplicity etc. with respect to source and reference sentences.\n\n- Another limitation of the proposed metric is memory and time complexity as it takes relatively more time to evaluate the sentences compared to BLEU, as authors acknowledged in Section 5. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for an interesting discussion. The authors present a supposedly task-independent evaluation metric for generation tasks with references that relies on BERT or similar pretrained language models and a BERT-internal alignment. Reviewers are moderately positive. I encourage the authors to think about a) whether their approach scales to language pairs where wordpieces are less comparable; b) whether second order similarly, e.g., using RSA, would be better than alignment-based similarity; c) whether this metric works in the extremes, e.g., can it distinguish between bad output and super-bad output (where in both cases alignment may be impossible), and can it distinguish between good output and super-good output (where BERT scores may be too biased by BERT's training objective). 
Decision: Accept (Poster)