Title: BERTSCORE: EVALUATING TEXT GENERATION WITH BERT

Abstract: We propose BERTSCORE, an automatic evaluation metric for text generation. Analogously to common metrics, BERTSCORE computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTSCORE correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTSCORE is more robust to challenging examples when compared to existing metrics.

1. INTRODUCTION:

Automatic evaluation of natural language generation, for example in machine translation and caption generation, requires comparing candidate sentences to annotated references. The goal is to evaluate semantic equivalence. However, commonly used methods rely on surface-form similarity only. For example, BLEU (Papineni et al., 2002) , the most common machine translation metric, simply counts n-gram overlap between the candidate and the reference. While this provides a simple and general measure, it fails to account for meaning-preserving lexical and compositional diversity.

In this paper, we introduce BERTSCORE, a language generation evaluation metric based on pretrained BERT contextual embeddings (Devlin et al., 2019) . BERTSCORE computes the similarity of two sentences as a sum of cosine similarities between their tokens' embeddings. BERTSCORE addresses two common pitfalls in n-gram-based metrics (Banerjee & Lavie, 2005) . First, such methods often fail to robustly match paraphrases. For example, given the reference people like foreign cars, BLEU and METEOR (Banerjee & Lavie, 2005) incorrectly give a higher score to people like visiting places abroad compared to consumers prefer imported cars. This leads to performance underestimation when semantically-correct phrases are penalized because they differ from the surface form of the reference. In contrast to string matching (e.g., in BLEU) or matching heuristics (e.g., in METEOR), we compute similarity using contextualized token embeddings, which have been shown to be effective for paraphrase detection (Devlin et al., 2019) . Second, n-gram models fail to capture distant dependencies and penalize semantically-critical ordering changes (Isozaki et al., 2010) . For example, given a small window of size two, BLEU will only mildly penalize swapping of cause and effect clauses (e.g. A because B instead of B because A), especially when the arguments A and B are long phrases. In contrast, contextualized embeddings are trained to effectively capture distant dependencies and ordering.

We experiment with BERTSCORE on machine translation and image captioning tasks using the outputs of 363 systems by correlating BERTSCORE and related metrics to available human judgments. Our experiments demonstrate that BERTSCORE correlates highly with human evaluations. In machine translation, BERTSCORE shows stronger system-level and segment-level correlations with human judgments than existing metrics on multiple common benchmarks and demonstrates strong model selection performance compared to BLEU. We also show that BERTSCORE is well-correlated with human annotators for image captioning, surpassing SPICE, a popular taskspecific metric (Anderson et al., 2016) . Finally, we test the robustness of BERTSCORE on the adversarial paraphrase dataset PAWS (Zhang et al., 2019) , and show that it is more robust to adversarial examples than other metrics. The code for BERTSCORE is available at https://github.com/Tiiiger/bert score.

2. PROBLEM STATEMENT AND PRIOR METRICS:

Natural language text generation is commonly evaluated using annotated reference sentences. Given a reference sentence x tokenized to k tokens ⟨x 1 , . . . , x k ⟩ and a candidate x tokenized to l tokens ⟨x 1 , . . . , xl ⟩, a generation evaluation metric is a function f (x, x) ∈ R. Better metrics have a higher correlation with human judgments. Existing metrics can be broadly categorized into using n-gram matching, edit distance, embedding matching, or learned functions.

2.1. n-GRAM MATCHING APPROACHES:

The most commonly used metrics for generation count the number of n-grams that occur in the reference x and candidate x. The higher the n is, the more the metric is able to capture word order, but it also becomes more restrictive and constrained to the exact form of the reference.

Formally, let S n

x and S n x be the lists of token n-grams (n ∈ Z + ) in the reference x and candidate x sentences. The number of matched n-grams is ∑

w∈S n

x I[w ∈ S n x ], where I[•] is an indicator function. The exact match precision (Exact-P n ) and recall (Exact-R n ) scores are:

Exact-P n = ∑ w∈S n x I[w ∈ S n x ] |S n x | and Exact-R n = ∑ w∈S n x I[w ∈ S n x ] |S n x |

.

Several popular metrics build upon one or both of these exact matching scores.

BLEU The most widely used metric in machine translation is BLEU (Papineni et al., 2002) , which includes three modifications to Exact-P n . First, each n-gram in the reference can be matched at most once. Second, the number of exact matches is accumulated for all reference-candidate pairs in the corpus and divided by the total number of n-grams in all candidate sentences. Finally, very short candidates are discouraged using a brevity penalty. Typically, BLEU is computed for multiple values of n (e.g. n = 1, 2, 3, 4) and the scores are averaged geometrically. A smoothed variant, SENT-BLEU (Koehn et al., 2007 ) is computed at the sentence level. In contrast to BLEU, BERTSCORE is not restricted to maximum n-gram length, but instead relies on contextualized embeddings that are able to capture dependencies of potentially unbounded length.

METEOR METEOR (Banerjee & Lavie, 2005) computes Exact-P 1 and Exact-R 1 while allowing backing-off from exact unigram matching to matching word stems, synonyms, and paraphrases. For example, running may match run if no exact match is possible. Non-exact matching uses an external stemmer, a synonym lexicon, and a paraphrase table. METEOR 1.5 (Denkowski & Lavie, 2014) weighs content and function words differently, and also applies importance weighting to different matching types. The more recent METEOR++ 2.0 (Guo & Hu, 2019) further incorporates a learned external paraphrase resource. Because METEOR requires external resources, only five languages are supported with the full feature set, and eleven are partially supported. Similar to METEOR, BERTSCORE allows relaxed matches, but relies on BERT embeddings that are trained on large amounts of raw text and are currently available for 104 languages. BERTSCORE also supports importance weighting, which we estimate with simple corpus statistics.

Other Related Metrics NIST (Doddington, 2002) is a revised version of BLEU that weighs each n-gram differently and uses an alternative brevity penalty. ∆BLEU (Galley et al., 2015) modifies multi-reference BLEU by including human annotated negative reference sentences. CHRF (Popović, 2015) compares character n-grams in the reference and candidate sentences. CHRF++ (Popović, 2017) extends CHRF to include word bigram matching. ROUGE (Lin, 2004 ) is a commonly used metric for summarization evaluation. ROUGE-n (Lin, 2004) computes Exact-R n (usually n = 1, 2), while ROUGE-L is a variant of Exact-R 1 with the numerator replaced by the length of the longest common subsequence. CIDER (Vedantam et al., 2015) is an image captioning metric that computes cosine similarity between tf-idf weighted n-grams. We adopt a similar approach to weigh tokens differently. Finally, Chaganty et al. (2018) and Hashimoto et al. (2019) combine automatic metrics with human judgments for text generation evaluation.

2.2. EDIT-DISTANCE-BASED METRICS:

Several methods use word edit distance or word error rate (Levenshtein, 1966) , which quantify similarity using the number of edit operations required to get from the candidate to the reference. TER (Snover et al., 2006) normalizes edit distance by the number of reference words, and ITER (Panja & Naskar, 2018) adds stem matching and better normalization. PER (Tillmann et al., 1997) computes position independent error rate, CDER (Leusch et al., 2006 ) models block reordering as an edit operation. CHARACTER (Wang et al., 2016) and EED (Stanchev et al., 2019) operate on the character level and achieve higher correlation with human judgements on some languages.

2.3. EMBEDDING-BASED METRICS:

Word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Grave et al., 2018; Nguyen et al., 2017; Athiwaratkun et al., 2018) are learned dense token representations. MEANT 2.0 (Lo, 2017) uses word embeddings and shallow semantic parses to compute lexical and structural similarity. YISI-1 (Lo et al., 2018) is similar to MEANT 2.0, but makes the use of semantic parses optional. Both methods use a relatively simple similarity computation, which inspires our approach, including using greedy matching (Corley & Mihalcea, 2005) and experimenting with a similar importance weighting to YISI-1. However, we use contextual embeddings, which capture the specific use of a token in a sentence, and potentially capture sequence information. We do not use external tools to generate linguistic structures, which makes our approach relatively simple and portable to new languages. Instead of greedy matching, WMD (Kusner et al., 2015) , WMD O (Chow et al., 2019) , and SMS (Clark et al., 2019) propose to use optimal matching based on earth mover's distance (Rubner et al., 1998) . The tradeoff1 between greedy and optimal matching was studied by Rus & Lintean (2012) . Sharma et al. (2018) compute similarity with sentence-level representations. In contrast, our token-level computation allows us to weigh tokens differently according to their importance.

2.4. LEARNED METRICS:

Various metrics are trained to optimize correlation with human judgments. BEER (Stanojević & Sima'an, 2014) uses a regression model based on character n-grams and word bigrams. BLEND (Ma et al., 2017) uses regression to combine 29 existing metrics. RUSE (Shimanaka et al., 2018) combines three pre-trained sentence embedding models. All these methods require costly human judgments as supervision for each dataset, and risk poor generalization to new domains, even within a known language and task (Chaganty et al., 2018) . Cui et al. (2018) and Lowe et al. (2017) train a neural model to predict if the input text is human-generated. This approach also has the risk of being optimized to existing data and generalizing poorly to new data. In contrast, the model underlying BERTSCORE is not optimized for any specific evaluation task.

3. BERTSCORE:

Given a reference sentence x = ⟨x 1 , . . . , x k ⟩ and a candidate sentence x = ⟨x 1 , . . . , xl ⟩, we use contextual embeddings to represent the tokens, and compute matching using cosine similarity, optionally weighted with inverse document frequency scores. Figure 1 illustrates the computation.

Token Representation We use contextual embeddings to represent the tokens in the input sentences x and x. In contrast to prior word embeddings (Mikolov et al., 2013; Pennington et al., 2014) , contextual embeddings, such as BERT (Devlin et al., 2019) and ELMO (Peters et al., 2018) , can generate different vector representations for the same word in different sentences depending on the surrounding words, which form the context of the target word. The models used to generate these embeddings are most commonly trained using various language modeling objectives, such as masked word prediction (Devlin et al., 2019) . We experiment with different models (Section 4), using the tokenizer provided with each model. Given a tokenized reference sentence x = ⟨x 1 , . . . , x k ⟩, the embedding model generates a sequence of vectors ⟨x 1 , . . . , x k ⟩. Similarly, the tokenized candidate x = ⟨x 1 , . . . , xm ⟩ is mapped to ⟨x 1 , . . . , xl ⟩. The main model we use is BERT, which tokenizes the input text into a sequence of word pieces (Wu et al., 2016) , where unknown words are split into several commonly observed sequences of characters. The representation for each word piece is computed with a Transformer encoder (Vaswani et al., 2017) by repeatedly applying self-attention and nonlinear transformations in an alternating fashion. BERT embeddings have been shown to benefit various NLP tasks (Devlin et al., 2019; Liu, 2019; Huang et al., 2019; Yang et al., 2019a) .

f H R q N I = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 2 I J / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 H 1 F p H s t 7 M 0 n Q j + h Q 8 p A z a q z U e O q X K 2 7 V n Y O s E i 8 n F c h R 7 5 e / e o O Y p R F K w w T V u u u 5 i f E z q g x n A q e l X q o x o W x M h 9 i 1 V N I I t Z / N D 5 2 S M 6 s M S B g r W 9 K Q u f p 7 I q O R 1 p M o s J 0 R N S O 9 7 M 3 E / 7 x u a s J r P + M y S Q 1 K t l g U p o K Y m M y + J g O u k B k x s Y Q y x e 2 t h I 2 o o s z Y b E o 2 B G / 5 5 V X S u q h 6 b t V r X F Z q N 3 k c R T i B U z g H D 6 6 g B n d Q h y Y w Q H i G V 3 h z H p w X 5 9 3 5 W L Q W n H z m G P 7 A + f w B 5 j m M / A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 y z i m w b R / D g j z p 6 t Z 3 6 0 f H R q N I = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 2 I J / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 H 1 F p H s t 7 M 0 n Q j + h Q 8 p A z a q z U e O q X K 2 7 V n Y O s E i 8 n F c h R 7 5 e / e o O Y p R F K w w T V u u u 5 i f E z q g x n A q e l X q o x o W x M h 9 i 1 V N I I t Z / N D 5 2 S M 6 s M S B g r W 9 K Q u f p 7 I q O R 1 p M o s J 0 R N S O 9 7 M 3 E / 7 x u a s J r P + M y S Q 1 K t l g U p o K Y m M y + J g O u k B k x s Y Q y x e 2 t h I 2 o o s z Y b E o 2 B G / 5 5 V X S u q h 6 b t V r X F Z q N 3 k c R T i B U z g H D 6 6 g B n d Q h y Y w Q H i G V 3 h z H p w X 5 9 3 5 W L Q W n H z m G P 7 A + f w B 5 j m M / A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 y z i m w b R / D g j z p 6 t Z 3 6 0 f H R q N I = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 2 I J / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 H 1 F p H s t 7 M 0 n Q j + h Q 8 p A z a q z U e O q X K 2 7 V n Y O s E i 8 n F c h R 7 5 e / e o O Y p R F K w w T V u u u 5 i f E z q g x n A q e l X q o x o W x M h 9 i 1 V N I I t Z / N D 5 2 S M 6 s M S B g r W 9 K Q u f p 7 I q O R 1 p M o s J 0 R N S O 9 7 M 3 E / 7 x u a s J r P + M y S Q 1 K t l g U p o K Y m M y + J g O u k B k x s Y Q y x e 2 t h I 2 o o s z Y b E o 2 B G / 5 5 V X S u q h 6 b t V r X F Z q N 3 k c R T i B U z g H D 6 6 g B n d Q h y Y w Q H i G V 3 h z H p w X 5 9 3 5 W L Q W n H z m G P 7 A + f w B 5 j m M / A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 y z i m w b R / D g j z p 6 t Z 3 6 0 f H R q N I = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 2 I J / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 H 1 F p H s t 7 M 0 n Q j + h Q 8 p A z a q z U e O q X K 2 7 V n Y O s E i 8 n F c h R 7 5 e / e o O Y p R F K w w T V u u u 5 i f E z q g x n A q e l X q o x o W x M h 9 i 1 V N I I t Z / N D 5 2 S M 6 s M S B g r W 9 K Q u f p 7 I q O R 1 p M o s J 0 R N S O 9 7 M 3 E / 7 x u a s J r P + M y S Q 1 K t l g U p o K Y m M y + J g O u k B k x s Y Q y x e 2 t h I 2 o o s z Y b E o 2 B G / 5 5 V X S u q h 6 b t V r X F Z q N 3 k c R T i B U z g H D 6 6 g B n d Q h y Y w Q H i G V 3 h z H p w X 5 9 3 5 W L Q W n H z m G P 7 A + f w B 5 j m M / A = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Q T n V R V S r n y z z n V U 7 d 5 b F 5 u 0 3 I w = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 9 z v P H J t R K w e c J p w P 6 I j J U L B K F q p 0 x 9 T z J 5 m g 2 r N r b t z k F X i F a Q G B Z q D 6 l d / G L M 0 4 g q Z p M b 0 P D d B P 6 M a B Z N 8 V u m n h i e U T e i I 9 y x V N O L G z + b n z s i Z V Y Y k j L U t h W S u / p 7 I a G T M N A p s Z 0 R x b J a 9 X P z P 6 6 U Y X v u Z U E m K X L H F o j C V B G O S / 0 6 G Q n O G c m o J Z V r Y W w k b U 0 0 Z 2 o Q q N g R v + e V V 0

8 G H u / N M D M v S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 9 z v P H J t R K w e c J p w P 6 I j J U L B K F q p 0 x 9 T z J 5 m g 2 r N r b t z k F X i F a Q G B Z q D 6 l d / G L M 0 4 g q Z p M b 0 P D d B P 6 M a B Z N 8 V u m n h i e U T e i I 9 y x V N O L G z + b n z s i Z V Y Y k j L U t h W S u / p 7 I a G T M N A p s Z 0 R x b J a 9 X P z P 6 6 U Y X v u Z U E m K X L H F o j C V B G O S / 0 6 G Q n O G c m o J Z V r Y W w k b U 0 0 Z 2 o Q q N g R v + e V V 0

8 G H u / N M D M v S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 9 z v P H J t R K w e c J p w P 6 I j J U L B K F q p 0 x 9 T z J 5 m g 2 r N r b t z k F X i F a Q G B Z q D 6 l d / G L M 0 4 g q Z p M b 0 P D d B P 6 M a B Z N 8 V u m n h i e U T e i I 9 y x V N O L G z + b n z s i Z V Y Y k j L U t h W S u / p 7 I a G T M N A p s Z 0 R x b J a 9 X P z P 6 6 U Y X v u Z U E m K X L H F o j C V B G O S / 0 6 G Q n O G c m o J Z V r Y W w k b U 0 0 Z 2 o Q q N g R v + e V V 0

i Z V Y Y k j L U t h W S u / p 7 I a G T M N A p s Z 0 R x b J a 9 X P z P 6 6 U Y X v u Z U E m K X L H F o j C V B G O S / 0 6 G Q n O G c m o J Z V r Y W w k b U 0 0 Z 2 o Q q N g R v + e V V 0

Similarity Measure The vector representation allows for a soft measure of similarity instead of exact-string (Papineni et al., 2002) or heuristic (Banerjee & Lavie, 2005) matching. The cosine similarity of a reference token x i and a candidate token xj is

x ⊤ i xj ∥xi∥∥xj ∥ .

We use pre-normalized vectors, which reduces this calculation to the inner product x ⊤ i xj . While this measure considers tokens in isolation, the contextual embeddings contain information from the rest of the sentence.

The complete score matches each token in x to a token in x to compute recall, and each token in x to a token in x to compute precision. We use greedy matching to maximize the matching similarity score,2 where each token is matched to the most similar token in the other sentence. We combine precision and recall to compute an F1 measure. For a reference x and candidate x, the recall, precision, and F1 scores are:

R BERT = 1 |x| ∑ xi∈x max xj ∈x x ⊤ i xj , P BERT = 1 |x| ∑ xj ∈x max xi∈x x ⊤ i xj , F BERT = 2 P BERT • R BERT P BERT + R BERT .

Importance Weighting Previous work on similarity measures demonstrated that rare words can be more indicative for sentence similarity than common words (Banerjee & Lavie, 2005; Vedantam et al., 2015) . BERTSCORE enables us to easily incorporate importance weighting. We experiment with inverse document frequency (idf) scores computed from the test corpus. Given M reference sentences {x (i) } M i=1 , the idf score of a word-piece token w is

idf(w) = -log 1 M M ∑ i=1 I[w ∈ x (i) ] ,

where I[•] is an indicator function. We do not use the full tf-idf measure because we process single sentences, where the term frequency (tf) is likely 1. For example, recall with idf weighting is

R BERT = ∑ xi∈x idf(x i ) max xj ∈x x ⊤ i xj ∑ xi∈x idf(x i )

.

Because we use reference sentences to compute idf, the idf scores remain the same for all systems evaluated on a specific test set. We apply plus-one smoothing to handle unknown word pieces.

Baseline Rescaling Because we use pre-normalized vectors, our computed scores have the same numerical range of cosine similarity (between -1 and 1). However, in practice we observe scores in a more limited range, potentially because of the learned geometry of contextual embeddings. While this characteristic does not impact BERTSCORE's capability to rank text generation systems, it makes the actual score less readable. We address this by rescaling BERTSCORE with respect to its empirical lower bound b as a baseline. We compute b using Common Crawl monolingual datasets. 3 For each language and contextual embedding model, we create 1M candidate-reference pairs by grouping two random sentences. Because of the random pairing and the corpus diversity, each pair has very low lexical and semantic overlapping. 4 We compute b by averaging BERTSCORE computed on these sentence pairs. Equipped with baseline b, we rescale BERTSCORE linearly. For example, the rescaled value RBERT of R BERT is:

RBERT = R BERT -b 1 -b .

After this operation RBERT is typically between 0 and 1. We apply the same rescaling procedure for P BERT and F BERT . This method does not affect the ranking ability and human correlation of BERTSCORE, and is intended solely to increase the score readability.

4. EXPERIMENTAL SETUP:

We evaluate our approach on machine translation and image captioning.

Contextual Embedding Models:

We evaluate twelve pre-trained contextual embedding models, including variants of BERT (Devlin et al., 2019) , RoBERTa (Liu et al., 2019b) , XLNet (Yang et al., 2019b) , and XLM (Lample & Conneau, 2019) . We present the best-performing models in Section 5. We use the 24-layer RoBERTa large model5 for English tasks, 12-layer BERT chinese model for Chinese tasks, and the 12-layer cased multilingual BERT multi model for other languages. 6 We show the performance of all other models in Appendix F. Contextual embedding models generate embedding representations at every layer in the encoder network. Past work has shown that intermediate layers produce more effective representations for semantic tasks (Liu et al., 2019a) . We use the WMT16 dataset (Bojar et al., 2016) as a validation set to select the best layer of each model (Appendix B).

Machine Translation:

Our main evaluation corpus is the WMT18 metric evaluation dataset (Ma et al., 2018) , which contains predictions of 149 translation systems across 14 language pairs, gold references, and two types of human judgment scores. Segment-level human judgments assign a score to each reference-candidate pair. System-level human judgments associate each system with a single score based on all pairs in the test set. WMT18 includes translations from English to Czech, German, Estonian, Finnish, Russian, and Turkish, and from the same set of languages to English. We follow the WMT18 standard practice and use absolute Pearson correlation |ρ| and Kendall rank correlation τ to evaluate metric quality, and compute significance with the Williams test (Williams, 1959) for |ρ| and bootstrap re-sampling for τ as suggested by Graham & Baldwin (2014) . We compute systemlevel scores by averaging BERTSCORE for every reference-candidate pair. We also experiment with hybrid systems by randomly sampling one candidate sentence from one of the available systems for each reference sentence (Graham & Liu, 2016 ). This enables system-level experiments with a higher number of systems. Human judgments of each hybrid system are created by averaging the WMT18 segment-level human judgments for the corresponding sentences in the sampled data. We compare BERTSCOREs to one canonical metric for each category introduced in Section 2, and include the comparison with all other participating metrics from WMT18 in Appendix F.

In addition to the standard evaluation, we design model selection experiments. We use 10K hybrid systems super-sampled from WMT18. We randomly select 100 out of 10K hybrid systems, and rank them using the automatic metrics. We repeat this process 100K times. We report the percentage of the metric ranking agreeing with the human ranking on the best system (Hits@1). In Tables 23 24 25 26 27 28 , 1 : Absolute Pearson correlations with system-level human judgments on WMT18. For each language pair, the left number is the to-English correlation, and the right is the from-English. We bold correlations of metrics not significantly outperformed by any other metric under Williams Test for that language pair and direction. The numbers in parenthesis are the number of systems used for each language pair and direction. Table 2 : Absolute Pearson correlations with system-level human judgments on WMT18. We use 10K hybrid super-sampled systems for each language pair and direction. For each language pair, the left number is the to-English correlation, and the right is the from-English. Bolding criteria is the same as in Table 1 .

we include two additional measures to the model selection study: (a) the mean reciprocal rank of the top metric-rated system according to the human ranking, and (b) the difference between the human score of the top human-rated system and that of the top metric-rated system.

Additionally, we report the same study on the WMT17 (Bojar et al., 2017) and the WMT16 (Bojar et al., 2016) datasests in Appendix F. 7 This adds 202 systems to our evaluation.

Image Captioning:

We use the human judgments of twelve submission entries from the COCO 2015 Captioning Challenge. Each participating system generates a caption for each image in the COCO validation set (Lin et al., 2014) , and each image has approximately five reference captions. Following Cui et al. (2018) , we compute the Pearson correlation with two system-level metrics: the percentage of captions that are evaluated as better or equal to human captions (M1) and the percentage of captions that are indistinguishable from human captions (M2). We compute BERTSCORE with multiple references by scoring the candidate with each available reference and returning the highest score. We compare with eight task-agnostic metrics: BLEU (Papineni et al., 2002) , METEOR (Banerjee & Lavie, 2005) , ROUGE-L (Lin, 2004) , CIDER (Vedantam et al., 2015) , BEER (Stanojević & Sima'an, 2014) , EED (Stanchev et al., 2019) , CHRF++ (Popović, 2017) , and CHARACTER (Wang et al., 2016) . We also compare with two task-specific metrics: SPICE (Anderson et al., 2016) and LEIC (Cui et al., 2018) . SPICE is computed using the similarity of scene graphs parsed from the reference and candidate captions. LEIC is trained to predict if a caption is written by a human given the image. Table 3 : Model selection accuracies (Hits@1) on WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10 -3 . We bold the highest numbers for each language pair and direction. Table 4 : Kendall correlations with segment-level human judgments on WMT18. For each language pair, the left number is the to-English correlation, and the right is the from-English. We bold correlations of metrics not significantly outperformed by any other metric under bootstrap sampling for that language pair and direction. The numbers in parenthesis are the number of candidate-reference sentence pairs for each language pair and direction.

5. RESULTS:

Machine Translation Tables 1 2 3 show system-level correlation to human judgements, correlations on hybrid systems, and model selection performance. We observe that BERTSCORE is consistently a top performer. In to-English results, RUSE (Shimanaka et al., 2018) shows competitive performance. However, RUSE is a supervised method trained on WMT16 and WMT15 human judgment data. In cases where RUSE models were not made available, such as for our from-English experiments, it is not possible to use RUSE without additional data and training. Table 4 shows segment-level correlations. We see that BERTSCORE exhibits significantly higher performance compared to the other metrics. The large improvement over BLEU stands out, making BERTSCORE particularly suitable to analyze specific examples, where SENTBLEU is less reliable. In Appendix A, we provide qualitative examples to illustrate the segment-level performance difference between SENTBLEU and BERTSCORE. At the segment-level, BERTSCORE even significantly outperforms RUSE. Overall, we find that applying importance weighting using idf at times provides small benefit, but in other cases does not help. Understanding better when such importance weighting is likely to help is an important direction for future work, and likely depends on the domain of the text and the available test data. We continue without idf weighting for the rest of our experiments. While recall R BERT , precision P BERT , and F1 F BERT alternate as the best measure in different setting, F1 F BERT performs reliably well across all the different settings. Our overall recommendation is therefore to use F1. We present additional results using the full set of 351 systems and evaluation metrics in Tables 12-28 in the appendix, including for experiments with idf importance weighting, different contextual embedding models, and model selection. (Parikh et al., 2016) , DIIN (Gong et al., 2018) , and fine-tuned BERT are reported by Zhang et al. (2019) . Numbers with * are scores on the held-out test set of QQP. We bold the highest correlations of taskspecific and task-agnostic metrics.

cant benefit for this task, suggesting people attribute higher importance to content words. Finally, LEIC (Cui et al., 2018) , a trained metric that takes images as additional inputs and is optimized specifically for the COCO data and this set of systems, outperforms all other methods.

Speed Despite the use of a large pre-trained model, computing BERTSCORE is relatively fast. We are able to process 192.5 candidate-reference pairs/second using a GTX-1080Ti GPU. The complete WMT18 en-de test set, which includes 2,998 sentences, takes 15.6sec to process, compared to 5.4sec with SacreBLEU (Post, 2018) , a common BLEU implementation. Given the sizes of commonly used test and validation sets, the increase in processing time is relatively marginal, and BERTSCORE is a good fit for using during validation (e.g., for stopping) and testing, especially when compared to the time costs of other development stages.

6. ROBUSTNESS ANALYSIS:

We test the robustness of BERTSCORE using adversarial paraphrase classification. We use the Quora Question Pair corpus (QQP; Iyer et al., 2017) We expect that pairs with higher scores are more likely to be paraphrases. To evaluate the automatic metrics on QQA, we use the first 5,000 sentences in the training set instead of the the test set because the test labels are not available. We treat the first sentence as the reference and the second sentence as the candidate. While the differences between the top configurations are often small, it is important for the user to be aware of the different trade-offs, and consider the domain and languages when selecting the exact configuration to use. In general, for machine translation evaluation, we suggest using F BERT , which we find the most reliable. For evaluating text generation in English, we recommend using the 24layer RoBERTa large model to compute BERTSCORE. For non-English language, the multilingual BERT multi is a suitable choice although BERTSCORE computed with this model has less stable performance on low-resource languages. We report the optimal hyperparameter for all models we experimented with in Appendix B Briefly following our initial preprint publication, Zhao et al. ( 2019) published a concurrently developed method related to ours, but with a focus on integrating contextual word embeddings with earth mover's distance (EMD; Rubner et al., 1998) rather than our simple matching process. They also propose various improvements compared to our use of contextualized embeddings. We study these improvements in Appendix C and show that integrating them into BERTSCORE makes it equivalent or better than the EMD-based approach. Largely though, the effect of the different improvements on BERTSCORE is more modest compared to their method. Shortly after our initial publication, YiSi-1 was updated to use BERT embeddings, showing improved performance (Lo, 2019) . This further corroborates our findings. Other recent related work includes training a model on top of BERT to maximize the correlation with human judgments (Mathur et al., 2019) and evaluating generation with a BERT model fine-tuned on paraphrasing (Yoshimura et al., 2019) . More recent work shows the potential of using BERTSCORE for training a summarization system (Li et al., 2019) and for domain-specific evaluation using SciBERT (Beltagy et al., 2019) to evaluate abstractive text summarization (Gabriel et al., 2019) .

In x: Hewlett-Packard to cut up to 30,000 jobs 119 39 441 x: Hewlett-Packard will reduce jobs up to 30.000 3.

x: According to opinion in Hungary, Serbia is "a safe third country". 23 96 465 x: According to Hungarian view, Serbia is a "safe third country." 4.

x: Experts believe November's Black Friday could be holding back spending. 73 147 492 x: Experts believe that the Black Friday in November has put the brakes on spending 5.

x: And it's from this perspective that I will watch him die. 37 111 414 x: And from this perspective, I will see him die. BLEU > FBERT 6.

x: In their view the human dignity of the man had been violated. 500 470 115 x: Look at the human dignity of the man injured. 8.

x: For example when he steered a shot from Ideye over the crossbar in the 56th minute. 516 524 185 x: So, for example, when he steered a shot of Ideye over the latte (56th).

7.:

x: A good prank is funny, but takes moments to reverse. 495 424 152 x: A good prank is funny, but it takes only moments before he becomes a boomerang.

Table 7 : Examples sentences where similarity ranks assigned by Human, F BERT , and BLEU differ significantly on WMT16 German-to-English evaluation task. x: gold reference, x: candidate outputs of MT systems. Rankings assigned by Human, F BERT , and BLEU are shown in the right three columns. The sentences are ranked by the similarity, i.e. rank 1 is the most similar pair assigned by a score. An ideal metric should rank similar to humans.

A QUALITATIVE ANALYSIS:

We study BERTSCORE and SENTBLEU using WMT16 German-to-English (Bojar et al., 2016) . We rank all 560 candidate-reference pairs by human score, BERTSCORE, or SENTBLEU from most similar to least similar. Ideally, the ranking assigned by BERTSCORE and SENTBLEU should be similar to the ranking assigned by the human score.

Table 7 first shows examples where BERTSCORE and SENTBLEU scores disagree about the ranking for a candidate-reference pair by a large number. We observe that BERTSCORE is effectively able to capture synonyms and changes in word order. For example, the reference and candidate sentences in pair 3 are almost identical except that the candidate replaces opinion in Hungary with Hungarian view and switches the order of the quotation mark (") and a. While BERTSCORE ranks the pair relatively high, SENTBLEU judges the pair as dissimilar, because it cannot match synonyms and is sensitive to the small word order changes. Pair 5 shows a set of changes that preserve the semantic meaning: replacing to cut with will reduce and swapping the order of 30,000 and jobs. BERTSCORE ranks the candidate translation similar to the human judgment, whereas SENTBLEU ranks it much lower. We also see that SENTBLEU potentially over-rewards n-gram overlap, even when phrases are used very differently. In pair 6, both the candidate and the reference contain the human dignity of the man. Yet the two sentences convey very different meaning. BERTSCORE agrees with the human judgment and ranks the pair low. In contrast, SENTBLEU considers the pair as relatively similar because of the significant word overlap. The bottom half of Table 7 shows examples where BERTSCORE and human judgments disagree about the ranking. We observe that BERTSCORE finds it difficult to detect factual errors. For example, BERTSCORE assigns high similarity to pair 11 when the translation replaces German language with English and pair 12 where the translation incorrectly outputs Tuesday when it is supposed to generate Monday. BERTSCORE also fails to identify that 5ft 11in is equivalent with 1.80 metres in pair 18. As a result, BERTSCORE assigns low similarity to the eighth pair in Table 7 . SENTBLEU also suffers from these limitations.

Figure 2 visualizes the BERTSCORE matching of two pairs of candidate and reference sentences. The figure illustrates how F BERT matches synonymous phrases, such as imported cars and foreign cars. We also see that F BERT effectively matches words even given a high ordering distortion, for example the token people in the figure.

B REPRESENTATION CHOICE:

As suggested by previous works (Peters et al., 2018; Reimers & Gurevych, 2019) , selecting a good layer or a good combination of layers from the BERT model is important. In designing BERTSCORE, we use WMT16 segment-level human judgment data as a development set to facilitate our representation choice. For Chinese models, we tune with the WMT17 "en-zh" data because the language pair "en-zh" is not available in WMT16. In Figure 3 , we plot the change of human correlation of F BERT over different layers of BERT, RoBERTa, XLNet and XLM models. Based on results from different models, we identify a common trend that F BERT computed with the intermediate representations tends to work better. We tune the number of layer to use for a range of publicly available models. 8 Table 8 shows the results of our hyperparameter search. C ABLATION STUDY OF MOVERSCORE Word Mover's Distance (WMD; Kusner et al., 2015) is a semantic similarity metric that relies on word embeddings and optimal transport. MOVERSCORE (Zhao et al., 2019) combines contextual embeddings and WMD for text generation evaluation. In contrast, BERTSCORE adopts a greedy approach to aggregate token-level information. In addition to using WMD for generation evaluation, Zhao et al. (2019) also introduce various other improvements. We do a detailed ablation study to understand the benefit of each improvement, and to investigate whether it can be applied to BERTSCORE. We use a 12-layer uncased BERT model on the WMT17 to-English segment-level data, the same setting as Zhao et al. (2019) .

We identify several differences between MOVERSCORE and BERTSCORE by analyzing the released source code. We isolate each difference, and mark it with a bracketed tag for our ablation study:

1.

[MNLI] Use a BERT model fine-tuned on MNLI (Williams et al., 2018) .

2.

[PMEANS] Apply power means (Rckl et al., 2018) to aggregate the information of different layers.9 

3. [IDF-L] For reference sentences, instead of computing the idf scores on the 560 sentences in the segment-level data ([IDF-S]), compute the idf scores on the 3,005 sentences in the system-level data.

4.. [SEP]:

For candidate sentences, recompute the idf scores on the candidate sentences. The weighting of reference tokens are kept the same as in [IDF-S]

5.

[RM] Exclude punctuation marks and sub-word tokens except the first sub-word in each word from the matching. 

D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION:

We use the human judgments provided from the MSR Abstractive Text Compression Dataset (Toutanova et al., 2016) to illustrate the applicability of BERTSCORE to abstractive text compression evaluation. The data includes three types of human scores: (a) meaning: how well a compressed text preserve the meaning of the original text; (b) grammar: how grammatically correct a compressed text is; and (c) combined: the average of the meaning and the grammar scores. We follow the experimental setup of Toutanova et al. (2016) and report Pearson correlation between BERTSCORE and the three types of human scores. Table 10 shows that R BERT has the highest correlation with human meaning judgments, and P BERT correlates highly with human grammar judgments. F BERT provides a balance between the two aspects. 

E BERTSCORE OF RECENT MT MODELS:

Table 11 shows the BLEU scores and the BERTSCOREs of pre-trained machine translation models on WMT14 English-to-German, WMT14 English-to-French, IWSLT14 German-to-English task.

We used publicly available pre-trained models from fairseq (Ott et al., 2019) . 11 Because a pretrained Transformer model on IWSLT is not released, we trained our own using the fairseq library. We use multilingual cased BERT base 12 for English-to-German and English-to-French pairs, and English uncased BERT base 13 for German-to-English pairs. Interestingly, the gap between a Dy-namicConv (Wu et al., 2019) trained on only WMT16 and a Transformer (Ott et al., 2018) trained on WMT16 and ParaCrawl 14 (about 30× more training data) becomes larger when evaluated with BERTSCORE rather than BLEU.

F ADDITIONAL RESULTS:

In this section, we present additional experimental results:

1. Segment-level and system-level correlation studies on three years of WMT metric evaluation task (WMT16-18) 2. Model selection study on WMT18 10K hybrid systems 3. System-level correlation study on 2015 COCO captioning challenge 4. Robustness study on PAWS-QQP.

Following BERT (Devlin et al., 2019) , a variety of Transformer-based (Vaswani et al., 2017) pretrained contextual embeddings have been proposed and released. We conduct additional experiments with four types of pre-trained embeddings: BERT, XLM (Lample & Conneau, 2019) , XLNet (Yang et al., 2019b) , and RoBERTa (Liu et al., 2019b) . XLM (Cross-lingual Language Model) is a Transformer pre-trained on the translation language modeling of predicting masked tokens from a pair of sentence in two different languages and masked language modeling tasks using multi-lingual training data. Yang et al. (2019b) modify the Transformer architecture and pre-train it on a permutation language modeling task resulting in some improvement on top of the original BERT when fine-tuned on several downstream tasks. Liu et al. (2019b) introduce RoBERTa (Robustly optimized BERT approach) and demonstrate that an optimized BERT model is comparable to or sometimes outperforms an XLNet on downstream tasks.

We perform a comprehensive study with the following pre-trained contextual embedding models:15 

• BERT models: bert-base-uncased, bert-large-uncased, bert-based-chinese, bert-base-multilingual-cased, and bert-base-cased-mrpc • RoBERTa models: roberta-base, roberta-large, and roberta-large-mnli • XLNet models: xlnet-base-cased and xlnet-base-large • XLM models: xlm-mlm-en-2048 and xlm-mlm-100-1280

F.1 WMT CORRELATION STUDY:

Experimental setup Because of missing data in the released WMT16 dataset (Bojar et al., 2016) , we are only able to experiment with to-English segment-level data, which contains the outputs of 50 different systems on 6 language pairs. We use this data as the validation set for hyperparameter tuning (Appendix B). Table 12 shows the Pearson correlations of all participating metrics and BERTSCOREs computed with different pre-trained models. Significance testing for this dataset does not include the baseline metrics because the released dataset does not contain the original outputs from the baseline metrics. We conduct significance testing between BERTSCORE results only.

The WMT17 dataset (Bojar et al., 2017) contains outputs of 152 different translations on 14 language pairs. We experiment on the segment-level and system-level data on both to-English and from-English language pairs. We exclude fi-en data from the segment-level experiment due to an error in the released data. We compare our results to all participating metrics and perform standard significance testing as done by Bojar et al. (2017) . show the results. The WMT18 dataset (Ma et al., 2018) contains outputs of 159 translation systems on 14 language pairs. In addition to the results in Tables 1 2 3 4 , we complement the study with the correlations of all participating metrics in WMT18 and results from using different contextual models for BERTSCORE.

Results Table 12 -22 collectively showcase the effectiveness of BERTSCORE in correlating with human judgments. The improvement of BERTSCORE is more pronounced on the segment-level than on the system-level. We also see that more optimized or larger BERT models can produce better contextual representations (e.g., comparing F RoBERTa-Large and F BERT-Large ). In contrast, the smaller XLNet performs better than a large one. Based on the evidence in Figure 8 and Tables 12-22, we hypothesize that the permutation language task, though leading to a good set of model weights for fine-tuning on downstream tasks, does not necessarily produce informative pre-trained embeddings for generation evaluation. We also observe that fine-tuning pre-trained models on a related task, such as natural language inference (Williams et al., 2018) , can lead to better human correlation in evaluating text generation. Therefore, for evaluating English sentences, we recommend computing BERTSCORE with a 24-layer RoBERTa model fine-tuned on the MNLI dataset. For evaluating Non-English sentences, both the multilingual BERT model and the XLM model trained on 100 languages are suitable candidates. We also recommend using domain-or language-specific contextual embeddings when possible, such as using BERT Chinese models for evaluating Chinese tasks. In general, we advise users to consider the target domain and languages when selecting the exact configuration to use.

F.2 MODEL SELECTION STUDY:

Experimental setup Similar to Section 4, we use the 10K hybrid systems super-sampled from WMT18. We randomly select 100 out of 10K hybrid systems, rank them using automatic metrics, and repeat this process 100K times. We add to the results in the main paper (Table 3 ) performance of all participating metrics in WMT18 and results from using different contextual embedding models for BERTSCORE. We reuse the hybrid configuration and metric outputs released in WMT18. In addition to the Hits@1 measure, we evaluate the metrics using (a) mean reciprocal rank (MRR) of the top metric-rated system in human rankings, and (b) the absolute human score difference (Diff) between the top metric-and human-rated systems. Hits@1 captures a metric's ability to select the best system. The other two measures quantify the amount of error a metric makes in the selection process. Tables 23 24 25 26 27 28 show the results from these experiments.

Results:

The additional results further support our conclusion from Table 3 : BERTSCORE demonstrates better model selection performance. We also observe that the supervised metric RUSE displays strong model selection ability.

F.3 IMAGE CAPTIONING ON COCO:

We follow the experimental setup described in Section 4. Table 29 shows the correlations of several pre-trained contextual embeddings. We observe that precision-based methods such as BLEU and P BERT are weakly correlated with human judgments on image captioning tasks. We hypothesize that this is because human judges prefer captions that capture the main objects in a picture for image captioning. In general, R BERT has a high correlation, even surpassing the task-specific metric SPICE Anderson et al. (2016) . While the fine-tuned RoBERTa-Large model does not result in the highest correlation, it is one of the best metrics.

F.4 ROBUSTNESS ANALYSIS ON PAWS-QQP:

We present the full results of the robustness study described in Section 6 in Table 30 . In general, we observe that BERTSCORE is more robust than other commonly used metrics. BERTSCORE computed with the 24-layer RoBERTa model performs the best. Fine-tuning RoBERTa-Large on MNLI (Williams et al., 2018) Table 24 : Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10 -3 . We bold the highest numbers for each language pair and direction. Table 25 : Absolute Difference (×100) of the top metric-rated and the top human-rated system on to-English WMT18 hybrid systems. Smaller difference signify higher agreement with human scores. We report the average of 100K samples and the 0.95 confidence intervals are below 10 -3 . We bold the lowest numbers for each language pair and direction. Table 28 : Absolute Difference (×100) of the top metric-rated and the top human-rated system on to-English WMT18 hybrid systems. Smaller difference indicate higher agreement with human scores. We report the average of 100K samples and the 0.95 confidence intervals are below 10 -3 . We bold the lowest numbers for each language pair and direction. 

:

We provide an ablation study of this design choice in Appendix C.

We compare greedy matching with optimal assignment in Appendix C.

https://commoncrawl.org/

BLEU computed on these pairs is around zero.

We use the tokenizer provided with each model. For all Hugging Face models that use the GPT-2 tokenizer, at the time of our experiments, the tokenizer adds a space to the beginning of each sentence.

All the models used are from https://github.com/huggingface/pytorch-transformers.

For WMT16, we only conduct segment-level experiments on to-English pairs due to errors in the dataset.

https://huggingface.co/pytorch-transformers/pretrained models.html

Zhao et al. (2019) uses the embeddings from the last five layers from BERT and L2-normalizes the embedding vectors at each layer before computing the P-MEANs and L2-normalizing the concatenated P-MEANS.

10 A public comment on the project page indicates that some of the techniques are not applied for these two language pairs (https://github.com/AIPHES/emnlp19-moverscore/issues/1).

Denoted by names specified at https://huggingface.co/pytorch-transformers/pretrained models.html.

Figures and Tables:

t e x i t s h a 1 _ b a s e 6 4 = " f 2 y z i m w b R / D g j z p 6 t Z 3 6 0

r 6 o e 2 7 d u 7 + s N W 6 K O M p w A q d w D h 5 c Q Q P u o A k t Y D C B Z 3 i F N y d x X p x 3 5 2 P R W n K K m W P 4 A + f z B 7 A 8 j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Q T n V R V S r n y z z n V U 7 d 5 b F 5 u 0 3 I w = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q

r 6 o e 2 7 d u 7 + s N W 6 K O M p w A q d w D h 5 c Q Q P u o A k t Y D C B Z 3 i F N y d x X p x 3 5 2 P R W n K K m W P 4 A + f z B 7 A 8 j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Q T n V R V S r n y z z n V U 7 d 5 b F 5 u 0 3 I w = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q

r 6 o e 2 7 d u 7 + s N W 6 K O M p w A q d w D h 5 c Q Q P u o A k t Y D C B Z 3 i F N y d x X p x 3 5 2 P R W n K K m W P 4 A + f z B 7 A 8 j 8 k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 5 Q T n V R V S r n y z z n V U 7 d 5 b F 5 u 0 3 I w = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P R i 8 c K 9 g P a U D b b T b t 0 s w m 7 E 7 G E / g g v H h T x 6 u / x 5 r 9 x 0 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 9 z v P H J t R K w e c J p w P 6 I j J U L B K F q p 0 x 9 T z J 5 m g 2 r N r b t z k F X i F a Q G B Z q D 6 l d / G L M 0 4 g q Z p M b 0 P D d B P 6 M a B Z N 8 V u m n h i e U T e i I 9 y x V N O L G z + b n z s

Figure 1: Illustration of the computation of the recall metric R BERT . Given the reference x and candidate x, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy matching in red, and include the optional idf importance weighting.

Figure 2: BERTSCORE visualization. The cosine similarity of each word matching in P BERT are color-coded.

Figure 3: Pearson correlation of F BERT computed with different models, across different layers, with segment-level human judgments on the WMT16 to-English machine translation task. The WMT17 English-Chinese data is used for the BERT Chinese model. Layer 0 corresponds to using BPE embeddings. Consistently, correlation drops significantly in the final layers.

.151 .803/.610 .756/.618 .461/.088 .228/.519 .095/.029 .658/.515 ITER .154/.000 .814/.692 .742/.733 .475/.111 .234/.532 .102/.030 .673/ -.184 .824/.703 .769/.763 .501/.082 .262/.544 .142/.031 .673/.629 FBERT (idf) .179/.178 .824/.722 .760/.764 .503/.082 .265/.539 .004/.030 .678/.595

Table content: <table><tr><td>Metric</td><td>en↔cs</td><td>en↔de</td><td>en↔et</td><td>en↔fi</td><td>en↔ru</td><td>en↔tr</td><td>en↔zh</td></tr><tr><td colspan="2">BLEU .134/RUSE .214/ -</td><td>.823/ -</td><td>.785/ -</td><td>.487/ -</td><td>.248/ -</td><td>.109/ -</td><td>.670/ -</td></tr><tr><td>YiSi-1</td><td colspan="7">.159/.178 .809/.671 .749/.671 .467/.230 .248/.544 .108/.398 .613/.594</td></tr><tr><td>PBERT</td><td colspan="7">.173/.180 .706/.663 .764/.771 .498/.078 .255/.545 .140/.372 .661/.551</td></tr><tr><td>RBERT</td><td colspan="7">.163/.184 .804/.730 .770/.722 .494/.148 .260/.542 .005/.030 .677/.657</td></tr><tr><td>FBERT</td><td>.175/</td><td/><td/><td/><td/><td/><td/></tr></table>

Pearson correlation on the 2015 COCO Captioning Challenge.

Table content: <table><tr><td>Metric</td><td>M1</td><td>M2</td><td>Type</td><td>Method</td><td>QQP</td><td>PAWSQQP</td></tr><tr><td>BLEU METEOR ROUGE-L</td><td colspan="2">-0.019  *  -0.005  *  0.606  *  0.594  *  0.090  *  0.096  *</td><td>Trained on QQP (supervised)</td><td>DecAtt DIIN BERT</td><td>0.939  *  0.952  *  0.963  *</td><td>0.263 0.324 0.351</td></tr><tr><td>CIDER</td><td>0.438  *</td><td>0.440  *</td><td>Trained on QQP</td><td>DecAtt</td><td>-</td><td>0.511</td></tr><tr><td>SPICE LEIC</td><td>0.759  *  0.939  *</td><td>0.750  *  0.949  *</td><td>+ PAWSQQP (supervised)</td><td>DIIN BERT</td><td>--</td><td>0.778 0.831</td></tr><tr><td>BEER</td><td>0.491</td><td>0.562</td><td/><td>BLEU</td><td>0.707</td><td>0.527</td></tr><tr><td>EED</td><td>0.545</td><td>0.599</td><td/><td>METEOR</td><td>0.755</td><td>0.532</td></tr><tr><td>CHRF++</td><td>0.702</td><td>0.729</td><td/><td>ROUGE-L</td><td>0.740</td><td>0.536</td></tr><tr><td>CHARACTER P BERT R BERT F BERT R BERT (idf)</td><td>0.800 -0.105 0.888 0.322 0.917</td><td>0.801 -0.041 0.863 0.350 0.889</td><td>Metric (Not trained on QQP or PAWSQQP)</td><td>CHRF++ BEER EED CHARACTER PBERT RBERT</td><td>0.577 0.741 0.743 0.698 0.757 0.744</td><td>0.608 0.564 0.611 0.650 0.687 0.685</td></tr><tr><td/><td/><td/><td/><td>FBERT</td><td>0.761</td><td>0.685</td></tr><tr><td/><td/><td/><td/><td>FBERT (idf)</td><td>0.777</td><td>0.693</td></tr></table>

Area under ROC curve (AUC) on QQP and PAWS QQP datasets. The scores of trained De-cATT

Table content: <table/>

and the adversarial paraphrases from the Paraphrase Adversaries from Word Scrambling dataset (PAWS;Zhang et al., 2019). Both datasets contain pairs of sentences labeled to indicate whether they are paraphrases or not. Positive examples in QQP are real duplicate questions, while negative examples are related, but different questions. Sentence pairs in PAWS are generated through word swapping. For example, in PAWS, Flights from New York to Florida may be changed to Flights from Florida to New York and a good classifier should identify that these two sentences are not paraphrases. PAWS includes two parts: PAWS QQP , which is based on the QQP data, and PAWS Wiki . We use the PAWS QQP development set which contains 667 sentences. For the automatic metrics, we use no paraphrase detection training data.

Table content: <table/>

Whenadversarial examples are provided in training, state-of-the-art models like DIIN(Gong et al., 2018) and fine-tuned BERT are able to identify the adversarial examples but their performance still decreases significantly from their performance on QQP. Most metrics have decent performance on QQP, but show a significant performance drop on PAWS QQP , almost down to chance performance. This suggests these metrics fail to to distinguish the harder adversarial examples. In contrast, the performance of BERTSCORE drops only slightly, showing more robustness than the other metrics.7 DISCUSSIONWe propose BERTSCORE, a new metric for evaluating generated text against gold standard references. BERTSCORE is purposely designed to be simple, task agnostic, and easy to use. Our analysis illustrates how BERTSCORE resolves some of the limitations of commonly used metrics, especially on challenging adversarial examples. We conduct extensive experiments with various configuration choices for BERTSCORE, including the contextual embedding model used and the use of importance weighting. Overall, our extensive experiments, including the ones in the appendix, show that BERTSCORE achieves better correlation than common metrics, and is effective for model selection. However, there is no one configuration of BERTSCORE that clearly outperforms all others.

Table content: <table/>

future work, we look forward to designing new task-specific metrics that use BERTSCORE as a subroutine and accommodate task-specific needs, similar to howWieting et al. (2019) suggests to use semantic similarity for machine translation training. Because BERTSCORE is fully differentiable, it also can be incorporated into a training procedure to compute a learning loss that reduces the mismatch between optimization and evaluation objectives. At the same time, Kingfisher will close 60 B & Q stores nationwide 2.

Table content: <table><tr><td colspan="2">Case No. Reference and Candidate Pairs</td><td colspan="3">Human FBERT BLEU</td></tr><tr><td>1.</td><td>x: At the same time Kingfisher is closing 60 B&amp;Q outlets across the country x:</td><td>38</td><td>125</td><td>530</td></tr><tr><td>BLEU</td><td/><td/><td/><td/></tr><tr><td>FBERT &gt;</td><td/><td/><td/><td/></tr></table>

Recommended layer of representation to use for BERTSCORE. The layers are chosen based on a held-out validation set (WMT16).

Table content: <table><tr><td>Model</td><td colspan="2">Total Number of Layers Best Layer</td></tr><tr><td>bert-base-uncased</td><td>12</td><td>9</td></tr><tr><td>bert-large-uncased</td><td>24</td><td>18</td></tr><tr><td>bert-base-cased-finetuned-mrpc</td><td>12</td><td>9</td></tr><tr><td>bert-base-multilingual-cased</td><td>12</td><td>9</td></tr><tr><td>bert-base-chinese</td><td>12</td><td>8</td></tr><tr><td>roberta-base</td><td>12</td><td>10</td></tr><tr><td>roberta-large</td><td>24</td><td>17</td></tr><tr><td>roberta-large-mnli</td><td>24</td><td>19</td></tr><tr><td>xlnet-base-cased</td><td>12</td><td>5</td></tr><tr><td>xlnet-large-cased</td><td>24</td><td>7</td></tr><tr><td>xlm-mlm-en-2048</td><td>12</td><td>7</td></tr><tr><td>xlm-mlm-100-1280</td><td>16</td><td>11</td></tr></table>

Ablation Study of MOVERSCORE and BERTSCORE using Pearson correlations on the WMT17 to-English segment-level data. Correlations that are not outperformed by others for that language pair under Williams Test are bolded. We observe that using WMD does not consistently improve BERTSCORE.

Table content: <table><tr><td>Ablation</td><td colspan="2">Metric cs-en de-en fi-en lv-en ru-en tr-en zh-en</td></tr><tr><td/><td colspan="2">WMD1 0.628 0.655 0.795 0.692 0.701 0.715 0.699</td></tr><tr><td>Vanilla</td><td colspan="2">WMD2 0.638 0.661 0.797 0.695 0.700 0.728 0.714</td></tr><tr><td/><td>FBERT</td><td>0.659 0.680 0.817 0.702 0.719 0.727 0.717</td></tr><tr><td/><td colspan="2">WMD1 0.636 0.662 0.824 0.709 0.716 0.728 0.713</td></tr><tr><td>IDF-S</td><td colspan="2">WMD2 0.643 0.662 0.821 0.708 0.712 0.732 0.715</td></tr><tr><td/><td>FBERT</td><td>0.657 0.681 0.823 0.713 0.725 0.718 0.711</td></tr><tr><td/><td colspan="2">WMD1 0.633 0.659 0.825 0.708 0.716 0.727 0.715</td></tr><tr><td>IDF-L</td><td colspan="2">WMD2 0.641 0.661 0.822 0.708 0.713 0.730 0.716</td></tr><tr><td/><td>FBERT</td><td>0.655 0.682 0.823 0.713 0.726 0.718 0.712</td></tr><tr><td/><td colspan="2">WMD1 0.651 0.660 0.819 0.703 0.714 0.724 0.715</td></tr><tr><td>IDF-L + SEP</td><td colspan="2">WMD2 0.659 0.662 0.816 0.702 0.712 0.729 0.715</td></tr><tr><td/><td>FBERT</td><td>0.664 0.681 0.818 0.709 0.724 0.716 0.710</td></tr><tr><td>IDF-L + SEP + RM</td><td colspan="2">WMD1 0.651 0.686 0.803 0.681 0.730 0.730 0.720 WMD2 0.664 0.687 0.797 0.679 0.728 0.735 0.718 FBERT 0.659 0.695 0.800 0.683 0.734 0.722 0.712</td></tr><tr><td>IDF-L + SEP + PMEANS</td><td colspan="2">WMD1 0.658 0.663 0.820 0.707 0.717 0.725 0.712 WMD2 0.667 0.665 0.817 0.707 0.717 0.727 0.712 FBERT 0.671 0.682 0.819 0.708 0.725 0.715 0.704</td></tr><tr><td>IDF-L + SEP + MNLI</td><td colspan="2">WMD1 0.659 0.679 0.822 0.732 0.718 0.746 0.725 WMD2 0.664 0.682 0.819 0.731 0.715 0.748 0.722 FBERT 0.668 0.701 0.825 0.737 0.727 0.744 0.725</td></tr><tr><td>IDF-L + SEP + PMEANS + MNLI</td><td colspan="2">WMD1 0.672 0.686 0.831 0.738 0.725 0.753 0.737 WMD2 0.677 0.690 0.828 0.736 0.722 0.755 0.735 FBERT 0.682 0.707 0.836 0.741 0.732 0.751 0.736</td></tr><tr><td>IDF-L + SEP</td><td colspan="2">WMD1 0.670 0.708 0.821 0.717 0.738 0.762 0.744</td></tr><tr><td>+ PMEANS + MNLI</td><td colspan="2">WMD2 0.679 0.709 0.814 0.716 0.736 0.762 0.738</td></tr><tr><td>+ RM</td><td>FBERT</td><td>0.676 0.717 0.824 0.719 0.740 0.757 0.738</td></tr></table>

Pearson correlations with human judgments on the MSR Abstractive Text Compression Dataset.

Table content: <table/>

*(Ott et al., 2018) 0.298 0.6587 0.6528 0.6558 0.8687 0.8664 0.8674 DynamicConv * * *(Wu et al., 2019) 0.297 0.6526 0.6464 0.6495 0.8664 0.8640 0.8650 BLEU scores and BERTSCOREs of publicly available pre-trained MT models in fairseq(Ott et al., 2019). We show both rescaled scores marked with ˆand raw BERTSCORE scores. * : trained on unconfirmed WMT data version, * * : trained on WMT16 + ParaCrawl, * * * : trained on WMT16, + : trained by us using fairseq.

Table content: <table><tr><td>Task</td><td>Model</td><td>BLEU PBERT RBERT FBERT PBERT RBERT FBERT</td></tr><tr><td colspan="2">ConvS2S (Gehring et al., 2017) ConvS2S (Gehring et al., 2017) Transformer-big  WMT14 WMT14 En-De Transformer-big (Ott et al., 2018) En-Fr DynamicConv (Wu et al., 2019)</td><td>0.266 0.6099 0.6055 0.6075 0.8499 0.8482 0.8488 0.408 0.6998 0.6821 0.6908 0.8876 0.8810 0.8841 0.432 0.7148 0.6978 0.7061 0.8932 0.8869 0.8899 0.432 0.7156 0.6989 0.7071 0.8936 0.8873 0.8902</td></tr><tr><td>IWSLT14 De-En</td><td colspan="2">Transformer-iwslt + (Ott et al., 2019) 0.350 0.6749 0.6590 0.6672 0.9452 0.9425 0.9438 LightConv (Wu et al., 2019) 0.348 0.6737 0.6542 0.6642 0.9450 0.9417 0.9433 DynamicConv (Wu et al., 2019) 0.352 0.6770 0.6586 0.6681 0.9456 0.9425 0.9440</td></tr></table>

can significantly improve the robustness against adversarial sentences. However, a fine-tuned BERT on MRPC (Microsoft Research Paraphrasing Corpus)(Dolan & Brockett, 2005) performs worse than its counterpart. Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Table content: <table><tr><td>Setting</td><td>Metric</td><td>cs-en</td><td>de-en</td><td>fi-en</td><td>ro-en</td><td>ru-en</td><td>tr-en</td></tr><tr><td/><td/><td>560</td><td>560</td><td>560</td><td>560</td><td>560</td><td>560</td></tr><tr><td/><td>DPMFCOMB</td><td>0.713</td><td>0.584</td><td>0.598</td><td>0.627</td><td>0.615</td><td>0.663</td></tr><tr><td/><td>METRICS-F</td><td>0.696</td><td>0.601</td><td>0.557</td><td>0.662</td><td>0.618</td><td>0.649</td></tr><tr><td/><td>COBALT-F.</td><td>0.671</td><td>0.591</td><td>0.554</td><td>0.639</td><td>0.618</td><td>0.627</td></tr><tr><td/><td>UPF-COBA.</td><td>0.652</td><td>0.550</td><td>0.490</td><td>0.616</td><td>0.556</td><td>0.626</td></tr><tr><td/><td>MPEDA</td><td>0.644</td><td>0.538</td><td>0.513</td><td>0.587</td><td>0.545</td><td>0.616</td></tr><tr><td/><td>CHRF2</td><td>0.658</td><td>0.457</td><td>0.469</td><td>0.581</td><td>0.534</td><td>0.556</td></tr><tr><td>Unsupervised</td><td>CHRF3 CHRF1</td><td>0.660 0.644</td><td>0.455 0.454</td><td>0.472 0.452</td><td>0.582 0.570</td><td>0.535 0.522</td><td>0.555 0.551</td></tr><tr><td/><td>UOW-REVAL</td><td>0.577</td><td>0.528</td><td>0.471</td><td>0.547</td><td>0.528</td><td>0.531</td></tr><tr><td/><td>WORDF3</td><td>0.599</td><td>0.447</td><td>0.473</td><td>0.525</td><td>0.504</td><td>0.536</td></tr><tr><td/><td>WORDF2</td><td>0.596</td><td>0.445</td><td>0.471</td><td>0.522</td><td>0.503</td><td>0.537</td></tr><tr><td/><td>WORDF1</td><td>0.585</td><td>0.435</td><td>0.464</td><td>0.508</td><td>0.497</td><td>0.535</td></tr><tr><td/><td>SENTBLEU</td><td>0.557</td><td>0.448</td><td>0.484</td><td>0.499</td><td>0.502</td><td>0.532</td></tr><tr><td/><td>DTED</td><td>0.394</td><td>0.254</td><td>0.361</td><td>0.329</td><td>0.375</td><td>0.267</td></tr><tr><td>Supervised</td><td>BEER</td><td>0.661</td><td>0.462</td><td>0.471</td><td>0.551</td><td>0.533</td><td>0.545</td></tr><tr><td/><td>PBERT-Base</td><td>0.729</td><td>0.617</td><td>0.719</td><td>0.651</td><td>0.684</td><td>0.678</td></tr><tr><td/><td>RBERT-Base</td><td>0.741</td><td>0.639</td><td>0.616</td><td>0.693</td><td>0.660</td><td>0.660</td></tr><tr><td/><td>FBERT-Base</td><td>0.747</td><td>0.640</td><td>0.661</td><td>0.723</td><td>0.672</td><td>0.688</td></tr><tr><td/><td>PBERT-Base (no idf)</td><td>0.723</td><td>0.638</td><td>0.662</td><td>0.700</td><td>0.633</td><td>0.696</td></tr><tr><td/><td>RBERT-Base (no idf)</td><td>0.745</td><td>0.656</td><td>0.638</td><td>0.697</td><td>0.653</td><td>0.674</td></tr><tr><td/><td>FBERT-Base (no idf)</td><td>0.747</td><td>0.663</td><td>0.666</td><td>0.714</td><td>0.662</td><td>0.703</td></tr><tr><td/><td>PBERT-Base-MRPC</td><td>0.697</td><td>0.618</td><td>0.614</td><td>0.676</td><td>0.62</td><td>0.695</td></tr><tr><td/><td>RBERT-Base-MRPC</td><td>0.723</td><td>0.636</td><td>0.587</td><td>0.667</td><td>0.648</td><td>0.664</td></tr><tr><td/><td>FBERT-Base-MRPC</td><td>0.725</td><td>0.644</td><td>0.617</td><td>0.691</td><td>0.654</td><td>0.702</td></tr><tr><td/><td>PBERT-Base-MRPC (idf)</td><td>0.713</td><td>0.613</td><td>0.630</td><td>0.693</td><td>0.635</td><td>0.691</td></tr><tr><td/><td>RBERT-Base-MRPC (idf)</td><td>0.727</td><td>0.631</td><td>0.573</td><td>0.666</td><td>0.642</td><td>0.662</td></tr><tr><td/><td>FBERT-Base-MRPC (idf)</td><td>0.735</td><td>0.637</td><td>0.620</td><td>0.700</td><td>0.658</td><td>0.697</td></tr><tr><td/><td>PBERT-Large</td><td>0.756</td><td>0.671</td><td>0.701</td><td>0.723</td><td>0.678</td><td>0.706</td></tr><tr><td/><td>RBERT-Large</td><td>0.768</td><td>0.684</td><td>0.677</td><td>0.720</td><td>0.686</td><td>0.699</td></tr><tr><td/><td>FBERT-Large</td><td>0.774</td><td>0.693</td><td>0.705</td><td>0.736</td><td>0.701</td><td>0.717</td></tr><tr><td/><td>PBERT-Large (idf)</td><td>0.758</td><td>0.653</td><td>0.704</td><td>0.734</td><td>0.685</td><td>0.705</td></tr><tr><td/><td>RBERT-Large (idf)</td><td>0.771</td><td>0.680</td><td>0.661</td><td>0.718</td><td>0.687</td><td>0.692</td></tr><tr><td/><td>FBERT-Large (idf)</td><td>0.774</td><td>0.678</td><td>0.700</td><td>0.740</td><td>0.701</td><td>0.711</td></tr><tr><td/><td>PRoBERTa-Base RRoBERTa-Base</td><td>0.738 0.745</td><td>0.642 0.669</td><td>0.671 0.645</td><td>0.712 0.698</td><td>0.669 0.682</td><td>0.671 0.653</td></tr><tr><td/><td>FRoBERTa-Base</td><td>0.761</td><td>0.674</td><td>0.686</td><td>0.732</td><td>0.697</td><td>0.689</td></tr><tr><td>Pre-Trained</td><td>PRoBERTa-Base (idf) RRoBERTa-Base (idf)</td><td>0.751 0.744</td><td>0.626 0.652</td><td>0.678 0.638</td><td>0.723 0.699</td><td>0.685 0.685</td><td>0.668 0.657</td></tr><tr><td/><td>FRoBERTa-Base (idf)</td><td>0.767</td><td>0.653</td><td>0.688</td><td>0.737</td><td>0.705</td><td>0.685</td></tr><tr><td/><td>PRoBERTa-Large RRoBERTa-Large</td><td>0.757 0.765</td><td>0.702 0.713</td><td>0.709 0.686</td><td>0.735 0.718</td><td>0.721 0.714</td><td>0.676 0.676</td></tr><tr><td/><td>FRoBERTa-Large</td><td>0.780</td><td>0.724</td><td>0.728</td><td>0.753</td><td>0.738</td><td>0.709</td></tr><tr><td/><td>PRoBERTa-Large (idf)</td><td>0.771</td><td>0.682</td><td>0.705</td><td>0.727</td><td>0.714</td><td>0.681</td></tr><tr><td/><td>RRoBERTa-Large (idf)</td><td>0.762</td><td>0.695</td><td>0.683</td><td>0.711</td><td>0.708</td><td>0.678</td></tr><tr><td/><td>FRoBERTa-Large (idf)</td><td>0.786</td><td>0.704</td><td>0.727</td><td>0.747</td><td>0.732</td><td>0.711</td></tr><tr><td/><td>PRoBERTa-Large-MNLI RRoBERTa-Large-MNLI</td><td>0.777 0.790</td><td>0.718 0.731</td><td>0.733 0.702</td><td>0.744 0.741</td><td>0.729 0.727</td><td>0.747 0.732</td></tr><tr><td/><td>FRoBERTa-Large-MNLI</td><td>0.795</td><td>0.736</td><td>0.733</td><td>0.757</td><td>0.744</td><td>0.756</td></tr><tr><td/><td>PRoBERTa-Large-MNLI (idf)</td><td>0.794</td><td>0.695</td><td>0.731</td><td>0.752</td><td>0.732</td><td>0.747</td></tr><tr><td/><td>RRoBERTa-Large-MNLI (idf)</td><td>0.792</td><td>0.706</td><td>0.694</td><td>0.737</td><td>0.724</td><td>0.733</td></tr><tr><td/><td>FRoBERTa-Large-MNLI (idf)</td><td>0.804</td><td>0.710</td><td>0.729</td><td>0.760</td><td>0.742</td><td>0.754</td></tr><tr><td/><td>PXLNet-Base</td><td>0.708</td><td>0.612</td><td>0.639</td><td>0.650</td><td>0.606</td><td>0.690</td></tr><tr><td/><td>RXLNet-Base FXLNet-Base</td><td>0.728 0.727</td><td>0.630 0.631</td><td>0.617 0.640</td><td>0.645 0.659</td><td>0.621 0.626</td><td>0.675 0.695</td></tr><tr><td/><td>PXLNet-Base (idf)</td><td>0.726</td><td>0.618</td><td>0.655</td><td>0.678</td><td>0.629</td><td>0.700</td></tr><tr><td/><td>RXLNet-Base (idf)</td><td>0.734</td><td>0.633</td><td>0.618</td><td>0.66</td><td>0.635</td><td>0.682</td></tr><tr><td/><td>FXLNet-Base (idf)</td><td>0.739</td><td>0.633</td><td>0.649</td><td>0.681</td><td>0.643</td><td>0.702</td></tr><tr><td/><td>PXL-NET-LARGE</td><td>0.710</td><td>0.577</td><td>0.643</td><td>0.647</td><td>0.616</td><td>0.684</td></tr><tr><td/><td>RXL-NET-LARGE FXL-NET-LARGE</td><td>0.732 0.733</td><td>0.600 0.600</td><td>0.610 0.643</td><td>0.636 0.655</td><td>0.627 0.637</td><td>0.668 0.691</td></tr><tr><td/><td>PXL-NET-LARGE (idf)</td><td>0.728</td><td>0.574</td><td>0.652</td><td>0.669</td><td>0.633</td><td>0.681</td></tr><tr><td/><td>RXL-NET-LARGE (idf)</td><td>0.735</td><td>0.592</td><td>0.597</td><td>0.642</td><td>0.629</td><td>0.662</td></tr><tr><td/><td>FXL-NET-LARGE (idf)</td><td>0.742</td><td>0.592</td><td>0.643</td><td>0.670</td><td>0.645</td><td>0.685</td></tr><tr><td/><td>PXLM-En</td><td>0.688</td><td>0.569</td><td>0.613</td><td>0.645</td><td>0.583</td><td>0.659</td></tr><tr><td/><td>RXLM-En</td><td>0.715</td><td>0.603</td><td>0.577</td><td>0.645</td><td>0.609</td><td>0.644</td></tr><tr><td/><td>FXLM-En PXLM-En (idf)</td><td>0.713 0.728</td><td>0.597 0.576</td><td>0.610 0.649</td><td>0.657 0.681</td><td>0.610 0.604</td><td>0.668 0.683</td></tr><tr><td/><td>RXLM-En (idf)</td><td>0.730</td><td>0.597</td><td>0.591</td><td>0.659</td><td>0.622</td><td>0.669</td></tr><tr><td/><td>FXLM-En (idf)</td><td>0.739</td><td>0.594</td><td>0.636</td><td>0.682</td><td>0.626</td><td>0.691</td></tr></table>

Absolute Pearson correlation (|r|) and Kendall correlation (τ ) with segment-level human judgments on WMT17 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Table content: <table><tr><td>Setting</td><td>Metric</td><td>cs-en</td><td>de-en</td><td>fi-en</td><td>lv-en</td><td>ru-en</td><td>tr-en</td><td>zh-en</td></tr><tr><td/><td/><td>4</td><td>11</td><td>6</td><td>9</td><td>9</td><td>10</td><td>16</td></tr><tr><td/><td>BLEU</td><td>0.971</td><td>0.923</td><td>0.903</td><td>0.979</td><td>0.912</td><td>0.976</td><td>0.864</td></tr><tr><td/><td>CDER</td><td>0.989</td><td>0.930</td><td>0.927</td><td>0.985</td><td>0.922</td><td>0.973</td><td>0.904</td></tr><tr><td/><td>CHARACTER</td><td>0.972</td><td>0.974</td><td>0.946</td><td>0.932</td><td>0.958</td><td>0.949</td><td>0.799</td></tr><tr><td/><td>CHRF</td><td>0.939</td><td>0.968</td><td>0.938</td><td>0.968</td><td>0.952</td><td>0.944</td><td>0.859</td></tr><tr><td/><td>CHRF++</td><td>0.940</td><td>0.965</td><td>0.927</td><td>0.973</td><td>0.945</td><td>0.960</td><td>0.880</td></tr><tr><td/><td>MEANT 2.0</td><td>0.926</td><td>0.950</td><td>0.941</td><td>0.970</td><td>0.962</td><td>0.932</td><td>0.838</td></tr><tr><td>Unsupervised</td><td>MEANT 2.0-NOSRL</td><td>0.902</td><td>0.936</td><td>0.933</td><td>0.963</td><td>0.960</td><td>0.896</td><td>0.800</td></tr><tr><td/><td>NIST</td><td>1.000</td><td>0.931</td><td>0.931</td><td>0.960</td><td>0.912</td><td>0.971</td><td>0.849</td></tr><tr><td/><td>PER</td><td>0.968</td><td>0.951</td><td>0.896</td><td>0.962</td><td>0.911</td><td>0.932</td><td>0.877</td></tr><tr><td/><td>TER</td><td>0.989</td><td>0.906</td><td>0.952</td><td>0.971</td><td>0.912</td><td>0.954</td><td>0.847</td></tr><tr><td/><td>TREEAGGREG</td><td>0.983</td><td>0.920</td><td>0.977</td><td>0.986</td><td>0.918</td><td>0.987</td><td>0.861</td></tr><tr><td/><td>UHH TSKM</td><td>0.996</td><td>0.937</td><td>0.921</td><td>0.990</td><td>0.914</td><td>0.987</td><td>0.902</td></tr><tr><td/><td>WER</td><td>0.987</td><td>0.896</td><td>0.948</td><td>0.969</td><td>0.907</td><td>0.925</td><td>0.839</td></tr><tr><td/><td>AUTODA</td><td>0.438</td><td>0.959</td><td>0.925</td><td>0.973</td><td>0.907</td><td>0.916</td><td>0.734</td></tr><tr><td/><td>BEER</td><td>0.972</td><td>0.960</td><td>0.955</td><td>0.978</td><td>0.936</td><td>0.972</td><td>0.902</td></tr><tr><td>Supervised</td><td>BLEND</td><td>0.968</td><td>0.976</td><td>0.958</td><td>0.979</td><td>0.964</td><td>0.984</td><td>0.894</td></tr><tr><td/><td>BLEU2VEC</td><td>0.989</td><td>0.936</td><td>0.888</td><td>0.966</td><td>0.907</td><td>0.961</td><td>0.886</td></tr><tr><td/><td>NGRAM2VEC</td><td>0.984</td><td>0.935</td><td>0.890</td><td>0.963</td><td>0.907</td><td>0.955</td><td>0.880</td></tr><tr><td/><td>PBERT-Base</td><td>0.975</td><td>0.936</td><td>0.991</td><td>0.993</td><td>0.918</td><td>0.981</td><td>0.892</td></tr><tr><td/><td>RBERT-Base</td><td>0.995</td><td>0.975</td><td>0.944</td><td>0.978</td><td>0.953</td><td>0.991</td><td>0.975</td></tr><tr><td/><td>FBERT-Base</td><td>0.987</td><td>0.961</td><td>0.979</td><td>0.991</td><td>0.937</td><td>0.991</td><td>0.953</td></tr><tr><td/><td>PBERT-Base (idf) RBERT-Base (idf)</td><td>0.983 0.997</td><td>0.937 0.981</td><td>0.998 0.962</td><td>0.992 0.968</td><td>0.939 0.977</td><td>0.985 0.985</td><td>0.878 0.949</td></tr><tr><td/><td>FBERT-Base (idf)</td><td>0.992</td><td>0.967</td><td>0.995</td><td>0.992</td><td>0.960</td><td>0.996</td><td>0.951</td></tr><tr><td/><td>PBERT-Base-MRPC</td><td>0.982</td><td>0.926</td><td>0.990</td><td>0.987</td><td>0.916</td><td>0.970</td><td>0.899</td></tr><tr><td/><td>RBERT-Base-MRPC</td><td>0.999</td><td>0.979</td><td>0.950</td><td>0.982</td><td>0.957</td><td>0.977</td><td>0.985</td></tr><tr><td/><td>FBERT-Base-MRPC</td><td>0.994</td><td>0.957</td><td>0.986</td><td>0.994</td><td>0.938</td><td>0.980</td><td>0.960</td></tr><tr><td/><td>PBERT-Base-MRPC (idf)</td><td>0.989</td><td>0.936</td><td>0.992</td><td>0.979</td><td>0.931</td><td>0.976</td><td>0.892</td></tr><tr><td/><td>RBERT-Base-MRPC (idf) FBERT-Base-MRPC (idf)</td><td>0.999 0.997</td><td>0.987 0.968</td><td>0.962 0.995</td><td>0.980 0.997</td><td>0.975 0.956</td><td>0.979 0.989</td><td>0.973 0.963</td></tr><tr><td/><td>PBERT-Large</td><td>0.981</td><td>0.937</td><td>0.991</td><td>0.996</td><td>0.921</td><td>0.987</td><td>0.905</td></tr><tr><td/><td>RBERT-Large</td><td>0.996</td><td>0.975</td><td>0.953</td><td>0.985</td><td>0.954</td><td>0.992</td><td>0.977</td></tr><tr><td/><td>FBERT-Large</td><td>0.990</td><td>0.960</td><td>0.981</td><td>0.995</td><td>0.938</td><td>0.992</td><td>0.957</td></tr><tr><td/><td>PBERT-Large (idf)</td><td>0.986</td><td>0.938</td><td>0.998</td><td>0.995</td><td>0.939</td><td>0.994</td><td>0.897</td></tr><tr><td/><td>RBERT-Large (idf) FBERT-Large (idf)</td><td>0.997 0.994</td><td>0.982 0.965</td><td>0.967 0.993</td><td>0.979 0.995</td><td>0.974 0.958</td><td>0.992 0.998</td><td>0.966 0.959</td></tr><tr><td/><td>PRoBERTa-Base</td><td>0.987</td><td>0.930</td><td>0.984</td><td>0.966</td><td>0.916</td><td>0.963</td><td>0.955</td></tr><tr><td/><td>RRoBERTa-Base</td><td>0.999</td><td>0.982</td><td>0.947</td><td>0.979</td><td>0.956</td><td>0.986</td><td>0.984</td></tr><tr><td/><td>FRoBERTa-Base</td><td>0.996</td><td>0.961</td><td>0.993</td><td>0.993</td><td>0.937</td><td>0.983</td><td>0.982</td></tr><tr><td/><td>PRoBERTa-Base (idf)</td><td>0.990</td><td>0.938</td><td>0.980</td><td>0.956</td><td>0.929</td><td>0.967</td><td>0.962</td></tr><tr><td/><td>RRoBERTa-Base (idf)</td><td>0.998</td><td>0.987</td><td>0.963</td><td>0.979</td><td>0.971</td><td>0.986</td><td>0.974</td></tr><tr><td/><td>FRoBERTa-Base (idf)</td><td>0.996</td><td>0.970</td><td>0.999</td><td>0.994</td><td>0.952</td><td>0.989</td><td>0.982</td></tr><tr><td>Pre-Trained</td><td>PRoBERTa-Large</td><td>0.989</td><td>0.948</td><td>0.984</td><td>0.949</td><td>0.927</td><td>0.960</td><td>0.967</td></tr><tr><td/><td>RRoBERTa-Large</td><td>0.998</td><td>0.988</td><td>0.957</td><td>0.983</td><td>0.969</td><td>0.982</td><td>0.984</td></tr><tr><td/><td>FRoBERTa-Large</td><td>0.996</td><td>0.973</td><td>0.997</td><td>0.991</td><td>0.949</td><td>0.984</td><td>0.987</td></tr><tr><td/><td>PRoBERTa-Large (idf)</td><td>0.989</td><td>0.959</td><td>0.975</td><td>0.935</td><td>0.944</td><td>0.968</td><td>0.974</td></tr><tr><td/><td>RRoBERTa-Large (idf)</td><td>0.995</td><td>0.991</td><td>0.962</td><td>0.979</td><td>0.981</td><td>0.981</td><td>0.970</td></tr><tr><td/><td>FRoBERTa-Large (idf)</td><td>0.996</td><td>0.982</td><td>0.998</td><td>0.991</td><td>0.965</td><td>0.991</td><td>0.984</td></tr><tr><td/><td>PRoBERTa-Large-MNLI</td><td>0.994</td><td>0.963</td><td>0.995</td><td>0.990</td><td>0.944</td><td>0.981</td><td>0.974</td></tr><tr><td/><td>RRoBERTa-Large-MNLI</td><td>0.995</td><td>0.991</td><td>0.962</td><td>0.981</td><td>0.973</td><td>0.985</td><td>0.984</td></tr><tr><td/><td>FRoBERTa-Large-MNLI</td><td>0.999</td><td>0.982</td><td>0.992</td><td>0.996</td><td>0.961</td><td>0.988</td><td>0.989</td></tr><tr><td/><td>PRoBERTa-Large-MNLI (idf)</td><td>0.995</td><td>0.970</td><td>0.997</td><td>0.985</td><td>0.955</td><td>0.988</td><td>0.979</td></tr><tr><td/><td>RRoBERTa-Large-MNLI (idf)</td><td>0.994</td><td>0.992</td><td>0.967</td><td>0.977</td><td>0.983</td><td>0.988</td><td>0.972</td></tr><tr><td/><td>FRoBERTa-Large-MNLI (idf)</td><td>0.999</td><td>0.989</td><td>0.996</td><td>0.997</td><td>0.972</td><td>0.994</td><td>0.987</td></tr><tr><td/><td>PXLNET-Base</td><td>0.988</td><td>0.938</td><td>0.993</td><td>0.993</td><td>0.914</td><td>0.974</td><td>0.960</td></tr><tr><td/><td>RXLNET-Base</td><td>0.999</td><td>0.978</td><td>0.956</td><td>0.977</td><td>0.946</td><td>0.981</td><td>0.980</td></tr><tr><td/><td>FXLNET-Base</td><td>0.996</td><td>0.963</td><td>0.986</td><td>0.991</td><td>0.932</td><td>0.981</td><td>0.978</td></tr><tr><td/><td>PXLNET-Base (idf)</td><td>0.992</td><td>0.951</td><td>0.998</td><td>0.996</td><td>0.930</td><td>0.982</td><td>0.939</td></tr><tr><td/><td>RXLNET-Base (idf)</td><td>0.999</td><td>0.986</td><td>0.968</td><td>0.973</td><td>0.964</td><td>0.987</td><td>0.955</td></tr><tr><td/><td>FXLNET-Base (idf)</td><td>0.998</td><td>0.974</td><td>0.996</td><td>0.994</td><td>0.950</td><td>0.990</td><td>0.970</td></tr><tr><td/><td>PXLNET-Large RXLNET-Large</td><td>0.991 0.996</td><td>0.944 0.981</td><td>0.996 0.945</td><td>0.995 0.971</td><td>0.924 0.961</td><td>0.982 0.986</td><td>0.943 0.958</td></tr><tr><td/><td>FXLNET-Large</td><td>0.999</td><td>0.969</td><td>0.986</td><td>0.992</td><td>0.945</td><td>0.992</td><td>0.961</td></tr><tr><td/><td>PXLNET-Large (idf)</td><td>0.995</td><td>0.955</td><td>0.999</td><td>0.996</td><td>0.941</td><td>0.985</td><td>0.937</td></tr><tr><td/><td>RXLNET-Large (idf)</td><td>0.993</td><td>0.985</td><td>0.951</td><td>0.960</td><td>0.975</td><td>0.974</td><td>0.910</td></tr><tr><td/><td>FXLNET-Large (idf)</td><td>1.000</td><td>0.978</td><td>0.994</td><td>0.993</td><td>0.962</td><td>0.994</td><td>0.954</td></tr><tr><td/><td>PXLM-En RXLM-En</td><td>0.983 0.998</td><td>0.933 0.978</td><td>0.994 0.949</td><td>0.989 0.983</td><td>0.918 0.957</td><td>0.973 0.985</td><td>0.928 0.972</td></tr><tr><td/><td>FXLM-En</td><td>0.994</td><td>0.960</td><td>0.985</td><td>0.995</td><td>0.938</td><td>0.984</td><td>0.964</td></tr><tr><td/><td>PXLM-En (idf)</td><td>0.986</td><td>0.940</td><td>0.997</td><td>0.992</td><td>0.939</td><td>0.979</td><td>0.916</td></tr><tr><td/><td>RXLM-En (idf)</td><td>0.999</td><td>0.983</td><td>0.966</td><td>0.980</td><td>0.975</td><td>0.991</td><td>0.952</td></tr><tr><td/><td>FXLM-En (idf)</td><td>0.995</td><td>0.967</td><td>0.996</td><td>0.998</td><td>0.959</td><td>0.993</td><td>0.958</td></tr></table>

Absolute Pearson correlations with system-level human judgments on WMT17 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Table content: <table><tr><td>Setting</td><td>Metric</td><td colspan="3">en-cs en-de en-lv</td><td>en-ru</td><td>en-tr</td><td>en-zh</td></tr><tr><td/><td/><td>14</td><td>16</td><td>17</td><td>9</td><td>8</td><td>11</td></tr><tr><td/><td>BLEU</td><td colspan="5">0.956 0.804 0.866 0.898 0.924</td><td>-</td></tr><tr><td/><td>CDER</td><td colspan="5">0.968 0.813 0.930 0.924 0.957</td><td>-</td></tr><tr><td/><td>CHARACTER</td><td colspan="6">0.981 0.938 0.897 0.939 0.975 0.933</td></tr><tr><td/><td>CHRF</td><td colspan="6">0.976 0.863 0.955 0.950 0.991 0.976</td></tr><tr><td/><td>CHRF++</td><td colspan="6">0.974 0.852 0.956 0.945 0.986 0.976</td></tr><tr><td/><td>MEANT 2.0</td><td>-</td><td>0.858</td><td>-</td><td>-</td><td>-</td><td>0.956</td></tr><tr><td>Unsupervised</td><td colspan="7">MEANT 2.0-NOSRL 0.976 0.770 0.959 0.957 0.991 0.943</td></tr><tr><td/><td>NIST</td><td colspan="5">0.962 0.769 0.935 0.920 0.986</td><td>-</td></tr><tr><td/><td>PER</td><td colspan="5">0.954 0.687 0.851 0.887 0.963</td><td>-</td></tr><tr><td/><td>TER</td><td colspan="5">0.955 0.796 0.909 0.933 0.967</td><td>-</td></tr><tr><td/><td>TREEAGGREG</td><td colspan="6">0.947 0.773 0.927 0.921 0.983 0.938</td></tr><tr><td/><td>UHH TSKM</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>WER</td><td colspan="5">0.954 0.802 0.906 0.934 0.956</td><td>-</td></tr><tr><td/><td>AUTODA</td><td colspan="6">0.975 0.603 0.729 0.850 0.601 0.976</td></tr><tr><td/><td>BEER</td><td colspan="6">0.970 0.842 0.930 0.944 0.980 0.914</td></tr><tr><td>Supervised</td><td>BLEND</td><td>-</td><td>-</td><td>-</td><td>0.953</td><td>-</td><td>-</td></tr><tr><td/><td>BLEU2VEC</td><td colspan="5">0.963 0.810 0.859 0.903 0.911</td><td>-</td></tr><tr><td/><td>NGRAM2VEC</td><td>-</td><td>-</td><td>0.862</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>PBERT-Multi</td><td colspan="6">0.959 0.798 0.960 0.946 0.981 0.970</td></tr><tr><td/><td>RBERT-Multi</td><td colspan="6">0.982 0.909 0.957 0.980 0.979 0.994</td></tr><tr><td/><td>FBERT-Multi</td><td colspan="6">0.976 0.859 0.959 0.966 0.980 0.992</td></tr><tr><td/><td>PBERT-Multi (idf)</td><td colspan="6">0.963 0.760 0.960 0.947 0.984 0.971</td></tr><tr><td/><td>RBERT-Multi (idf)</td><td colspan="6">0.985 0.907 0.955 0.981 0.984 0.982</td></tr><tr><td>Pre-Trained</td><td>FBERT-Multi (idf)</td><td colspan="6">0.979 0.841 0.958 0.968 0.984 0.991</td></tr><tr><td/><td>PXLM-100</td><td colspan="6">0.967 0.825 0.965 0.953 0.974 0.977</td></tr><tr><td/><td>RXLM-100</td><td colspan="6">0.980 0.902 0.965 0.982 0.977 0.979</td></tr><tr><td/><td>FXLM-100</td><td colspan="6">0.979 0.868 0.969 0.971 0.976 0.986</td></tr><tr><td/><td>PXLM-100 (idf)</td><td colspan="6">0.968 0.809 0.965 0.955 0.980 0.975</td></tr><tr><td/><td>RXLM-100 (idf)</td><td colspan="6">0.981 0.894 0.964 0.984 0.983 0.968</td></tr><tr><td/><td>FXLM-100 (idf)</td><td colspan="6">0.979 0.856 0.966 0.973 0.982 0.979</td></tr></table>

Absolute Pearson correlations with system-level human judgments on WMT17 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Table content: <table><tr><td>Setting</td><td>Metric</td><td>cs-en</td><td>de-en</td><td>et-en</td><td>fi-en</td><td>ru-en</td><td>tr-en</td><td>zh-en</td></tr><tr><td/><td/><td>5K</td><td>78K</td><td>57K</td><td>16K</td><td>10K</td><td>9K</td><td>33K</td></tr><tr><td/><td>CHARACTER</td><td>0.256</td><td>0.450</td><td>0.286</td><td>0.185</td><td>0.244</td><td>0.172</td><td>0.202</td></tr><tr><td/><td>ITER</td><td>0.198</td><td>0.396</td><td>0.235</td><td>0.128</td><td>0.139</td><td>-0.029</td><td>0.144</td></tr><tr><td/><td>METEOR++</td><td>0.270</td><td>0.457</td><td>0.329</td><td>0.207</td><td>0.253</td><td>0.204</td><td>0.179</td></tr><tr><td>Unsupervised</td><td>SENTBLEU UHH TSKM</td><td>0.233 0.274</td><td>0.415 0.436</td><td>0.285 0.300</td><td>0.154 0.168</td><td>0.228 0.235</td><td>0.145 0.154</td><td>0.178 0.151</td></tr><tr><td/><td>YISI-0</td><td>0.301</td><td>0.474</td><td>0.330</td><td>0.225</td><td>0.294</td><td>0.215</td><td>0.205</td></tr><tr><td/><td>YISI-1</td><td>0.319</td><td>0.488</td><td>0.351</td><td>0.231</td><td>0.300</td><td>0.234</td><td>0.211</td></tr><tr><td/><td>YISI-1 SRL</td><td>0.317</td><td>0.483</td><td>0.345</td><td>0.237</td><td>0.306</td><td>0.233</td><td>0.209</td></tr><tr><td/><td>BEER</td><td>0.295</td><td>0.481</td><td>0.341</td><td>0.232</td><td>0.288</td><td>0.229</td><td>0.214</td></tr><tr><td>Supervised</td><td>BLEND</td><td>0.322</td><td>0.492</td><td>0.354</td><td>0.226</td><td>0.290</td><td>0.232</td><td>0.217</td></tr><tr><td/><td>RUSE</td><td>0.347</td><td>0.498</td><td>0.368</td><td>0.273</td><td>0.311</td><td>0.259</td><td>0.218</td></tr><tr><td/><td>PBERT-Base RBERT-Base</td><td>0.349 0.370</td><td>0.522 0.528</td><td>0.373 0.378</td><td>0.264 0.291</td><td>0.325 0.333</td><td>0.264 0.257</td><td>0.232 0.244</td></tr><tr><td/><td>FBERT-Base</td><td>0.373</td><td>0.531</td><td>0.385</td><td>0.287</td><td>0.341</td><td>0.266</td><td>0.243</td></tr><tr><td/><td>PBERT-Base (idf)</td><td>0.352</td><td>0.524</td><td>0.382</td><td>0.27</td><td>0.326</td><td>0.277</td><td>0.235</td></tr><tr><td/><td>RBERT-Base (idf)</td><td>0.368</td><td>0.536</td><td>0.388</td><td>0.300</td><td>0.340</td><td>0.284</td><td>0.244</td></tr><tr><td/><td>FBERT-Base (idf)</td><td>0.375</td><td>0.535</td><td>0.393</td><td>0.294</td><td>0.339</td><td>0.289</td><td>0.243</td></tr><tr><td/><td>PBERT-Base-MRPC</td><td>0.343</td><td>0.520</td><td>0.365</td><td>0.247</td><td>0.333</td><td>0.25</td><td>0.227</td></tr><tr><td/><td>RBERT-Base-MRPC FBERT-Base-MRPC</td><td>0.370 0.366</td><td>0.524 0.529</td><td>0.373 0.377</td><td>0.277 0.271</td><td>0.34 0.342</td><td>0.261 0.263</td><td>0.244 0.242</td></tr><tr><td/><td>PBERT-Base-MRPC (idf)</td><td>0.348</td><td>0.522</td><td>0.371</td><td>0.25</td><td>0.318</td><td>0.256</td><td>0.224</td></tr><tr><td/><td>RBERT-Base-MRPC (idf)</td><td>0.379</td><td>0.531</td><td>0.383</td><td>0.285</td><td>0.339</td><td>0.266</td><td>0.242</td></tr><tr><td/><td>FBERT-Base-MRPC (idf)</td><td>0.373</td><td>0.534</td><td>0.383</td><td>0.274</td><td>0.342</td><td>0.275</td><td>0.242</td></tr><tr><td/><td>PBERT-LARGE</td><td>0.361</td><td>0.529</td><td>0.380</td><td>0.276</td><td>0.340</td><td>0.266</td><td>0.241</td></tr><tr><td/><td>RBERT-LARGE FBERT-LARGE</td><td>0.386 0.402</td><td>0.532 0.537</td><td>0.386 0.390</td><td>0.297 0.296</td><td>0.347 0.344</td><td>0.268 0.274</td><td>0.247 0.252</td></tr><tr><td/><td>PBERT-LARGE (idf)</td><td>0.377</td><td>0.532</td><td>0.390</td><td>0.287</td><td>0.342</td><td>0.292</td><td>0.246</td></tr><tr><td/><td>RBERT-LARGE (idf)</td><td>0.386</td><td>0.544</td><td>0.396</td><td>0.308</td><td>0.356</td><td>0.287</td><td>0.251</td></tr><tr><td/><td>FBERT-LARGE (idf)</td><td>0.388</td><td>0.545</td><td>0.399</td><td>0.309</td><td>0.358</td><td>0.300</td><td>0.257</td></tr><tr><td/><td>PRoBERTa-Base</td><td>0.368</td><td>0.53</td><td>0.371</td><td>0.274</td><td>0.318</td><td>0.265</td><td>0.235</td></tr><tr><td/><td>RRoBERTa-Base</td><td>0.383</td><td>0.536</td><td>0.376</td><td>0.283</td><td>0.336</td><td>0.253</td><td>0.245</td></tr><tr><td/><td>FRoBERTa-Base PRoBERTa-Base (idf)</td><td>0.391 0.379</td><td>0.540 0.528</td><td>0.383 0.372</td><td>0.273 0.261</td><td>0.339 0.314</td><td>0.270 0.265</td><td>0.249 0.232</td></tr><tr><td/><td>RRoBERTa-Base (idf)</td><td>0.389</td><td>0.539</td><td>0.384</td><td>0.288</td><td>0.332</td><td>0.267</td><td>0.245</td></tr><tr><td/><td>FRoBERTa-Base (idf)</td><td>0.400</td><td>0.540</td><td>0.385</td><td>0.274</td><td>0.337</td><td>0.277</td><td>0.247</td></tr><tr><td>Pre-Trained</td><td>PRoBERTa-LARGE</td><td>0.387</td><td>0.541</td><td>0.389</td><td>0.283</td><td>0.345</td><td>0.280</td><td>0.248</td></tr><tr><td/><td>RRoBERTa-LARGE</td><td>0.388</td><td>0.546</td><td>0.391</td><td>0.304</td><td>0.343</td><td>0.290</td><td>0.255</td></tr><tr><td/><td>FRoBERTa-LARGE PRoBERTa-LARGE (idf)</td><td>0.404 0.391</td><td>0.550 0.540</td><td>0.397 0.387</td><td>0.296 0.280</td><td>0.353 0.334</td><td>0.292 0.284</td><td>0.264 0.252</td></tr><tr><td/><td>RRoBERTa-LARGE (idf)</td><td>0.386</td><td>0.548</td><td>0.394</td><td>0.305</td><td>0.338</td><td>0.295</td><td>0.252</td></tr><tr><td/><td>FRoBERTa-LARGE (idf)</td><td>0.408</td><td>0.550</td><td>0.395</td><td>0.293</td><td>0.346</td><td>0.296</td><td>0.260</td></tr><tr><td/><td>PRoBERTa-Large-MNLI</td><td>0.397</td><td>0.549</td><td>0.396</td><td>0.299</td><td>0.351</td><td>0.295</td><td>0.253</td></tr><tr><td/><td>RRoBERTa-Large-MNLI</td><td>0.404</td><td>0.553</td><td>0.393</td><td>0.313</td><td>0.351</td><td>0.279</td><td>0.253</td></tr><tr><td/><td>FRoBERTa-Large-MNLI PRoBERTa-Large-MNLI (idf)</td><td>0.418 0.414</td><td>0.557 0.552</td><td>0.402 0.399</td><td>0.312 0.301</td><td>0.362 0.349</td><td>0.290 0.306</td><td>0.258 0.249</td></tr><tr><td/><td>RRoBERTa-Large-MNLI (idf)</td><td>0.412</td><td>0.555</td><td>0.400</td><td>0.316</td><td>0.357</td><td>0.289</td><td>0.258</td></tr><tr><td/><td>FRoBERTa-Large-MNLI (idf)</td><td>0.417</td><td>0.559</td><td>0.403</td><td>0.309</td><td>0.357</td><td>0.307</td><td>0.258</td></tr><tr><td/><td>PXLNet-Base</td><td>0.335</td><td>0.514</td><td>0.359</td><td>0.243</td><td>0.308</td><td>0.247</td><td>0.232</td></tr><tr><td/><td>RXLNet-Base</td><td>0.351</td><td>0.515</td><td>0.362</td><td>0.261</td><td>0.311</td><td>0.227</td><td>0.232</td></tr><tr><td/><td>FXLNet-Base</td><td>0.351</td><td>0.517</td><td>0.365</td><td>0.257</td><td>0.315</td><td>0.25</td><td>0.237</td></tr><tr><td/><td>PXLNet-Base (idf) RXLNet-Base (idf)</td><td>0.339 0.364</td><td>0.516 0.521</td><td>0.366 0.371</td><td>0.258 0.268</td><td>0.307 0.317</td><td>0.261 0.242</td><td>0.236 0.238</td></tr><tr><td/><td>FXLNet-Base (idf)</td><td>0.355</td><td>0.524</td><td>0.374</td><td>0.265</td><td>0.320</td><td>0.261</td><td>0.241</td></tr><tr><td/><td>PXL-NET-LARGE</td><td>0.344</td><td>0.522</td><td>0.371</td><td>0.252</td><td>0.316</td><td>0.264</td><td>0.233</td></tr><tr><td/><td>RXL-NET-LARGE</td><td>0.358</td><td>0.524</td><td>0.374</td><td>0.275</td><td>0.332</td><td>0.249</td><td>0.239</td></tr><tr><td/><td>FXL-NET-LARGE</td><td>0.357</td><td>0.530</td><td>0.380</td><td>0.265</td><td>0.334</td><td>0.263</td><td>0.238</td></tr><tr><td/><td>PXL-NET-LARGE (idf) RXL-NET-LARGE (idf)</td><td>0.348 0.366</td><td>0.520 0.529</td><td>0.373 0.378</td><td>0.260 0.278</td><td>0.319 0.331</td><td>0.265 0.266</td><td>0.235 0.241</td></tr><tr><td/><td>FXL-NET-LARGE (idf)</td><td>0.375</td><td>0.530</td><td>0.382</td><td>0.274</td><td>0.332</td><td>0.274</td><td>0.240</td></tr><tr><td/><td>PXLM-En</td><td>0.349</td><td>0.516</td><td>0.366</td><td>0.244</td><td>0.310</td><td>0.259</td><td>0.233</td></tr><tr><td/><td>RXLM-En</td><td>0.358</td><td>0.518</td><td>0.364</td><td>0.264</td><td>0.320</td><td>0.244</td><td>0.237</td></tr><tr><td/><td>FXLM-En</td><td>0.358</td><td>0.525</td><td>0.373</td><td>0.259</td><td>0.322</td><td>0.258</td><td>0.238</td></tr><tr><td/><td>PXLM-En (idf)</td><td>0.355</td><td>0.527</td><td>0.374</td><td>0.254</td><td>0.311</td><td>0.28</td><td>0.238</td></tr><tr><td/><td>RXLM-En (idf) FXLM-En (idf)</td><td>0.362 0.367</td><td>0.528 0.531</td><td>0.376 0.382</td><td>0.274 0.273</td><td>0.333 0.330</td><td>0.26 0.275</td><td>0.24 0.246</td></tr></table>

Kendall correlations with segment-level human judgments on WMT18 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Table content: <table><tr><td>Setting</td><td>Metric</td><td colspan="3">en-cs en-de en-et</td><td>en-fi</td><td>en-ru</td><td>en-tr</td><td>en-zh</td></tr><tr><td/><td/><td>5K</td><td>20K</td><td>32K</td><td>10K</td><td>22K</td><td>1K</td><td>29K</td></tr><tr><td/><td>CHARACTER</td><td colspan="7">0.414 0.604 0.464 0.403 0.352 0.404 0.313</td></tr><tr><td/><td>ITER</td><td colspan="6">0.333 0.610 0.392 0.311 0.291 0.236</td><td>-</td></tr><tr><td>Unsupervised</td><td>SENTBLEU YISI-0</td><td colspan="7">0.389 0.620 0.414 0.355 0.330 0.261 0.311 0.471 0.661 0.531 0.464 0.394 0.376 0.318</td></tr><tr><td/><td>YISI-1</td><td colspan="7">0.496 0.691 0.546 0.504 0.407 0.418 0.323</td></tr><tr><td/><td>YISI-1 SRL</td><td>-</td><td>0.696</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.310</td></tr><tr><td>Supervised</td><td>BEER BLEND</td><td colspan="7">0.518 0.686 0.558 0.511 0.403 0.374 0.302 ----0.394 --</td></tr><tr><td/><td>PBERT-Multi</td><td colspan="7">0.541 0.715 0.549 0.486 0.414 0.328 0.337</td></tr><tr><td/><td>RBERT-Multi</td><td colspan="7">0.570 0.728 0.594 0.565 0.420 0.411 0.367</td></tr><tr><td/><td>FBERT-Multi</td><td colspan="7">0.562 0.728 0.586 0.546 0.423 0.399 0.364</td></tr><tr><td/><td colspan="2">PBERT-Multi (idf) 0.525</td><td>0.7</td><td>0.54</td><td colspan="4">0.495 0.423 0.352 0.338</td></tr><tr><td/><td colspan="8">RBERT-Multi (idf) 0.569 0.727 0.601 0.561 0.423 0.420 0.374</td></tr><tr><td>Pre-Trained</td><td colspan="8">FBERT-Multi (idf) 0.553 0.721 0.585 0.537 0.425 0.406 0.366</td></tr><tr><td/><td>PXLM-100</td><td colspan="7">0.496 0.711 0.561 0.527 0.417 0.364 0.340</td></tr><tr><td/><td>RXLM-100</td><td colspan="7">0.564 0.724 0.612 0.584 0.418 0.432 0.363</td></tr><tr><td/><td>FXLM-100</td><td colspan="7">0.533 0.727 0.599 0.573 0.421 0.408 0.362</td></tr><tr><td/><td>PXLM-100 (idf)</td><td colspan="7">0.520 0.710 0.572 0.546 0.421 0.370 0.328</td></tr><tr><td/><td>RXLM-100 (idf)</td><td colspan="7">0.567 0.722 0.609 0.587 0.420 0.439 0.365</td></tr><tr><td/><td>FXLM-100 (idf)</td><td colspan="7">0.554 0.724 0.601 0.584 0.422 0.389 0.355</td></tr></table>

Kendall correlations with segment-level human judgments on WMT18 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Table content: <table><tr><td>Setting</td><td>Metric</td><td>cs-en</td><td>de-en</td><td>et-en</td><td>fi-en</td><td>ru-en</td><td>tr-en</td><td>zh-en</td></tr><tr><td/><td/><td>5</td><td>16</td><td>14</td><td>9</td><td>8</td><td>5</td><td>14</td></tr><tr><td/><td>BLEU</td><td>0.970</td><td>0.971</td><td>0.986</td><td>0.973</td><td>0.979</td><td>0.657</td><td>0.978</td></tr><tr><td/><td>CDER</td><td>0.972</td><td>0.980</td><td>0.990</td><td>0.984</td><td>0.980</td><td>0.664</td><td>0.982</td></tr><tr><td/><td>CHARACTER</td><td>0.970</td><td>0.993</td><td>0.979</td><td>0.989</td><td>0.991</td><td>0.782</td><td>0.950</td></tr><tr><td/><td>ITER</td><td>0.975</td><td>0.990</td><td>0.975</td><td>0.996</td><td>0.937</td><td>0.861</td><td>0.980</td></tr><tr><td/><td>METEOR++</td><td>0.945</td><td>0.991</td><td>0.978</td><td>0.971</td><td>0.995</td><td>0.864</td><td>0.962</td></tr><tr><td/><td>NIST</td><td>0.954</td><td>0.984</td><td>0.983</td><td>0.975</td><td>0.973</td><td>0.970</td><td>0.968</td></tr><tr><td>Unsupervised</td><td>PER</td><td>0.970</td><td>0.985</td><td>0.983</td><td>0.993</td><td>0.967</td><td>0.159</td><td>0.931</td></tr><tr><td/><td>TER</td><td>0.950</td><td>0.970</td><td>0.990</td><td>0.968</td><td>0.970</td><td>0.533</td><td>0.975</td></tr><tr><td/><td>UHH TSKM</td><td>0.952</td><td>0.980</td><td>0.989</td><td>0.982</td><td>0.980</td><td>0.547</td><td>0.981</td></tr><tr><td/><td>WER</td><td>0.951</td><td>0.961</td><td>0.991</td><td>0.961</td><td>0.968</td><td>0.041</td><td>0.975</td></tr><tr><td/><td>YISI-0</td><td>0.956</td><td>0.994</td><td>0.975</td><td>0.978</td><td>0.988</td><td>0.954</td><td>0.957</td></tr><tr><td/><td>YISI-1</td><td>0.950</td><td>0.992</td><td>0.979</td><td>0.973</td><td>0.991</td><td>0.958</td><td>0.951</td></tr><tr><td/><td>YISI-1 SRL</td><td>0.965</td><td>0.995</td><td>0.981</td><td>0.977</td><td>0.992</td><td>0.869</td><td>0.962</td></tr><tr><td/><td>BEER</td><td>0.958</td><td>0.994</td><td>0.985</td><td>0.991</td><td>0.982</td><td>0.870</td><td>0.976</td></tr><tr><td>Supervised</td><td>BLEND</td><td>0.973</td><td>0.991</td><td>0.985</td><td>0.994</td><td>0.993</td><td>0.801</td><td>0.976</td></tr><tr><td/><td>RUSE</td><td>0.981</td><td>0.997</td><td>0.990</td><td>0.991</td><td>0.988</td><td>0.853</td><td>0.981</td></tr><tr><td/><td>PBERT-Base</td><td>0.965</td><td>0.995</td><td>0.986</td><td>0.973</td><td>0.976</td><td>0.941</td><td>0.974</td></tr><tr><td/><td>RBERT-Base</td><td>0.994</td><td>0.991</td><td>0.979</td><td>0.992</td><td>0.991</td><td>0.067</td><td>0.988</td></tr><tr><td/><td>FBERT-Base</td><td>0.982</td><td>0.994</td><td>0.983</td><td>0.986</td><td>0.985</td><td>0.949</td><td>0.984</td></tr><tr><td/><td>PBERT-Base (idf)</td><td>0.961</td><td>0.993</td><td>0.987</td><td>0.988</td><td>0.976</td><td>0.984</td><td>0.973</td></tr><tr><td/><td>RBERT-Base (idf) FBERT-Base (idf)</td><td>0.996 0.981</td><td>0.994 0.995</td><td>0.977 0.984</td><td>0.995 0.995</td><td>0.995 0.988</td><td>0.874 0.994</td><td>0.983 0.981</td></tr><tr><td/><td>PBERT-Base-MRPC</td><td>0.957</td><td>0.994</td><td>0.989</td><td>0.953</td><td>0.976</td><td>0.798</td><td>0.977</td></tr><tr><td/><td>RBERT-Base-MRPC</td><td>0.992</td><td>0.994</td><td>0.983</td><td>0.988</td><td>0.993</td><td>0.707</td><td>0.990</td></tr><tr><td/><td>FBERT-Base-MRPC</td><td>0.975</td><td>0.995</td><td>0.987</td><td>0.975</td><td>0.986</td><td>0.526</td><td>0.986</td></tr><tr><td/><td>PBERT-Base-MRPC (idf)</td><td>0.957</td><td>0.997</td><td>0.989</td><td>0.967</td><td>0.975</td><td>0.894</td><td>0.980</td></tr><tr><td/><td>RBERT-Base-MRPC (idf)</td><td>0.991</td><td>0.997</td><td>0.981</td><td>0.994</td><td>0.993</td><td>0.052</td><td>0.987</td></tr><tr><td/><td>FBERT-Base-MRPC (idf)</td><td>0.975</td><td>0.998</td><td>0.987</td><td>0.985</td><td>0.987</td><td>0.784</td><td>0.987</td></tr><tr><td/><td>PBERT-Large</td><td>0.978</td><td>0.992</td><td>0.987</td><td>0.971</td><td>0.977</td><td>0.920</td><td>0.978</td></tr><tr><td/><td>RBERT-Large</td><td>0.997</td><td>0.990</td><td>0.985</td><td>0.990</td><td>0.992</td><td>0.098</td><td>0.990</td></tr><tr><td/><td>FBERT-Large</td><td>0.989</td><td>0.992</td><td>0.987</td><td>0.983</td><td>0.985</td><td>0.784</td><td>0.986</td></tr><tr><td/><td>PBERT-Large (idf)</td><td>0.977</td><td>0.992</td><td>0.988</td><td>0.986</td><td>0.976</td><td>0.980</td><td>0.977</td></tr><tr><td/><td>RBERT-Large (idf)</td><td>0.998</td><td>0.993</td><td>0.983</td><td>0.996</td><td>0.995</td><td>0.809</td><td>0.986</td></tr><tr><td/><td>FBERT-Large (idf)</td><td>0.989</td><td>0.993</td><td>0.986</td><td>0.993</td><td>0.987</td><td>0.976</td><td>0.984</td></tr><tr><td/><td>PRoBERTa-Base</td><td>0.970</td><td>0.995</td><td>0.991</td><td>0.998</td><td>0.976</td><td>0.796</td><td>0.980</td></tr><tr><td/><td>RRoBERTa-Base</td><td>0.996</td><td>0.996</td><td>0.982</td><td>0.998</td><td>0.994</td><td>0.477</td><td>0.991</td></tr><tr><td/><td>FRoBERTa-Base</td><td>0.984</td><td>0.997</td><td>0.989</td><td>0.999</td><td>0.987</td><td>0.280</td><td>0.989</td></tr><tr><td/><td>PRoBERTa-Base (idf)</td><td>0.966</td><td>0.993</td><td>0.991</td><td>0.994</td><td>0.977</td><td>0.880</td><td>0.984</td></tr><tr><td/><td>RRoBERTa-Base (idf)</td><td>0.995</td><td>0.998</td><td>0.981</td><td>0.998</td><td>0.995</td><td>0.230</td><td>0.989</td></tr><tr><td/><td>FRoBERTa-Base (idf)</td><td>0.981</td><td>0.998</td><td>0.989</td><td>0.997</td><td>0.988</td><td>0.741</td><td>0.990</td></tr><tr><td>Pre-Trained</td><td>PRoBERTa-Large RRoBERTa-Large</td><td>0.980 0.998</td><td>0.998 0.997</td><td>0.990 0.986</td><td>0.995 0.997</td><td>0.982 0.995</td><td>0.791 0.054</td><td>0.981 0.990</td></tr><tr><td/><td>FRoBERTa-Large</td><td>0.990</td><td>0.999</td><td>0.990</td><td>0.998</td><td>0.990</td><td>0.499</td><td>0.988</td></tr><tr><td/><td>PRoBERTa-Large (idf)</td><td>0.972</td><td>0.997</td><td>0.993</td><td>0.985</td><td>0.982</td><td>0.920</td><td>0.983</td></tr><tr><td/><td>RRoBERTa-Large (idf)</td><td>0.996</td><td>0.997</td><td>0.984</td><td>0.997</td><td>0.995</td><td>0.578</td><td>0.989</td></tr><tr><td/><td>FRoBERTa-Large (idf)</td><td>0.985</td><td>0.999</td><td>0.992</td><td>0.992</td><td>0.991</td><td>0.826</td><td>0.989</td></tr><tr><td/><td>PRoBERTa-Large-MNLI RRoBERTa-Large-MNLI</td><td>0.989 1.000</td><td>0.998 0.996</td><td>0.994 0.988</td><td>0.998 0.996</td><td>0.985 0.995</td><td>0.908 0.097</td><td>0.982 0.991</td></tr><tr><td/><td>FRoBERTa-Large-MNLI</td><td>0.996</td><td>0.998</td><td>0.992</td><td>0.998</td><td>0.992</td><td>0.665</td><td>0.989</td></tr><tr><td/><td>PRoBERTa-Large-MNLI (idf)</td><td>0.986</td><td>0.998</td><td>0.994</td><td>0.993</td><td>0.986</td><td>0.989</td><td>0.985</td></tr><tr><td/><td>RRoBERTa-Large-MNLI (idf)</td><td>0.999</td><td>0.997</td><td>0.986</td><td>0.997</td><td>0.993</td><td>0.633</td><td>0.990</td></tr><tr><td/><td>FRoBERTa-Large-MNLI (idf)</td><td>0.995</td><td>0.998</td><td>0.991</td><td>0.996</td><td>0.993</td><td>0.963</td><td>0.990</td></tr><tr><td/><td>PXLNET-Base RXLNET-Base</td><td>0.970 0.994</td><td>0.996 0.997</td><td>0.986 0.979</td><td>0.990 0.995</td><td>0.979 0.994</td><td>0.739 0.795</td><td>0.982 0.990</td></tr><tr><td/><td>FXLNET-Base</td><td>0.983</td><td>0.997</td><td>0.983</td><td>0.993</td><td>0.987</td><td>0.505</td><td>0.988</td></tr><tr><td/><td>PXLNET-Base (idf)</td><td>0.968</td><td>0.998</td><td>0.986</td><td>0.990</td><td>0.978</td><td>0.923</td><td>0.982</td></tr><tr><td/><td>RXLNET-Base (idf)</td><td>0.993</td><td>0.998</td><td>0.978</td><td>0.996</td><td>0.994</td><td>0.439</td><td>0.988</td></tr><tr><td/><td>FXLNET-Base (idf)</td><td>0.981</td><td>0.999</td><td>0.984</td><td>0.995</td><td>0.989</td><td>0.722</td><td>0.988</td></tr><tr><td/><td>PXLNET-Large</td><td>0.969</td><td>0.998</td><td>0.986</td><td>0.995</td><td>0.979</td><td>0.880</td><td>0.981</td></tr><tr><td/><td>RXLNET-Large FXLNET-Large</td><td>0.995 0.983</td><td>0.997 0.998</td><td>0.977 0.983</td><td>0.997 0.997</td><td>0.995 0.988</td><td>0.430 0.713</td><td>0.988 0.988</td></tr><tr><td/><td>PXLNET-Large (idf)</td><td>0.963</td><td>0.996</td><td>0.986</td><td>0.995</td><td>0.978</td><td>0.939</td><td>0.979</td></tr><tr><td/><td>RXLNET-Large (idf)</td><td>0.992</td><td>0.997</td><td>0.975</td><td>0.993</td><td>0.996</td><td>0.531</td><td>0.982</td></tr><tr><td/><td>FXLNET-Large (idf)</td><td>0.978</td><td>0.997</td><td>0.983</td><td>0.996</td><td>0.990</td><td>0.886</td><td>0.984</td></tr><tr><td/><td>PXLM-En</td><td>0.965</td><td>0.996</td><td>0.990</td><td>0.978</td><td>0.980</td><td>0.946</td><td>0.981</td></tr><tr><td/><td>RXLM-En FXLM-En</td><td>0.990 0.978</td><td>0.995 0.997</td><td>0.984 0.988</td><td>0.996 0.990</td><td>0.996 0.989</td><td>0.286 0.576</td><td>0.987 0.987</td></tr><tr><td/><td>PXLM-En (idf)</td><td>0.960</td><td>0.996</td><td>0.990</td><td>0.987</td><td>0.980</td><td>0.989</td><td>0.981</td></tr><tr><td/><td>RXLM-En (idf)</td><td>0.991</td><td>0.997</td><td>0.983</td><td>0.996</td><td>0.998</td><td>0.612</td><td>0.985</td></tr><tr><td/><td>FXLM-En (idf)</td><td>0.976</td><td>0.998</td><td>0.988</td><td>0.994</td><td>0.992</td><td>0.943</td><td>0.985</td></tr></table>

Absolute Pearson correlations with system-level human judgments on WMT18 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Table content: <table><tr><td>Setting</td><td>Metric</td><td colspan="3">en-cs en-de en-et</td><td>en-fi</td><td>en-ru</td><td>en-tr</td><td>en-zh</td></tr><tr><td/><td/><td>5</td><td>16</td><td>14</td><td>12</td><td>9</td><td>8</td><td>14</td></tr><tr><td/><td>BLEU</td><td colspan="7">0.995 0.981 0.975 0.962 0.983 0.826 0.947</td></tr><tr><td/><td>CDER</td><td colspan="7">0.997 0.986 0.984 0.964 0.984 0.861 0.961</td></tr><tr><td/><td>CHARACTER</td><td colspan="7">0.993 0.989 0.956 0.974 0.983 0.833 0.983</td></tr><tr><td/><td>ITER</td><td colspan="6">0.915 0.984 0.981 0.973 0.975 0.865</td><td>-</td></tr><tr><td/><td>METEOR++</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>NIST</td><td colspan="7">0.999 0.986 0.983 0.949 0.990 0.902 0.950</td></tr><tr><td>Unsupervised</td><td>PER</td><td colspan="7">0.991 0.981 0.958 0.906 0.988 0.859 0.964</td></tr><tr><td/><td>TER</td><td colspan="7">0.997 0.988 0.981 0.942 0.987 0.867 0.963</td></tr><tr><td/><td>UHH TSKM</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>WER</td><td colspan="7">0.997 0.986 0.981 0.945 0.985 0.853 0.957</td></tr><tr><td/><td>YISI-0</td><td colspan="7">0.973 0.985 0.968 0.944 0.990 0.990 0.957</td></tr><tr><td/><td>YISI-1</td><td colspan="7">0.987 0.985 0.979 0.940 0.992 0.976 0.963</td></tr><tr><td/><td>YISI-1 SRL</td><td>-</td><td>0.990</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.952</td></tr><tr><td/><td>BEER</td><td colspan="7">0.992 0.991 0.980 0.961 0.988 0.965 0.928</td></tr><tr><td>Supervised</td><td>BLEND</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.988</td><td>-</td><td>-</td></tr><tr><td/><td>RUSE</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>PBERT-Multi</td><td colspan="7">0.994 0.988 0.981 0.957 0.990 0.935 0.954</td></tr><tr><td/><td>RBERT-Multi</td><td colspan="7">0.997 0.990 0.980 0.980 0.989 0.976</td></tr><tr><td/><td>FBERT-Multi</td><td colspan="7">0.997 0.989 0.982 0.972 0.990 0.908 0.967</td></tr><tr><td/><td colspan="8">PBERT-Multi (idf) 0.992 0.986 0.974 0.954 0.991 0.969 0.954</td></tr><tr><td/><td colspan="8">RBERT-Multi (idf) 0.997 0.993 0.982 0.982 0.992 0.901 0.984</td></tr><tr><td>Pre-Trained</td><td colspan="7">FBERT-Multi (idf) 0.995 0.990 0.981 0.972 0.991 0.941</td><td/></tr><tr><td/><td>PXLM-100</td><td colspan="7">0.984 0.992 0.993 0.972 0.993 0.962 0.965</td></tr><tr><td/><td>RXLM-100</td><td colspan="7">0.991 0.992 0.992 0.989 0.992 0.895 0.983</td></tr><tr><td/><td>FXLM-100</td><td colspan="7">0.988 0.993 0.993 0.986 0.993 0.935 0.976</td></tr><tr><td/><td>PXLM-100 (idf)</td><td colspan="7">0.982 0.992 0.994 0.975 0.993 0.968 0.964</td></tr><tr><td/><td>RXLM-100 (idf)</td><td colspan="7">0.993 0.993 0.991 0.989 0.993 0.911 0.986</td></tr><tr><td/><td>FXLM-100 (idf)</td><td colspan="7">0.989 0.993 0.994 0.985 0.993 0.945 0.979</td></tr></table>

Absolute Pearson correlations with system-level human judgments on WMT18 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Table content: <table><tr><td>Setting</td><td>Metric</td><td>cs-en</td><td>de-en</td><td>et-en</td><td>fi-en</td><td>ru-en</td><td>tr-en</td><td>zh-en</td></tr><tr><td/><td/><td>10K</td><td>10K</td><td>10K</td><td>10K</td><td>10K</td><td>10K</td><td>10K</td></tr><tr><td/><td>BLEU</td><td>0.956</td><td>0.969</td><td>0.981</td><td>0.962</td><td>0.972</td><td>0.586</td><td>0.968</td></tr><tr><td/><td>CDER</td><td>0.964</td><td>0.980</td><td>0.988</td><td>0.976</td><td>0.974</td><td>0.577</td><td>0.973</td></tr><tr><td/><td>CHARACTER</td><td>0.960</td><td>0.992</td><td>0.975</td><td>0.979</td><td>0.984</td><td>0.680</td><td>0.942</td></tr><tr><td/><td>ITER</td><td>0.966</td><td>0.990</td><td>0.975</td><td>0.989</td><td>0.943</td><td>0.742</td><td>0.978</td></tr><tr><td/><td>METEOR++</td><td>0.937</td><td>0.990</td><td>0.975</td><td>0.962</td><td>0.989</td><td>0.787</td><td>0.954</td></tr><tr><td/><td>NIST</td><td>0.942</td><td>0.982</td><td>0.980</td><td>0.965</td><td>0.965</td><td>0.862</td><td>0.959</td></tr><tr><td>Unsupervised</td><td>PER</td><td>0.937</td><td>0.982</td><td>0.978</td><td>0.983</td><td>0.955</td><td>0.043</td><td>0.923</td></tr><tr><td/><td>TER</td><td>0.942</td><td>0.970</td><td>0.988</td><td>0.960</td><td>0.963</td><td>0.450</td><td>0.967</td></tr><tr><td/><td>UHH TSKM</td><td>0.943</td><td>0.979</td><td>0.987</td><td>0.974</td><td>0.973</td><td>0.443</td><td>0.972</td></tr><tr><td/><td>WER</td><td>0.942</td><td>0.961</td><td>0.989</td><td>0.953</td><td>0.962</td><td>0.072</td><td>0.967</td></tr><tr><td/><td>YISI-0</td><td>0.947</td><td>0.992</td><td>0.972</td><td>0.969</td><td>0.982</td><td>0.863</td><td>0.950</td></tr><tr><td/><td>YISI-1</td><td>0.942</td><td>0.991</td><td>0.976</td><td>0.964</td><td>0.985</td><td>0.881</td><td>0.943</td></tr><tr><td/><td>YISI-1 SRL</td><td>0.957</td><td>0.994</td><td>0.978</td><td>0.968</td><td>0.986</td><td>0.785</td><td>0.954</td></tr><tr><td/><td>BEER</td><td>0.950</td><td>0.993</td><td>0.983</td><td>0.982</td><td>0.976</td><td>0.723</td><td>0.968</td></tr><tr><td>Supervised</td><td>BLEND</td><td>0.965</td><td>0.990</td><td>0.982</td><td>0.985</td><td>0.986</td><td>0.724</td><td>0.969</td></tr><tr><td/><td>RUSE</td><td>0.974</td><td>0.996</td><td>0.988</td><td>0.983</td><td>0.982</td><td>0.780</td><td>0.973</td></tr><tr><td/><td>PBERT-Base</td><td>0.954</td><td>0.992</td><td>0.984</td><td>0.980</td><td>0.970</td><td>0.917</td><td>0.965</td></tr><tr><td/><td>RBERT-Base FBERT-Base</td><td>0.988 0.973</td><td>0.994 0.994</td><td>0.974 0.981</td><td>0.987 0.987</td><td>0.988 0.982</td><td>0.801 0.924</td><td>0.975 0.973</td></tr><tr><td/><td>PBERT-Base (idf)</td><td>0.957</td><td>0.994</td><td>0.983</td><td>0.966</td><td>0.970</td><td>0.875</td><td>0.966</td></tr><tr><td/><td>RBERT-Base (idf)</td><td>0.986</td><td>0.990</td><td>0.976</td><td>0.984</td><td>0.984</td><td>0.019</td><td>0.980</td></tr><tr><td/><td>FBERT-Base (idf)</td><td>0.974</td><td>0.993</td><td>0.980</td><td>0.978</td><td>0.978</td><td>0.853</td><td>0.976</td></tr><tr><td/><td>PBERT-Base-MRPC</td><td>0.949</td><td>0.995</td><td>0.986</td><td>0.960</td><td>0.969</td><td>0.832</td><td>0.972</td></tr><tr><td/><td>RBERT-Base-MRPC</td><td>0.983</td><td>0.997</td><td>0.979</td><td>0.986</td><td>0.986</td><td>0.099</td><td>0.980</td></tr><tr><td/><td>FBERT-Base-MRPC PBERT-Base-MRPC (idf)</td><td>0.967 0.949</td><td>0.997 0.994</td><td>0.984 0.986</td><td>0.978 0.946</td><td>0.981 0.969</td><td>0.722 0.743</td><td>0.979 0.969</td></tr><tr><td/><td>RBERT-Base-MRPC (idf)</td><td>0.984</td><td>0.994</td><td>0.980</td><td>0.980</td><td>0.986</td><td>0.541</td><td>0.982</td></tr><tr><td/><td>FBERT-Base-MRPC (idf)</td><td>0.967</td><td>0.995</td><td>0.984</td><td>0.968</td><td>0.979</td><td>0.464</td><td>0.978</td></tr><tr><td/><td>PBERT-Large</td><td>0.969</td><td>0.991</td><td>0.985</td><td>0.979</td><td>0.970</td><td>0.915</td><td>0.969</td></tr><tr><td/><td>RBERT-Large</td><td>0.990</td><td>0.993</td><td>0.980</td><td>0.988</td><td>0.988</td><td>0.745</td><td>0.978</td></tr><tr><td/><td>FBERT-Large PBERT-Large (idf)</td><td>0.982 0.970</td><td>0.993 0.991</td><td>0.984 0.984</td><td>0.986 0.963</td><td>0.981 0.971</td><td>0.909 0.858</td><td>0.976 0.970</td></tr><tr><td/><td>RBERT-Large (idf)</td><td>0.989</td><td>0.990</td><td>0.982</td><td>0.982</td><td>0.985</td><td>0.047</td><td>0.982</td></tr><tr><td/><td>FBERT-Large (idf)</td><td>0.981</td><td>0.991</td><td>0.984</td><td>0.976</td><td>0.978</td><td>0.722</td><td>0.978</td></tr><tr><td/><td>PRoBERTa-Base</td><td>0.959</td><td>0.992</td><td>0.988</td><td>0.986</td><td>0.971</td><td>0.809</td><td>0.976</td></tr><tr><td/><td>RRoBERTa-Base</td><td>0.987</td><td>0.997</td><td>0.978</td><td>0.989</td><td>0.988</td><td>0.238</td><td>0.981</td></tr><tr><td/><td>FRoBERTa-Base</td><td>0.973</td><td>0.997</td><td>0.987</td><td>0.989</td><td>0.982</td><td>0.674</td><td>0.982</td></tr><tr><td/><td>PRoBERTa-Base (idf) RRoBERTa-Base (idf)</td><td>0.963 0.988</td><td>0.994 0.996</td><td>0.988 0.979</td><td>0.989 0.989</td><td>0.970 0.987</td><td>0.711 0.353</td><td>0.972 0.983</td></tr><tr><td/><td>FRoBERTa-Base (idf)</td><td>0.976</td><td>0.997</td><td>0.986</td><td>0.990</td><td>0.980</td><td>0.277</td><td>0.980</td></tr><tr><td>Pre-Trained</td><td>PRoBERTa-Large</td><td>0.965</td><td>0.995</td><td>0.990</td><td>0.976</td><td>0.976</td><td>0.846</td><td>0.975</td></tr><tr><td/><td>RRoBERTa-Large</td><td>0.989</td><td>0.997</td><td>0.982</td><td>0.989</td><td>0.988</td><td>0.540</td><td>0.981</td></tr><tr><td/><td>FRoBERTa-Large</td><td>0.978</td><td>0.998</td><td>0.989</td><td>0.983</td><td>0.985</td><td>0.760</td><td>0.981</td></tr><tr><td/><td>PRoBERTa-Large (idf) RRoBERTa-Large (idf)</td><td>0.972 0.990</td><td>0.997 0.996</td><td>0.988 0.983</td><td>0.986 0.989</td><td>0.976 0.989</td><td>0.686 0.096</td><td>0.973 0.982</td></tr><tr><td/><td>FRoBERTa-Large (idf)</td><td>0.982</td><td>0.998</td><td>0.988</td><td>0.989</td><td>0.983</td><td>0.453</td><td>0.980</td></tr><tr><td/><td>PRoBERTa-Large-MNLI</td><td>0.978</td><td>0.997</td><td>0.991</td><td>0.984</td><td>0.980</td><td>0.914</td><td>0.977</td></tr><tr><td/><td>RRoBERTa-Large-MNLI</td><td>0.991</td><td>0.996</td><td>0.984</td><td>0.989</td><td>0.987</td><td>0.566</td><td>0.982</td></tr><tr><td/><td>FRoBERTa-Large-MNLI</td><td>0.987</td><td>0.998</td><td>0.989</td><td>0.988</td><td>0.986</td><td>0.873</td><td>0.982</td></tr><tr><td/><td>PRoBERTa-Large-MNLI (idf)</td><td>0.982</td><td>0.998</td><td>0.992</td><td>0.990</td><td>0.978</td><td>0.822</td><td>0.974</td></tr><tr><td/><td>RRoBERTa-Large-MNLI (idf) FRoBERTa-Large-MNLI (idf)</td><td>0.992 0.989</td><td>0.996 0.998</td><td>0.985 0.990</td><td>0.988 0.990</td><td>0.988 0.985</td><td>0.022 0.583</td><td>0.983 0.980</td></tr><tr><td/><td>PXLNET-Base</td><td>0.960</td><td>0.997</td><td>0.984</td><td>0.982</td><td>0.972</td><td>0.849</td><td>0.974</td></tr><tr><td/><td>RXLNET-Base</td><td>0.985</td><td>0.997</td><td>0.975</td><td>0.988</td><td>0.988</td><td>0.303</td><td>0.980</td></tr><tr><td/><td>FXLNET-Base</td><td>0.974</td><td>0.998</td><td>0.981</td><td>0.986</td><td>0.982</td><td>0.628</td><td>0.980</td></tr><tr><td/><td>PXLNET-Base (idf)</td><td>0.962</td><td>0.995</td><td>0.983</td><td>0.982</td><td>0.972</td><td>0.657</td><td>0.974</td></tr><tr><td/><td>RXLNET-Base (idf) FXLNET-Base (idf)</td><td>0.986 0.975</td><td>0.996 0.996</td><td>0.976 0.980</td><td>0.987 0.985</td><td>0.987 0.981</td><td>0.666 0.259</td><td>0.982 0.980</td></tr><tr><td/><td>PXLNET-Large</td><td>0.955</td><td>0.995</td><td>0.983</td><td>0.986</td><td>0.972</td><td>0.875</td><td>0.970</td></tr><tr><td/><td>RXLNET-Large</td><td>0.984</td><td>0.996</td><td>0.972</td><td>0.984</td><td>0.989</td><td>0.491</td><td>0.975</td></tr><tr><td/><td>FXLNET-Large</td><td>0.971</td><td>0.996</td><td>0.980</td><td>0.987</td><td>0.984</td><td>0.821</td><td>0.976</td></tr><tr><td/><td>PXLNET-Large (idf)</td><td>0.961</td><td>0.997</td><td>0.983</td><td>0.987</td><td>0.973</td><td>0.816</td><td>0.973</td></tr><tr><td/><td>RXLNET-Large (idf) FXLNET-Large (idf)</td><td>0.987 0.976</td><td>0.996 0.997</td><td>0.975 0.980</td><td>0.989 0.989</td><td>0.988 0.982</td><td>0.320 0.623</td><td>0.981 0.980</td></tr><tr><td/><td>PXLM-En</td><td>0.953</td><td>0.995</td><td>0.988</td><td>0.979</td><td>0.974</td><td>0.918</td><td>0.972</td></tr><tr><td/><td>RXLM-En</td><td>0.983</td><td>0.996</td><td>0.980</td><td>0.988</td><td>0.991</td><td>0.561</td><td>0.977</td></tr><tr><td/><td>FXLM-En</td><td>0.969</td><td>0.997</td><td>0.986</td><td>0.986</td><td>0.985</td><td>0.869</td><td>0.977</td></tr><tr><td/><td>PXLM-En (idf)</td><td>0.957</td><td>0.996</td><td>0.987</td><td>0.970</td><td>0.974</td><td>0.862</td><td>0.973</td></tr><tr><td/><td>RXLM-En (idf)</td><td>0.982</td><td>0.995</td><td>0.981</td><td>0.988</td><td>0.989</td><td>0.213</td><td>0.980</td></tr><tr><td/><td>FXLM-En (idf)</td><td>0.970</td><td>0.996</td><td>0.985</td><td>0.982</td><td>0.982</td><td>0.519</td><td>0.978</td></tr></table>

Absolute Pearson correlations with human judgments on WMT18 to-English language pairs for 10K hybrid systems. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Table content: <table><tr><td>Setting</td><td>Metric</td><td>cs-en</td><td>de-en</td><td>et-en</td><td>fi-en</td><td>ru-en</td><td>tr-en</td><td>zh-en</td></tr><tr><td/><td>BLEU</td><td>0.338</td><td>0.894</td><td>0.866</td><td>0.666</td><td>0.447</td><td>0.265</td><td>0.799</td></tr><tr><td/><td>CDER</td><td>0.362</td><td>0.890</td><td>0.870</td><td>0.689</td><td>0.451</td><td>0.256</td><td>0.799</td></tr><tr><td/><td>CHARACTER</td><td>0.349</td><td>0.854</td><td>0.814</td><td>0.690</td><td>0.429</td><td>0.254</td><td>0.739</td></tr><tr><td/><td>ITER</td><td>0.356</td><td>0.901</td><td>0.856</td><td>0.676</td><td>0.454</td><td>0.278</td><td>0.811</td></tr><tr><td/><td>METEOR++</td><td>0.369</td><td>0.895</td><td>0.798</td><td>0.662</td><td>0.470</td><td>0.174</td><td>0.757</td></tr><tr><td/><td>NIST</td><td>0.338</td><td>0.894</td><td>0.857</td><td>0.672</td><td>0.446</td><td>0.323</td><td>0.803</td></tr><tr><td>Unsupervised</td><td>PER</td><td>0.325</td><td>0.866</td><td>0.771</td><td>0.663</td><td>0.435</td><td>0.021</td><td>0.754</td></tr><tr><td/><td>TER</td><td>0.342</td><td>0.885</td><td>0.873</td><td>0.673</td><td>0.447</td><td>0.063</td><td>0.792</td></tr><tr><td/><td>UHH TSKM</td><td>0.387</td><td>0.894</td><td>0.873</td><td>0.671</td><td>0.460</td><td>0.063</td><td>0.788</td></tr><tr><td/><td>WER</td><td>0.353</td><td>0.876</td><td>0.868</td><td>0.674</td><td>0.443</td><td>0.034</td><td>0.790</td></tr><tr><td/><td>YISI-0</td><td>0.344</td><td>0.881</td><td>0.834</td><td>0.681</td><td>0.452</td><td>0.275</td><td>0.776</td></tr><tr><td/><td>YISI-1</td><td>0.352</td><td>0.896</td><td>0.864</td><td>0.671</td><td>0.470</td><td>0.285</td><td>0.765</td></tr><tr><td/><td>YISI-1 SRL</td><td>0.351</td><td>0.901</td><td>0.871</td><td>0.682</td><td>0.464</td><td>0.086</td><td>0.770</td></tr><tr><td/><td>BEER</td><td>0.364</td><td>0.899</td><td>0.871</td><td>0.684</td><td>0.460</td><td>0.125</td><td>0.811</td></tr><tr><td>Supervised</td><td>BLEND</td><td>0.382</td><td>0.904</td><td>0.880</td><td>0.681</td><td>0.473</td><td>0.077</td><td>0.767</td></tr><tr><td/><td>RUSE</td><td>0.417</td><td>0.906</td><td>0.885</td><td>0.686</td><td>0.468</td><td>0.273</td><td>0.809</td></tr><tr><td/><td>PBERT-Base</td><td>0.386</td><td>0.901</td><td>0.880</td><td>0.674</td><td>0.481</td><td>0.318</td><td>0.793</td></tr><tr><td/><td>RBERT-Base</td><td>0.383</td><td>0.899</td><td>0.877</td><td>0.683</td><td>0.486</td><td>0.100</td><td>0.804</td></tr><tr><td/><td>FBERT-Base</td><td>0.388</td><td>0.903</td><td>0.879</td><td>0.678</td><td>0.484</td><td>0.331</td><td>0.808</td></tr><tr><td/><td>(idf)</td><td>0.390</td><td>0.902</td><td>0.877</td><td>0.681</td><td>0.475</td><td>0.318</td><td>0.786</td></tr><tr><td/><td>RBERT-Base (idf)</td><td>0.390</td><td>0.896</td><td>0.874</td><td>0.686</td><td>0.475</td><td>0.077</td><td>0.811</td></tr><tr><td/><td>FBERT-Base (idf)</td><td>0.393</td><td>0.902</td><td>0.876</td><td>0.685</td><td>0.483</td><td>0.225</td><td>0.806</td></tr><tr><td/><td>PBERT-Base-MRPC</td><td>0.392</td><td>0.832</td><td>0.872</td><td>0.686</td><td>0.475</td><td>0.319</td><td>0.791</td></tr><tr><td/><td>RBERT-Base-MRPC</td><td>0.397</td><td>0.908</td><td>0.870</td><td>0.691</td><td>0.478</td><td>0.025</td><td>0.811</td></tr><tr><td/><td>FBERT-Base-MRPC</td><td>0.398</td><td>0.907</td><td>0.872</td><td>0.690</td><td>0.481</td><td>0.335</td><td>0.806</td></tr><tr><td/><td>PBERT-Base-MRPC (idf)</td><td>0.392</td><td>0.896</td><td>0.870</td><td>0.689</td><td>0.467</td><td>0.316</td><td>0.797</td></tr><tr><td/><td>RBERT-Base-MRPC (idf)</td><td>0.400</td><td>0.906</td><td>0.867</td><td>0.691</td><td>0.479</td><td>0.018</td><td>0.817</td></tr><tr><td/><td>FBERT-Base-MRPC (idf)</td><td>0.400</td><td>0.905</td><td>0.869</td><td>0.693</td><td>0.475</td><td>0.097</td><td>0.812</td></tr><tr><td/><td>PBERT-Large</td><td>0.398</td><td>0.901</td><td>0.880</td><td>0.678</td><td>0.481</td><td>0.327</td><td>0.799</td></tr><tr><td/><td>RBERT-Large</td><td>0.391</td><td>0.897</td><td>0.879</td><td>0.690</td><td>0.490</td><td>0.085</td><td>0.810</td></tr><tr><td/><td>FBERT-Large</td><td>0.397</td><td>0.898</td><td>0.882</td><td>0.684</td><td>0.486</td><td>0.328</td><td>0.810</td></tr><tr><td/><td>PBERT-Large (idf)</td><td>0.398</td><td>0.900</td><td>0.875</td><td>0.685</td><td>0.475</td><td>0.323</td><td>0.794</td></tr><tr><td/><td>RBERT-Large (idf)</td><td>0.395</td><td>0.895</td><td>0.873</td><td>0.692</td><td>0.488</td><td>0.080</td><td>0.813</td></tr><tr><td/><td>FBERT-Large (idf)</td><td>0.398</td><td>0.899</td><td>0.875</td><td>0.691</td><td>0.482</td><td>0.086</td><td>0.810</td></tr><tr><td/><td>PRoBERTa-Base RRoBERTa-Base</td><td>0.372 0.366</td><td>0.814 0.902</td><td>0.866 0.870</td><td>0.697 0.683</td><td>0.475 0.483</td><td>0.313 0.026</td><td>0.795 0.813</td></tr><tr><td/><td>FRoBERTa-Base</td><td>0.374</td><td>0.904</td><td>0.870</td><td>0.694</td><td>0.480</td><td>0.224</td><td>0.808</td></tr><tr><td/><td>PRoBERTa-Base (idf)</td><td>0.373</td><td>0.825</td><td>0.865</td><td>0.697</td><td>0.470</td><td>0.303</td><td>0.802</td></tr><tr><td/><td>RRoBERTa-Base (idf)</td><td>0.374</td><td>0.898</td><td>0.866</td><td>0.688</td><td>0.486</td><td>0.028</td><td>0.816</td></tr><tr><td/><td>FRoBERTa-Base (idf)</td><td>0.380</td><td>0.904</td><td>0.866</td><td>0.696</td><td>0.479</td><td>0.037</td><td>0.812</td></tr><tr><td>Pre-Trained</td><td>PRoBERTa-Large RRoBERTa-Large</td><td>0.375 0.366</td><td>0.833 0.895</td><td>0.871 0.874</td><td>0.693 0.689</td><td>0.474 0.480</td><td>0.327 0.039</td><td>0.800 0.816</td></tr><tr><td/><td>FRoBERTa-Large</td><td>0.378</td><td>0.907</td><td>0.874</td><td>0.694</td><td>0.480</td><td>0.324</td><td>0.811</td></tr><tr><td/><td>PRoBERTa-Large (idf)</td><td>0.384</td><td>0.905</td><td>0.866</td><td>0.694</td><td>0.475</td><td>0.220</td><td>0.806</td></tr><tr><td/><td>RRoBERTa-Large (idf)</td><td>0.368</td><td>0.885</td><td>0.869</td><td>0.692</td><td>0.487</td><td>0.030</td><td>0.819</td></tr><tr><td/><td>FRoBERTa-Large (idf)</td><td>0.382</td><td>0.907</td><td>0.868</td><td>0.696</td><td>0.484</td><td>0.048</td><td>0.815</td></tr><tr><td/><td>PRoBERTa-Large-MNLI RRoBERTa-Large-MNLI</td><td>0.383 0.378</td><td>0.909 0.880</td><td>0.880 0.877</td><td>0.698 0.692</td><td>0.480 0.481</td><td>0.323 0.078</td><td>0.795 0.811</td></tr><tr><td/><td>FRoBERTa-Large-MNLI</td><td>0.385</td><td>0.909</td><td>0.879</td><td>0.697</td><td>0.484</td><td>0.286</td><td>0.809</td></tr><tr><td/><td>PRoBERTa-Large-MNLI (idf)</td><td>0.389</td><td>0.905</td><td>0.874</td><td>0.698</td><td>0.478</td><td>0.268</td><td>0.803</td></tr><tr><td/><td>RRoBERTa-Large-MNLI (idf)</td><td>0.380</td><td>0.874</td><td>0.870</td><td>0.691</td><td>0.483</td><td>0.079</td><td>0.814</td></tr><tr><td/><td>FRoBERTa-Large-MNLI (idf)</td><td>0.387</td><td>0.906</td><td>0.872</td><td>0.696</td><td>0.482</td><td>0.082</td><td>0.811</td></tr><tr><td/><td>PXLNET-Base</td><td>0.385</td><td>0.875</td><td>0.869</td><td>0.692</td><td>0.469</td><td>0.342</td><td>0.796</td></tr><tr><td/><td>RXLNET-Base FXLNET-Base</td><td>0.381 0.385</td><td>0.907 0.907</td><td>0.869 0.871</td><td>0.693 0.694</td><td>0.477 0.476</td><td>0.026 0.128</td><td>0.809 0.810</td></tr><tr><td/><td>PXLNET-Base (idf)</td><td>0.381</td><td>0.904</td><td>0.864</td><td>0.699</td><td>0.464</td><td>0.289</td><td>0.794</td></tr><tr><td/><td>RXLNET-Base (idf)</td><td>0.384</td><td>0.903</td><td>0.863</td><td>0.696</td><td>0.479</td><td>0.013</td><td>0.812</td></tr><tr><td/><td>FXLNET-Base (idf)</td><td>0.384</td><td>0.905</td><td>0.864</td><td>0.699</td><td>0.472</td><td>0.032</td><td>0.809</td></tr><tr><td/><td>PXLNET-Large</td><td>0.392</td><td>0.844</td><td>0.873</td><td>0.689</td><td>0.367</td><td>0.338</td><td>0.799</td></tr><tr><td/><td>RXLNET-Large FXLNET-Large</td><td>0.389 0.393</td><td>0.905 0.907</td><td>0.871 0.876</td><td>0.690 0.691</td><td>0.482 0.483</td><td>0.031 0.348</td><td>0.800 0.812</td></tr><tr><td/><td>PXLNET-Large (idf)</td><td>0.393</td><td>0.899</td><td>0.870</td><td>0.694</td><td>0.387</td><td>0.333</td><td>0.794</td></tr><tr><td/><td>RXLNET-Large (idf)</td><td>0.395</td><td>0.901</td><td>0.868</td><td>0.690</td><td>0.483</td><td>0.023</td><td>0.810</td></tr><tr><td/><td>FXLNET-Large (idf)</td><td>0.396</td><td>0.906</td><td>0.870</td><td>0.693</td><td>0.478</td><td>0.128</td><td>0.811</td></tr><tr><td/><td>PXLM-En</td><td>0.394</td><td>0.891</td><td>0.880</td><td>0.685</td><td>0.476</td><td>0.322</td><td>0.802</td></tr><tr><td/><td>RXLM-En</td><td>0.401</td><td>0.903</td><td>0.875</td><td>0.692</td><td>0.483</td><td>0.082</td><td>0.803</td></tr><tr><td/><td>FXLM-En PXLM-En (idf)</td><td>0.400 0.391</td><td>0.909 0.903</td><td>0.878 0.874</td><td>0.689 0.684</td><td>0.483 0.480</td><td>0.234 0.293</td><td>0.811 0.797</td></tr><tr><td/><td>RXLM-En (idf)</td><td>0.402</td><td>0.900</td><td>0.868</td><td>0.688</td><td>0.477</td><td>0.068</td><td>0.806</td></tr><tr><td/><td>FXLM-En (idf)</td><td>0.398</td><td>0.905</td><td>0.871</td><td>0.688</td><td>0.487</td><td>0.079</td><td>0.809</td></tr></table>

Pearson correlation on the 2015 COCO Captioning Challenge. The M1 and M2 measures are described in Section 4. We bold the best correlating task-specific and task-agnostic metrics in each setting LEIC uses images as additional inputs. Numbers with * are cited fromCui et al. (2018).

Table content: <table><tr><td>Metric</td><td>M1</td><td>M2</td></tr><tr><td>BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CIDER SPICE LEIC</td><td>0.124  *  0.037  *  0.004  *  -0.019  *  0.606  *  0.090  *  0.438  *  0.759  *  0.939  *</td><td>0.135  *  0.048  *  0.016  *  -0.005  *  0.594  *  0.096  *  0.440  *  0.750  *  0.949  *</td></tr><tr><td>BEER</td><td>0.491</td><td>0.562</td></tr><tr><td>EED</td><td>0.545</td><td>0.599</td></tr><tr><td>CHRF++</td><td>0.702</td><td>0.729</td></tr><tr><td>CHARACTER</td><td>0.800</td><td>0.801</td></tr><tr><td>PBERT-Base</td><td>0.313</td><td>0.344</td></tr><tr><td>RBERT-Base</td><td>0.679</td><td>0.622</td></tr><tr><td>FBERT-Base</td><td>0.531</td><td>0.519</td></tr><tr><td>PBERT-Base (idf)</td><td>0.243</td><td>0.286</td></tr><tr><td>RBERT-Base (idf)</td><td>0.834</td><td>0.783</td></tr><tr><td>FBERT-Base (idf)</td><td>0.579</td><td>0.581</td></tr><tr><td>PBERT-Base-MRPC RBERT-Base-MRPC</td><td>0.252 0.644</td><td>0.331 0.641</td></tr><tr><td>FBERT-Base-MRPC</td><td>0.470</td><td>0.512</td></tr><tr><td>PBERT-Base-MRPC (idf)</td><td>0.264</td><td>0.300</td></tr><tr><td>RBERT-Base-MRPC (idf)</td><td>0.794</td><td>0.767</td></tr><tr><td>FBERT-Base-MRPC (idf)</td><td>0.575</td><td>0.583</td></tr><tr><td>PBERT-Large</td><td>0.454</td><td>0.486</td></tr><tr><td>RBERT-Large</td><td>0.756</td><td>0.697</td></tr><tr><td>FBERT-Large</td><td>0.649</td><td>0.634</td></tr><tr><td>PBERT-Large (idf) RBERT-Large (idf)</td><td>0.327 0.873</td><td>0.372 0.821</td></tr><tr><td>FBERT-Large (idf)</td><td>0.645</td><td>0.647</td></tr><tr><td>PRoBERTa-Base</td><td>-0.223</td><td>-0.179</td></tr><tr><td>RRoBERTa-Base</td><td>0.827</td><td>0.800</td></tr><tr><td>FRoBERTa-Base</td><td>0.176</td><td>0.191</td></tr><tr><td>PRoBERTa-Base (idf)</td><td>-0.256</td><td>-0.267</td></tr><tr><td>RRoBERTa-Base (idf)</td><td>0.901</td><td>0.869</td></tr><tr><td>FRoBERTa-Base (idf)</td><td>0.188</td><td>0.157</td></tr><tr><td>PRoBERTa-Large RRoBERTa-Large</td><td>-0.105 0.888</td><td>-0.041 0.863</td></tr><tr><td>FRoBERTa-Large</td><td>0.322</td><td>0.350</td></tr><tr><td>PRoBERTa-Large (idf)</td><td>0.063</td><td>-0.011</td></tr><tr><td>RRoBERTa-Large (idf)</td><td>0.917</td><td>0.889</td></tr><tr><td>FRoBERTa-Large (idf)</td><td>0.519</td><td>0.453</td></tr><tr><td>PRoBERTa-Large-MNLI</td><td>0.129</td><td>0.208</td></tr><tr><td>RRoBERTa-Large-MNLI</td><td>0.820</td><td>0.823</td></tr><tr><td>FRoBERTa-Large-MNLI PRoBERTa-Large-MNLI (idf)</td><td>0.546 0.081</td><td>0.592 0.099</td></tr><tr><td>RRoBERTa-Large-MNLI (idf)</td><td>0.906</td><td>0.875</td></tr><tr><td>FRoBERTa-Large-MNLI (idf)</td><td>0.605</td><td>0.596</td></tr><tr><td>PXLNet-Base</td><td>-0.046</td><td>0.080</td></tr><tr><td>RXLNet-Base</td><td>0.409</td><td>0.506</td></tr><tr><td>FXLNet-Base</td><td>0.146</td><td>0.265</td></tr><tr><td>PXLNet-Base (idf)</td><td>0.006</td><td>0.145</td></tr><tr><td>RXLNet-Base (idf) FXLNet-Base (idf)</td><td>0.655 0.270</td><td>0.720 0.391</td></tr><tr><td>PXLNet-Large</td><td>-0.188</td><td>-0.115</td></tr><tr><td>RXLNet-Large</td><td>0.178</td><td>0.195</td></tr><tr><td>FXLNet-Large</td><td>-0.014</td><td>0.036</td></tr><tr><td>PXLNet-Large (idf)</td><td>-0.186</td><td>-0.072</td></tr><tr><td>RXLNet-Large (idf)</td><td>0.554</td><td>0.555</td></tr><tr><td>FXLNet-Large (idf)</td><td>0.151</td><td>0.234</td></tr><tr><td>PXLM-En</td><td>0.230</td><td>0.220</td></tr><tr><td>RXLM-En</td><td>0.333</td><td>0.263</td></tr><tr><td>FXLM-En PXLM-En (idf)</td><td>0.297 0.266</td><td>0.243 0.275</td></tr><tr><td>RXLM-En (idf)</td><td>0.700</td><td>0.640</td></tr><tr><td>FXLM-En (idf)</td><td>0.499</td><td>0.470</td></tr></table>

Peer review (iclr2020):
Paper Contributions This paper introduces a new text generation scoring approach using BERT, called BERTScore. Using BERT embeddings and optionally idf scores, a greedy matching is performed between all reference and candidate words, with cosine similarity between vector representations as the scoring. From this, a precision, recall and F1 score can be derived. This notably outperforms BLEU, as well as other metrics, most but not all of the time. The paper offers a broad range of comparisons and analysis. Decision I*m leaning towards accepting the paper on the basis of the following. Strong points taken in consideration: - Simple, well-motivated metric that uses powerful BERT-style models, without being slow to compute either. - Good performance empirically on WMT. I*m less convinced on COCO since using the image is fair game there. - Code is provided, and it is simple and adaptable for future work. - Experimentation is detailed and reproducible. Weaker points taken in consideration: - Work conducted in parallel matches or exceeds the performance of BERTScore. This shouldn*t necessarily be a reason to choose not to publish this work in my opinion, but it should be taken into consideration. I like that the authors were open and clear regarding this in their discussion. - The authors haven*t come up with a recommendation for a single configuration of their approach. In one place they recommend F-BERT without idf, in another they argue for picking and choosing based on context, with little help about how to choose. I think practitioners are only going to be willing to switch away from BLEU, for example, if a single one-size-fits-all metric is proposed instead. I identify this ambiguity between BERTScore versions as an important weakness of the paper. - It*s unclear throughout whether words or wordpieces are the main token being considered. Most discussion and definitions use *words*, but in section 3, subsection Token Representation, it appears to be clearly stated that BERTScore uses a BERT model based on word pieces. I recommend adjust the language to be more consistent throughout. Also, scoring examples with word pieces would be more consistent with this as well, imo. Notably, I*m actually unsure whether you compute IDF over words or word pieces, and how this is applied. - Finally, I found some weaknesses in the Importance Weighting section (though this isn*t too important since IDF isn*t part of the recommended BERTScore I believe). The IDF scores would be stronger if they were computed on a bigger in-domain corpus than the gold test set. This would add extra steps to using BERTScore though and make things more complicated in practice, but this should nevertheless probably be tried, or at least discussed in the paper. Also, the plus-one smoothing handles unknown words (or word piece?) and I*m not sure why. If we*re using the test set to compute IDF, and the sentences we*re looking *are* in the test set, then there shouldn*t be unknown words and no smoothing is required. So overall, I still think this deserves publication because it*s valuable information for researchers, and the metric itself could be immediately useful to some as well. However, the weaknesses mentioned make me hesitate to fully endorse the work.