SlowGuess/ACL_2020_OCR huggingface dataset:
Thibault Sellam Dipanjan Das Ankur P. Parikh
{tsellam, dipanjand, aparikh}@google.com
Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.
In the last few years, research in natural text generation (NLG) has made significant progress, driven largely by the neural encoder-decoder paradigm (Sutskever et al., 2014; Bahdanau et al., 2015) which can tackle a wide array of tasks including translation (Koehn, 2009), summarization (Mani, 1999; Chopra et al., 2016), structured-data-to-text generation (McKeown, 1992; Kukich, 1983; Wiseman et al., 2017) dialog (Smith and Hipp, 1994; Vinyals and Le, 2015) and image captioning (Fang et al., 2015). However, progress is increasingly impeded by the shortcomings of existing metrics (Wiseman et al., 2017; Ma et al., 2019; Tian et al., 2019).
Human evaluation is often the best indicator of the quality of a system. However, designing crowd sourcing experiments is an expensive and high-latency process, which does not easily fit in a daily model development pipeline. Therefore, NLG researchers commonly use automatic
evaluation metrics, which provide an acceptable proxy for quality and are very cheap to compute. This paper investigates sentence-level, reference-based metrics, which describe the extent to which a candidate sentence is similar to a reference one. The exact definition of similarity may range from string overlap to logical entailment.
The first generation of metrics relied on handcrafted rules that measure the surface similarity between the sentences. To illustrate, BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), two popular metrics, rely on N-gram overlap. Because those metrics are only sensitive to lexical variation, they cannot appropriately reward semantic or syntactic variations of a given reference. Thus, they have been repeatedly shown to correlate poorly with human judgment, in particular when all the systems to compare have a similar level of accuracy (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018).
Increasingly, NLG researchers have addressed those problems by injecting learned components in their metrics. To illustrate, consider the WMT Metrics Shared Task, an annual benchmark in which translation metrics are compared on their ability to imitate human assessments. The last two years of the competition were largely dominated by neural net-based approaches, RUSE, YiSi and ESIM (Ma et al., 2018, 2019). Current approaches largely fall into two categories. Fully learned metrics, such as BEER, RUSE, and ESIM are trained end-to-end, and they typically rely on handcrafted features and/or learned embeddings. Conversely, hybrid metrics, such as YiSi and BERTscore combine trained elements, e.g., contextual embeddings, with handwritten logic, e.g., as token alignment rules. The first category typically offers great expressivity: if a training set of human ratings data is available, the metrics may take full advantage of it and fit the ratings distribution tightly. Fur
7881
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
thermore, learned metrics can be tuned to measure task-specific properties, such as fluency, faithfulness, grammar, or style. On the other hand, hybrid metrics offer robustness. They may provide better results when there is little to no training data, and they do not rely on the assumption that training and test data are identically distributed.
And indeed, the IID assumption is particularly problematic in NLG evaluation because of domain drifts, that have been the main target of the metrics literature, but also because of quality drifts: NLG systems tend to get better over time, and therefore a model trained on ratings data from 2015 may fail to distinguish top performing systems in 2019, especially for newer research tasks. An ideal learned metric would be able to both take full advantage of available ratings data for training, and be robust to distribution drifts, i.e., it should be able to extrapolate.
Our insight is that it is possible to combine expressivity and robustness by pre-training a fully learned metric on large amounts of synthetic data, before fine-tuning it on human ratings. To this end, we introduce BLEURT, $^{1}$  a text generation metric based on BERT (Devlin et al., 2019). A key ingredient of BLEURT is a novel pre-training scheme, which uses random perturbations of Wikipedia sentences augmented with a diverse set of lexical and semantic-level supervision signals.
To demonstrate our approach, we train BLEURT for English and evaluate it under different generalization regimes. We first verify that it provides state-of-the-art results on all recent years of the WMT Metrics Shared task (2017 to 2019, to-English language pairs). We then stress-test its ability to cope with quality drifts with a synthetic benchmark based on WMT 2017. Finally, we show that it can easily adapt to a different domain with three tasks from a data-to-text dataset, WebNLG 2017 (Gardent et al., 2017). Ablations show that our synthetic pretraining scheme increases performance in the IID setting, and is critical to ensure robustness when the training data is scarce, skewed, or out-of-domain.
The code and pre-trained models are available online<sup>2</sup>.
Define  $\boldsymbol{x} = (x_{1},\dots,x_{r})$  to be the reference sentence of length  $r$  where each  $x_{i}$  is a token and let  $\tilde{\boldsymbol{x}} = (\tilde{x}_1,\dots,\tilde{x}_p)$  be a prediction sentence of length  $p$ . Let  $\{(x_i,\tilde{x}_i,y_i)\}_{n = 1}^N$  be a training dataset of size  $N$  where  $y_{i}\in \mathbb{R}$  is the human rating that indicates how good  $\tilde{\boldsymbol{x}}_i$  is with respect to  $\boldsymbol{x}_i$ . Given the training data, our goal is to learn a function  $f:(\boldsymbol{x},\tilde{\boldsymbol{x}})\to y$  that predicts the human rating.
3 Fine-Tuning BERT for Quality Evaluation
Given the small amounts of rating data available, it is natural to leverage unsupervised representations for this task. In our model, we use BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019), which is an unsupervised technique that learns contextualized representations of sequences of text. Given  $\mathbf{x}$  and  $\tilde{\mathbf{x}}$ , BERT is a Transformer (Vaswani et al., 2017) that returns a sequence of contextualized vectors:
$$
\boldsymbol {v} _ {\text {[ C L S ]}}, \boldsymbol {v} _ {x _ {1}}, \dots , \boldsymbol {v} _ {x _ {r}}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {\tilde {x} _ {p}} = \operatorname {B E R T} (\boldsymbol {x}, \tilde {\boldsymbol {x}})
$$
where  $\pmb{v}_{[\mathrm{CLS}]}$  is the representation for the special [CLS] token. As described by Devlin et al. (2019), we add a linear layer on top of the [CLS] vector to predict the rating:
$$
\hat {y} = \boldsymbol {f} (\boldsymbol {x}, \tilde {\boldsymbol {x}}) = \boldsymbol {W} \tilde {\boldsymbol {v}} _ {[ \mathrm {C L S} ]} + \boldsymbol {b}
$$
where  $W$  and  $\pmb{b}$  are the weight matrix and bias vector respectively. Both the above linear layer as well as the BERT parameters are trained (i.e. fine-tuned) on the supervised data which typically numbers in a few thousand examples. We use the regression loss  $\ell_{\mathrm{supervised}} = \frac{1}{N}\sum_{n = 1}^{N}\| y_i - \hat{y}\|^2.$
Although this approach is quite straightforward, we will show in Section 5 that it gives state-of-the-art results on WMT Metrics Shared Task 17-19, which makes it a high-performing evaluation metric. However, fine-tuning BERT requires a sizable amount of IID data, which is less than ideal for a metric that should generalize to a variety of tasks and model drift.
4 Pre-Training on Synthetic Data
The key aspect of our approach is a pre-training technique that we use to "warm up" BERT before fine-tuning on rating data. We generate a large
3To clarify, our pre-training scheme is an addition, not a replacement to BERT's initial training (Devlin et al., 2019) and happens after it.
$^{1}$ Bilingual Evaluation Understudy with Representations from Transformers. We refer the intrigued reader to Papineni et al. 2002 for a justification of the term understudy.  $^{2}$ http://github.com/google-research/bleurt
7882
number of synthetic reference-candidate pairs  $(z, \tilde{z})$ , and we train BERT on several lexical- and semantic-level supervision signals with a multitask loss. As our experiments will show, BLEURT generalizes much better after this phase, especially with incomplete training data.
Any pre-training approach requires a dataset and a set of pre-training tasks. Ideally, the setup should resemble the final NLG evaluation task, i.e., the sentence pairs should be distributed similarly and the pre-training signals should correlate with human ratings. Unfortunately, we cannot have access to the NLG models that we will evaluate in the future. Therefore, we optimized our scheme for generality, with three requirements. (1) The set of reference sentences should be large and diverse, so that BLEURT can cope with a wide range of NLG domains and tasks. (2) The sentence pairs should contain a wide variety of lexical, syntactic, and semantic dissimilarities. The aim here is to anticipate all variations that an NLG system may produce, e.g., phrase substitution, paraphrases, noise, or omissions. (3) The pre-training objectives should effectively capture those phenomena, so that BLEURT can learn to identify them. The following sections present our approach.
4.1 Generating Sentence Pairs
One way to expose BLEURT to a wide variety of sentence differences is to use existing sentence pairs datasets (Bowman et al., 2015; Williams et al., 2018; Wang et al., 2019). These sets are a rich source of related sentences, but they may fail to capture the errors and alterations that NLG systems produce (e.g., omissions, repetitions, nonsensical substitutions). We opted for an automatic approach instead, that can be scaled arbitrarily and at little cost: we generate synthetic sentence pairs  $(z,\tilde{z})$  by randomly perturbing 1.8 million segments  $z$  from Wikipedia. We use three techniques: mask-filling with BERT, backtranslation, and randomly dropping out words. We obtain about 6.5 million perturbations  $\tilde{z}$ . Let us describe those techniques.
Mask-filling with BERT: BERT's initial training task is to fill gaps (i.e., masked tokens) in tokenized sentences. We leverage this functionality by inserting masks at random positions in the Wikipedia sentences, and fill them with the language model. Thus, we introduce lexical alter
ations while maintaining the fluency of the sentence. We use two masking strategies—we either introduce the masks at random positions in the sentences, or we create contiguous sequences of masked tokens. More details are provided in the Appendix.
Backtranslation: We generate paraphrases and perturbations with backtranslation, that is, round trips from English to another language and then back to English with a translation model (Bannard and Callison-Burch, 2005; Ganitkevitch et al., 2013; Sennrich et al., 2016). Our primary aim is to create variants of the reference sentence that preserves semantics. Additionally, we use the mispredictions of the backtranslation models as a source of realistic alterations.
Dropping words: We found it useful in our experiments to randomly drop words from the synthetic examples above to create other examples. This method prepares BLEURT for "pathological" behaviors or NLG systems, e.g., void predictions, or sentence truncation.
4.2 Pre-Training Signals
The next step is to augment each sentence pair  $(z, \tilde{z})$  with a set of pre-training signals  $\{\tau_k\}$ , where  $\tau_k$  is the target vector of pre-training task  $k$ . Good pre-training signals should capture a wide variety of lexical and semantic differences. They should also be cheap to obtain, so that the approach can scale to large amounts of synthetic data. The following section presents our 9 pretraining tasks, summarized in Table 1. Additional implementation details are in the Appendix.
Automatic Metrics: We create three signals  $\tau_{\mathrm{BLEU}}$ ,  $\tau_{\mathrm{ROUGE}}$ , and  $\tau_{\mathrm{BERTscore}}$  with sentence BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTscore (Zhang et al., 2020) respectively (we use precision, recall and F-score for the latter two).
Backtranslation Likelihood: The idea behind this signal is to leverage existing translation models to measure semantic equivalence. Given a pair  $(z, \tilde{z})$ , this training signal measures the probability that  $\tilde{z}$  is a backtranslation of  $z$ ,  $P(\tilde{z} | z)$ , normalized by the length of  $\tilde{z}$ . Let  $P_{\mathrm{en} \rightarrow \mathrm{fr}}(z_{\mathrm{fr}} | z)$  be a translation model that assigns probabilities to French sentences  $z_{\mathrm{fr}}$  conditioned on English sentences  $z$  and let  $P_{\mathrm{fr} \rightarrow \mathrm{en}}(z | z_{\mathrm{fr}})$  be a translation model that assigns probabilities to English
7883
Task Type	Pre-training Signals	Loss Type
BLEU	τBLEU	Regression
ROUGE	τROUGE-P, τROUGE-R, τROUGE-F	Regression
BERTscore	τBERTscore = (τBERTscore-P, τBERTscore-R, τBERTscore-F)	Regression
Backtrans. likelihood	τen-fr, z|z̅, τen-fr, ŷz | z, τen-de, z| ŷz, τen-de, ŷz | z	Regression
Entailment	τentail = (τEntail, τContradict, τNeutral)	Multiclass
Backtrans. flag	τbacktran_flag	Multiclass
Table 1: Our pre-training signals.
sentences given french sentences. If  $|\tilde{z}|$  is the number of tokens in  $\tilde{z}$ , we define our score as  $\tau_{\mathrm{en - fr},\tilde{z}|z} = \frac{\log P(\tilde{z}|z)}{|\tilde{z}|}$ , with:
$$
P (\tilde {\boldsymbol {z}} | \boldsymbol {z}) = \sum_ {\boldsymbol {z} _ {\mathrm {f r}}} P _ {\mathrm {f r} \to \mathrm {e n}} (\tilde {\boldsymbol {z}} | \boldsymbol {z} _ {\mathrm {f r}}) P _ {\mathrm {e n} \to \mathrm {f r}} (\boldsymbol {z} _ {\mathrm {f r}} | \boldsymbol {z})
$$
Because computing the summation over all possible French sentences is intractable, we approximate the sum using  $z_{\mathrm{fr}}^{*} = \arg \max P_{\mathrm{en}\to \mathrm{fr}}(z_{\mathrm{fr}}|z)$  and we assume that  $P_{\mathrm{en}\to \mathrm{fr}}(z_{\mathrm{fr}}^{*}|z)\approx 1$ :
$$
P (\tilde {\boldsymbol {z}} | \boldsymbol {z}) \approx P _ {\mathrm {f r} \to \mathrm {e n}} (\tilde {\boldsymbol {z}} | \boldsymbol {z} _ {\mathrm {f r}} ^ {*})
$$
We can trivially reverse the procedure to compute  $P(z|\tilde{z})$ , thus we create 4 pre-training signals  $\tau_{\mathrm{en - fr},z|\tilde{z}}$ ,  $\tau_{\mathrm{en - fr},\tilde{z}|z}$ ,  $\tau_{\mathrm{en - de},z|\tilde{z}}$ ,  $\tau_{\mathrm{en - de},\tilde{z}|z}$  with two pairs of languages (en ↔ de and en ↔ fr) in both directions.
Textual Entailment: The signal  $\pmb{\tau}_{\mathrm{entail}}$  expresses whether  $z$  entails or contradicts  $\tilde{z}$  using a classifier. We report the probability of three labels: Entail, Contradict, and Neutral, using BERT finetuned on an entailment dataset, MNLI (Devlin et al., 2019; Williams et al., 2018).
Backtranslation flag: The signal  $\tau_{\mathrm{backtran\_flag}}$  is a Boolean that indicates whether the perturbation was generated with backtranslation or with mask-filling.
4.3 Modeling
For each pre-training task, our model uses either a regression or a classification loss. We then aggregate the task-level losses with a weighted sum.
Let  $\pmb{\tau}_{k}$  describe the target vector for each task, e.g., the probabilities for the classes Entail, Contradict, Neutral, or the precision, recall, and F-score for ROUGE. If  $\pmb{\tau}_{k}$  is a regression task, then the loss used is the  $\ell_{2}$  loss i.e.  $\ell_{k} = \| \pmb{\tau}_{k} - \hat{\pmb{\tau}}_{k}\|_{2}^{2} / |\pmb{\tau}_{k}|$  where  $|\pmb{\tau}_{k}|$  is the dimension of  $\pmb{\tau}_{k}$  and  $\hat{\pmb{\tau}}_k$  is computed by using a task-specific linear layer on top of the [CLS] embedding:  $\hat{\pmb{\tau}}_k =$
$\pmb{W}_{\tau_k}\tilde{\pmb{v}}_{[\mathrm{CLS}]} + \pmb{b}_{\tau_k}$ . If  $\pmb{\tau}_k$  is a classification task, we use a separate linear layer to predict a logit for each class  $c$ :  $\hat{\tau}_{kc} = \pmb{W}_{\tau_{kc}}\tilde{\pmb{v}}_{[\mathrm{CLS}]} + \pmb{b}_{\tau_{kc}}$ , and we use the multiclass cross-entropy loss. We define our aggregate pre-training loss function as follows:
$$
\ell_ {\text {p r e - t r a i n i n g}} = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {k = 1} ^ {K} \gamma_ {k} \ell_ {k} \left(\boldsymbol {\tau} _ {k} ^ {m}, \hat {\boldsymbol {\tau}} _ {k} ^ {m}\right) \tag {1}
$$
where  $\tau_{k}^{m}$  is the target vector for example  $m$ ,  $M$  is number of synthetic examples, and  $\gamma_{k}$  are hyperparameter weights obtained with grid search (more details in the Appendix).
In this section, we report our experimental results for two tasks, translation and data-to-text. First, we benchmark BLEURT against existing text generation metrics on the last 3 years of the WMT Metrics Shared Task (Bojar et al., 2017). We then evaluate its robustness to quality drifts with a series of synthetic datasets based on WMT17. We test BLEURT's ability to adapt to different tasks with the WebNLG 2017 Challenge Dataset (Gardent et al., 2017). Finally, we measure the contribution of each pre-training task with ablation experiments.
Our Models: Unless specified otherwise, all BLEURT models are trained in three steps: regular BERT pre-training (Devlin et al., 2019), pre-training on synthetic data (as explained in Section 4), and fine-tuning on task-specific ratings (translation and/or data-to-text). We experiment with two versions of BLEURT, BLEURT and BLEURTbase, respectively based on BERT-Large (24 layers, 1024 hidden units, 16 heads) and BERT-Base (12 layers, 768 hidden units, 12 heads) (Devlin et al., 2019), both uncased. We use batch size 32, learning rate 1e-5, and 800,000 steps for pre-training and 40,000 steps for fine-tuning. We provide the full detail of our training setup in the Appendix.
7884
model	cs-en
τ / r	de-en
τ / r	fi-en
τ / r	lv-en
τ / r	ru-en
τ / r	tr-en
τ / r	zh-en
τ / r	avg
τ / r
sentBLEU	29.6 / 43.2	28.9 / 42.2	38.6 / 56.0	23.9 / 38.2	34.3 / 47.7	34.3 / 54.0	37.4 / 51.3	32.4 / 47.5
MoverScore	47.6 / 67.0	51.2 / 70.8	NA	NA	53.4 / 73.8	56.1 / 76.2	53.1 / 74.4	52.3 / 72.4
BERTscore w/ BERT	48.0 / 66.6	50.3 / 70.1	61.4 / 81.4	51.6 / 72.3	53.7 / 73.0	55.6 / 76.0	52.2 / 73.1	53.3 / 73.2
BERTscore w/ roBERTa	54.2 / 72.6	56.9 / 76.0	64.8 / 83.2	56.2 / 75.7	57.2 / 75.2	57.9 / 76.1	58.8 / 78.9	58.0 / 76.8
chrF++	35.0 / 52.3	36.5 / 53.4	47.5 / 67.8	33.3 / 52.0	41.5 / 58.8	43.2 / 61.4	40.5 / 59.3	39.6 / 57.9
BEER	34.0 / 51.1	36.1 / 53.0	48.3 / 68.1	32.8 / 51.5	40.2 / 57.7	42.8 / 60.0	39.5 / 58.2	39.1 / 57.1
BLEURTbase -pre	51.5 / 68.2	52.0 / 70.7	66.6 / 85.1	60.8 / 80.5	57.5 / 77.7	56.9 / 76.0	52.1 / 72.1	56.8 / 75.8
BLEURTbase	55.7 / 73.4	56.3 / 75.7	68.0 / 86.8	64.7 / 83.3	60.1 / 80.1	62.4 / 81.7	59.5 / 80.5	61.0 / 80.2
BLEURT -pre	56.0 / 74.7	57.1 / 75.7	67.2 / 86.1	62.3 / 81.7	58.4 / 78.3	61.6 / 81.4	55.9 / 76.5	59.8 / 79.2
BLEURT	59.3 / 77.3	59.9 / 79.2	69.5 / 87.8	64.4 / 83.5	61.3 / 81.1	62.9 / 82.4	60.2 / 81.4	62.5 / 81.8
Table 2: Agreement with human ratings on the WMT17 Metrics Shared Task. The metrics are Kendall Tau  $(\tau)$  and the Pearson correlation  $(r$ , the official metric of the shared task), divided by 100.
model	cs-en
τ / DA	de-en
τ / DA	et-en
τ / DA	fi-en
τ / DA	ru-en
τ / DA	tr-en
τ / DA	zh-en
τ / DA	avg
τ / DA
sentBLEU	20.0 / 22.5	31.6 / 41.5	26.0 / 28.2	17.1 / 15.6	20.5 / 22.4	22.9 / 13.6	21.6 / 17.6	22.8 / 23.2
BERTscore w/ BERT	29.5 / 40.0	39.9 / 53.8	34.7 / 39.0	26.0 / 29.7	27.8 / 34.7	31.7 / 27.5	27.5 / 25.2	31.0 / 35.7
BERTscore w/ roBERTa	31.2 / 41.1	42.2 / 55.5	37.0 / 40.3	27.8 / 30.8	30.2 / 35.4	32.8 / 30.2	29.2 / 26.3	32.9 / 37.1
Meteor++	22.4 / 26.8	34.7 / 45.7	29.7 / 32.9	21.6 / 20.6	22.8 / 25.3	27.3 / 20.4	23.6 / 17.5*	26.0 / 27.0
RUSE	27.0 / 34.5	36.1 / 49.8	32.9 / 36.8	25.5 / 27.5	25.0 / 31.1	29.1 / 25.9	24.6 / 21.5*	28.6 / 32.4
YiSi1	23.5 / 31.7	35.5 / 48.8	30.2 / 35.1	21.5 / 23.1	23.3 / 30.0	26.8 / 23.4	23.1 / 20.9	26.3 / 30.4
YiSi1 SRL 18	23.3 / 31.5	34.3 / 48.3	29.8 / 34.5	21.2 / 23.7	22.6 / 30.6	26.1 / 23.3	22.9 / 20.7	25.7 / 30.4
BLEURTbase -pre	33.0 / 39.0	41.5 / 54.6	38.2 / 39.6	30.7 / 31.1	30.7 / 34.9	32.9 / 29.8	28.3 / 25.6	33.6 / 36.4
BLEURTbase	34.5 / 42.9	43.5 / 55.6	39.2 / 40.5	31.5 / 30.9	31.0 / 35.7	35.0 / 29.4	29.6 / 26.9	34.9 / 37.4
BLEURT -pre	34.5 / 42.1	42.7 / 55.4	39.2 / 40.6	31.4 / 31.6	31.4 / 34.2	33.4 / 29.3	28.9 / 25.6	34.5 / 37.0
BLEURT	35.6 / 42.3	44.2 / 56.7	40.0 / 41.4	32.1 / 32.5	31.9 / 36.0	35.5 / 31.5	29.7 / 26.0	35.6 / 38.1
Table 3: Agreement with human ratings on the WMT18 Metrics Shared Task. The metrics are Kendall Tau  $(\tau)$  and WMT's Direct Assessment metrics divided by 100. The star * indicates results that are more than 0.2 percentage points away from the official WMT results (up to 0.4 percentage points away).
5.1 WMT Metrics Shared Task
Datasets and Metrics: We use years 2017 to 2019 of the WMT Metrics Shared Task, to-English language pairs. For each year, we used the official WMT test set, which include several thousand pairs of sentences with human ratings from the news domain. The training sets contain 5,360, 9,492, and 147,691 records for each year. The test sets for years 2018 and 2019 are noisier, as reported by the organizers and shown by the overall lower correlations.
We evaluate the agreement between the automatic metrics and the human ratings. For each year, we report two metrics: Kendall's Tau  $\tau$  (for consistency across experiments), and the official WMT metric for that year (for completeness). The official WMT metric is either Pearson's correlation or a robust variant of Kendall's Tau called DARR, described in the Appendix. All the numbers come from our own implementation of the benchmark. Our results are globally consistent with the official results but we report small differences in 2018 and 2019, marked in the tables.
Models: We experiment with four versions of BLEURT: BLEURT, BLEURTbase, BLEURT-pre and BLEURTbase -pre. The first two models are based on BERT-large and BERT-base. In the latter two versions, we skip the pre-training phase and fine-tune directly on the WMT ratings. For each year of the WMT shared task, we use the test set from the previous years for training and validation. We describe our setup in further detail in the Appendix. We compare BLEURT to participant data from the shared task and automatic metrics that we ran ourselves. In the former case, we use the best-performing contestants for each year, that is,  $\mathrm{chrF + + }$  , BEER, Meteor  $^{+ + }$  , RUSE, Yisil, ESIM and Yisil-SRL (Mathur et al., 2019). All the contestants use the same WMT training data, in addition to existing sentence or token embeddings. In the latter case, we use Moses sentenceBLEU, BERTscore (Zhang et al., 2020), and MoverScore (Zhao et al., 2019). For BERTscore, we use BERT-large uncased for fairness, and roBERTa (the recommended version) for completeness (Liu et al., 2019). We run MoverScore on WMT 2017 using the scripts published by the authors.
Results: Tables 2, 3, 4 show the results. For years 2017 and 2018, a BLEURT-based metric
4The official scripts are public but they suffer from documentation and dependency issues, as shown by a README file in the 2019 edition which explicitly discourages using them.
7885