SlowGuess/ACL_2020_OCR huggingface dataset:
model	de-en
τ / DA	fi-en
τ / DA	gu-en
τ / DA	kk-en
τ / DA	lt-en
τ / DA	ru-en
τ / DA	zh-en
τ / DA	avg
τ / DA
sentBLEU	19.4 / 5.4	20.6 / 23.3	17.3 / 18.9	30.0 / 37.6	23.8 / 26.2	19.4 / 12.4	28.7 / 32.2	22.7 / 22.3
BERTscore w/ BERT	26.2 / 17.3	27.6 / 34.7	25.8 / 29.3	36.9 / 44.0	30.8 / 37.4	25.2 / 20.6	37.5 / 41.4	30.0 / 32.1
BERTscore w/ roBERTa	29.1 / 19.3	29.7 / 35.3	27.7 / 32.4	37.1 / 43.1	32.6 / 38.2	26.3 / 22.7	41.4 / 43.8	32.0 / 33.6
ESIM	28.4 / 16.6	28.9 / 33.7	27.1 / 30.4	38.4 / 43.3	33.2 / 35.9	26.6 / 19.9	38.7 / 39.6	31.6 / 31.3
YiSi1 SRL 19	26.3 / 19.8	27.8 / 34.6	26.6 / 30.6	36.9 / 44.1	30.9 / 38.0	25.3 / 22.0	38.9 / 43.1	30.4 / 33.2
BLEURTbase -pre	30.1 / 15.8	30.4 / 35.4	26.8 / 29.7	37.8 / 41.8	34.2 / 39.0	27.0 / 20.7	40.1 / 39.8	32.3 / 31.7
BLEURTbase	31.0 / 16.6	31.3 / 36.2	27.9 / 30.6	39.5 / 44.6	35.2 / 39.4	28.5 / 21.5	41.7 / 41.6	33.6 / 32.9
BLEURT -pre	31.1 / 16.9	31.3 / 36.5	27.6 / 31.3	38.4 / 42.8	35.0 / 40.0	27.5 / 21.4	41.6 / 41.4	33.2 / 32.9
BLEURT	31.2 / 16.9	31.7 / 36.3	28.3 / 31.9	39.5 / 44.6	35.2 / 40.6	28.3 / 22.3	42.7 / 42.4	33.8 / 33.6
Table 4: Agreement with human ratings on the WMT19 Metrics Shared Task. The metrics are Kendall Tau  $(\tau)$  and WMT's Direct Assessment metrics divided by 100. All the values reported for Yisi1_SRL and ESIM fall within 0.2 percentage of the official WMT results.
Figure 1: Distribution of the human ratings in the train/validation and test datasets for different skew factors.
dominates the benchmark for each language pair (Tables 2 and 3). BLEURT and BLEURTbase are also competitive for year 2019: they yield the best results for all language pairs on Kendall's Tau, and they come first for 3 out of 7 pairs on DARR. As expected, BLEURT dominates BLEURTbase in the majority of cases. Pre-training consistently improves the results of BLEURT and BLEURTbase. We observe the largest effect on year 2017, where it adds up to 7.4 Kendall Tau points for BLEURTbase (zh-en). The effect is milder on years 2018 and 2019, up to 2.1 points (tr-en, 2018). We explain the difference by the fact that the training data used for 2017 is smaller than the datasets used for the following years, so pre-training is likelier to help. In general pretraining yields higher returns for BERT-base than for BERT-large—in fact, BLEURTbase with pretraining is often better than BLEURT without.
Takeaways: Pre-training delivers consistent improvements, especially for BLEURT-base. BLEURT yields state-of-the-art performance for all years of the WMT Metrics Shared task.
5.2 Robustness to Quality Drift
We assess our claim that pre-training makes BLEURT robust to quality drifts, by constructing
Figure 2: Agreement between BLEURT and human ratings for different skew factors in train and test.
a series of tasks for which it is increasingly pressured to extrapolate. All the experiments that follow are based on the WMT Metrics Shared Task 2017, because the ratings for this edition are particularly reliable.[5]
Methodology: We create increasingly challenging datasets by sub-sampling the records from the WMT Metrics shared task, keeping low-rated translations for training and high-rated translations for test. The key parameter is the skew factor  $\alpha$ , that measures how much the training data is left-skewed and the test data is right-skewed. Figure 1 demonstrates the ratings distribution that we used in our experiments. The training data shrinks as  $\alpha$  increases: in the most extreme case ( $\alpha = 3.0$ ), we use only  $11.9\%$  of the original 5,344 training records. We give the full detail of our sampling methodology in the Appendix.
We use BLEURT with and without pre-training and we compare to Moses sentBLEU and BERTscore. We use BERT-large uncased for both BLEURT and BERTscore.
5The organizers managed to collect 15 adequacy scores for each translation, and thus the ratings are almost perfectly repeatable (Bojar et al., 2017)
7886
Figure 3: Absolute Kendall Tau of BLEU, Meteor, and BLEURT with human judgements on the WebNLG dataset, varying the size of the data used for training and validation.
Results: Figure 2 presents BLEURT's performance as we vary the train and test skew independently. Our first observation is that the agreements fall for all metrics as we increase the test skew. This effect was already described in the 2019 WMT Metrics report (Ma et al., 2019). A common explanation is that the task gets more difficult as the ratings get closer—it is easier to discriminate between "good" and "bad" systems than to rank "good" systems.
Training skew has a disastrous effect on BLEURT without pre-training: it is below BERTscore for  $\alpha = 1.0$ , and it falls under sentBLEU for  $\alpha \geq 1.5$ . Pre-trained BLEURT is much more robust: the only case in which it falls under the baselines is  $\alpha = 3.0$ , the most extreme drift, for which incorrect translations are used for train while excellent ones for test.
Takeaways: Pre-training makes BLEURT significantly more robust to quality drifts.
5.3 WebNLG Experiments
In this section, we evaluate BLEURT's performance on three tasks from a data-to-text dataset, the WebNLG Challenge 2017 (Shimorina et al., 2019). The aim is to assess BLEURT's capacity to adapt to new tasks with limited training data.
Dataset and Evaluation Tasks: The WebNLG challenge benchmarks systems that produce natural language description of entities (e.g., buildings, cities, artists) from sets of 1 to 5 RDF triples. The organizers released the human assessments for 9 systems over 223 inputs, that is, 4,677 sentence
pairs in total (we removed null values). Each input comes with 1 to 3 reference descriptions. The submissions are evaluated on 3 aspects: semantics, grammar, and fluency. We treat each type of rating as a separate modeling task. The data has no natural split between train and test, therefore we experiment with several schemes. We allocate  $0\%$  to about  $50\%$  of the data to training, and we split on both the evaluated systems or the RDF inputs in order to test different generalization regimes.
Systems and Baselines: BLEURT -pre -wmt, is a public BERT-large uncased checkpoint directly trained on the WebNLG ratings. BLEURT -wmtwas first pre-trained on synthetic data, then fine-tuned on WebNLG data. BLEURT was trained in three steps: first on synthetic data, then on WMT data (16-18), and finally on WebNLG data. When a record comes with several references, we run BLEURT on each reference and report the highest value (Zhang et al., 2020).
We report four baselines: BLEU, TER, Meteor, and BERTscore. The first three were computed by the WebNLG competition organizers. We ran the latter one ourselves, using BERTlarge uncased for a fair comparison.
Results: Figure 3 presents the correlation of the metrics with human assessments as we vary the share of data allocated to training. The more pretrained BLEURT is, the quicker it adapts. The vanilla BERT approach BLEURT -pre -wmt requires about a third of the WebNLG data to dominate the baselines on the majority of tasks, and it still lags behind on semantics (split by system). In
Figure 4: Improvement in Kendall Tau on WMT 17 varying the pre-training tasks.
contrast, BLEURT -wmt is competitive with as little as 836 records, and BLEURT is comparable with BERTscore with zero fine-tuning.
Takeaways: Thanks to pre-training, BLEURT can quickly adapt to the new tasks. BLEURT finetuned twice (first on synthetic data, then on WMT data) provides acceptable results on all tasks without training data.
5.4 Ablation Experiments
Figure 4 presents our ablation experiments on WMT 2017, which highlight the relative importance of each pre-training task. On the left side, we compare BLEURT pre-trained on a single task to BLEURT without pre-training. On the right side, we compare full BLEURT to BLEURT pretrained on all tasks except one. Pre-training on BERTscore, entailment, and the backtranslation scores yield improvements (symmetrically, ablating them degrades BLEURT). Oppositely, BLEU and ROUGE have a negative impact. We conclude that pre-training on high quality signals helps BLEURT, but that metrics that correlate less well with human judgment may in fact harm the model.[6]
The WMT shared metrics competition (Bojar et al., 2016; Ma et al., 2018, 2019) has inspired
the creation of many learned metrics, some of which use regression or deep learning (Stanojevic and Sima'an, 2014; Ma et al., 2017; Shimanaka et al., 2018; Chen et al., 2017; Mathur et al., 2019). Other metrics have been introduced, such as the recent MoverScore (Zhao et al., 2019) which combines contextual embeddings and Earth Mover's Distance. We provide a head-to-head comparison with the best performing of those in our experiments. Other approaches do not attempt to estimate quality directly, but use information extraction or question answering as a proxy (Wiseman et al., 2017; Goodrich et al., 2019; Eyal et al., 2019). Those are complementary to our work.
There has been recent work that uses BERT for evaluation. BERTScore (Zhang et al., 2020) proposes replacing the hard n-gram overlap of BLEU with a soft-overlap using BERT embeddings. We use it in all our experiments. Bertr (Mathur et al., 2019) and YiSi (Mathur et al., 2019) also make use of BERT embeddings to capture similarity. SumQE (Xenouleas et al., 2019) fine-tunes BERT for quality estimation as we describe in Section 3. Our focus is different—we train metrics that are not only state-of-the-art in conventional IID experimental setups, but also robust in the presence of scarce and out-of-distribution training data. To our knowledge no existing work has explored pretraining and extrapolation in the context of NLG.
Previous studies have used noising for referenceless evaluation (Dušek et al., 2019). Noisy pre-training has also been proposed before for other tasks such as paraphrasing (Wieting et al., 2016; Tomar et al., 2017) but generally not with synthetic data. Generating synthetic data via paraphrases and perturbations has been commonly used for generating adversarial examples (Jia and Liang, 2017; Iyyer et al., 2018; Belinkov and Bisk, 2018; Ribeiro et al., 2018), an orthogonal line of research.
We presented BLEURT, a reference-based text generation metric for English. Because the metric is trained end-to-end, BLEURT can model human assessment with superior accuracy. Furthermore, pre-training makes the metrics robust particularly robust to both domain and quality drifts. Future research directions include multilingual NLG evaluation, and hybrid methods involving both humans and classifiers.
<sup>6</sup>Do those results imply that BLEU and ROUGE should be removed from future versions of BLEURT? Doing so may indeed yield slight improvements on the WMT Metrics 2017 shared task. On the other hand the removal may hurt future tasks in which BLEU or ROUGE actually correlate with human assessments. We therefore leave the question open.
7888
Thanks to Eunsol Choi, Nicholas FitzGerald, Jacob Devlin, and to the members of the Google AI Language team for the proof-reading, feedback, and suggestions. We also thank Madhavan Kidambi and Ming-Wei Chang, who implemented blank-filling with BERT.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR.
Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR.
Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of WMT.
Ondrej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojevic. 2016. Results of the wmt16 metrics shared task. In Proceedings of WMT.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of EMNLP.
Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. Proceedings of ACL.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. Proceedings of ACL.
Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of NAACL HLT.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL HLT.
Ondrej Dusek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser. 2019. Automatic quality estimation for natural language generation: Ranting (jointly rating and ranking). Proceedings of INLG.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of NAACL HLT.
Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dolkar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of CVPR.
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings NAACL HLT.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of INLG.
Ben Goodrich, Mohammad Ahmad Saleh, Peter Liu, and Vinay Rao. 2019. Assessing the factual accuracy of text generation. In Proceedings of ACM SIGKDD.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. Proceedings of NAACL HLT.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. Proceedings of EMNLP.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Karen Kukich. 1983. Design of a knowledge-based report generator. In Proceedings of ACL.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. Proceedings of EMNLP.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692.
Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018. Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of WMT.
Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined mt metric based on direct assessment-casict-dcu submission to wmt17 metrics task. In Proceedings of WMT.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of WMT.
Inderjeet Mani. 1999. Advances in automatic text summarization. MIT press.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of ACL.
Kathleen McKeown. 1992. Text generation. Cambridge University Press.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. Proceedings of EMNLP.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. Proceedings of ACL.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of WMT.
Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2019. Webnlg challenge: Human evaluation results. Technical report.
Ronnie W Smith and D Richard Hipp. 1994. Spoken natural language dialog systems: A practical approach. Oxford University Press.
Milos Stanojevic and Khalil Sima'an. 2014. Beer: Better evaluation as ranking. In Proceedings of WMT.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv:1910.08684.
Gaurav Singh Tomar, Thyago Duque, Oscar Täckström, Jakob Uszkoreit, and Dipanjan Das. 2017. Neural paraphrase identification of questions with noisy pretraining. Proceedings of the First Workshop on Subword and Character Level Models in NLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. Proceedings of ICML.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. Proceedings of ICLR.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of NAACL HLT.
Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. Proceedings of EMNLP.
Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. Sumque: a bert-based summary quality estimation model supplementary material. In Proceedings of EMNLP.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* Proceedings of ICLR.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. Proceedings of EMNLP.
A Implementation Details of the Pre-Training Phase
This section provides implementation details for some of the pre-training techniques described in the main paper.
A.1 Data Generation
Random Masking: We use two masking strategies. The first strategy samples random words in the sentence and it replaces them with masks (one for each token). Thus, the masks are scattered across the sentence. The second strategy creates contiguous sequences: it samples a start position  $s$ , a length  $l$  (uniformly distributed), and it masks all the tokens spanned by words between positions  $s$  and  $s + l$ . In both cases, we use up to 15 masks per sentence. Instead of running the language model once and picking the most likely token at each position, we use beam search (the beam size 8 by default). This enforces consistency and avoids repeated sequences, e.g., “, , , ”
Backtranslation: Consider English and French. Given a forward translation model  $P_{\mathrm{en} \rightarrow \mathrm{fr}}(z_{\mathrm{fr}} | z_{\mathrm{en}})$  and backward translation model  $P_{\mathrm{fr} \rightarrow \mathrm{en}}(z_{\mathrm{en}} | z_{\mathrm{fr}})$ , we generate  $\tilde{z}$  as follows:
$$
\tilde{z} = \operatorname *{arg  max}_{z_{\mathrm{en}}}\bigl(P_{\mathrm{fr}\to \mathrm{en}}(z_{\mathrm{en}}|z_{\mathrm{fr}}^{*})\bigr)
$$
where  $z_{\mathrm{fr}}^{*} = \arg \max_{z_{\mathrm{fr}}} (P_{\mathrm{fr}\to \mathrm{en}}(z_{\mathrm{fr}}|z))$ . For the translations, we use a Transformer model (Vaswani et al., 2017), trained on English-German with the tensor2tensor framework.
Word dropping: Given a synthetic example  $(z, \tilde{z})$  we generate a pair  $(z, \tilde{z}^{\prime})$ , by randomly dropping words from  $\tilde{z}$ . We draw the number of words to drop uniformly, up to the length of the sentence. We apply this transformation on about  $30\%$  of the data generated with the previous method.
A.2 Pre-Training Tasks
We now provide additional details on the signals we used for pre-training.
Automatic Metrics: As shown in the table, we use three types of signals: BLEU, ROUGE, and BERTscore. For BLEU, we used the original Moses SENTENCEBLEU $^{8}$  implementation, using the Moses tokenizer and the default parameters. For ROUGE, we used the seq2seq implementation of ROUGE-N. $^{9}$  We used a custom implementation of BERTSCORE, based on BERT-large uncased. ROUGE and BERTscore return three scores: precision, recall, and F-score. We use all three quantities.
Backtranslation Likelihood: We compute all the losses using custom Transformer model (Vaswani et al., 2017), trained on two language pairs (English-French and English-German) with the tensor2tensor framework.
Normalization: All the regression labels are normalized before training.
Setting the weights of the pre-training tasks: We set the weights  $\gamma_{k}$  with grid search, optimizing BLEURT's performance on WMT 17's
validation set. To reduce the size of the grid, we make groups of pre-training tasks that share the same weights:  $(\tau_{\mathrm{BLEU}},\tau_{\mathrm{ROUGE}},\tau_{\mathrm{BERTscore}})$ $(\tau_{\mathrm{en - fr},z|\tilde{z}},\tau_{\mathrm{en - fr},\tilde{z} |z},\tau_{\mathrm{en - de},z|\tilde{z}},\tau_{\mathrm{en - de},\tilde{z} |z}),$  and  $(\tau_{\mathrm{entail}},\tau_{\mathrm{backtran\_flag}})$
B Experiments-Supplementary Material
B.1 Training Setup for All Experiments
We user BERT's public checkpoints $^{10}$  with Adam (the default optimizer), learning rate 1e-5, and batch size 32. Unless specified otherwise, we use 800,00 training steps for pre-training and 40,000 steps for fine-tuning. We run training and evaluation in parallel: we run the evaluation every 1,500 steps and store the checkpoint that performs best on a held-out validation set (more details on the data splits and our choice of metrics in the following sections). We use Google Cloud TPUs v2 for learning, and Nvidia Tesla V100 accelerators for evaluation and test. Our code uses Tensorflow 1.15 and Python 2.7.
B.2 WMT Metric Shared Task
Metrics. The metrics used to compare the evaluation systems vary across the years. The organizers use Pearson's correlation on standardized human judgments across all segments in 2017, and a custom variant of Kendall's Tau named "DARR" on raw human judgments in 2018 and 2019. The latter metrics operates as follows. The organizers gather all the translations for the same reference segment, they enumerate all the possible pairs (translation $_1$ , translation $_2$ ), and they discard all the pairs which have a "similar" score (less than 25 points away on a 100 points scale). For each remaining pair, they then determine which translation is the best according both human judgment and the candidate metric. Let  $|\text{Concordant}|$  be the number of pairs on which the NLG metrics agree and  $|\text{Discordant}|$  be those on which they disagree, then the score is computed as follows:
$$
\begin{array}{c} | \text {C o n c o r d a n t} | - | \text {D i s c o r d a n t} | \\ \hline | \text {C o n c o r d a n t} | + | \text {D i s c o r d a n t} | \end{array}
$$
The idea behind the 25 points filter is to make the evaluation more robust, since the judgments collected for WMT 2018 and 2019 are noisy. Kendall's Tau is identical, but it does not use the filter.
$^{7}$ https://github.com/tensorflow/tensor2tensor
$^{8}$ https://github.com/moses-smt/mosesdecoder/blob/master/mert/sentenceBLEU.cpp
$^{9}$ https://github.com/google/seq2seq/blob/master/seq2seq/metrics/rouge.py
10 https://github.com/google-research/bert
7891