SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 5: Improvement in Kendall Tau accuracy on all language pairs of the WMT Metrics Shared Task 2017, varying the number of pre-training steps. 0 steps corresponds to 0.555 Kendall Tau for BLEURTbase and 0.580 for BLEURT.
Training setup. To separate training and validation data, we set aside a fixed ratio of records in such a way that there is no "leak" between the datasets (i.e., train and validation records that share the same source). We use  $10\%$  of the data for validation for years 2017 and 2018, and  $5\%$  for year 2019. We report results for the models that yield the highest Kendall Tau across all records on validation data. The weights associated to each pretraining task (see our Modeling section) are set with grid search, using the train/validation setup of WMT 2017.
Baselines. we use three metrics: the Moses implementation of sentenceBLEU,11 BERTscore,12 and MoverScore,13 which are all available online. We run the Moses tokenizer on the reference and candidate segments before computing sentenceBLEU.
B.3 Robustness to Quality Drift
Data Re-sampling Methodology: We sample the training and test separately, as follows. We split the data in 10 bins of equal size. We then sample each record in the dataset with probabilities  $\frac{1}{B^{\alpha}}$  and  $\frac{1}{(11 - B)^{\alpha}}$  for train and test respectively, where  $B$  is the bin index of the record between 1 and 10, and  $\alpha$  is a predefined skew factor. The skew factor  $\alpha$  controls the drift: a value of 0 has no effect (the ratings are centered around 0), and value of 3.0 yields extreme differences. Note that
the sizes of the datasets decrease as  $\alpha$  increases: we use  $50.7\%$ ,  $30.3\%$ ,  $20.4\%$ , and  $11.9\%$  of the original 5,344 training records for  $\alpha = 0.5$ , 1.0, 1.5, and 3.0 respectively.
B.4 Ablation Experiment-How Much Pre-Training Time is Necessary?
To understand the relationship between pretraining time and downstream accuracy, we pretrain several versions of BLEURT and we fine-tune them on WMT17 data, varying the number of pretraining steps. Figure 5 presents the results. Most gains are obtained during the first 400,000 steps, that is, after about 2 epochs over our synthetic dataset.
11https://github.com/moses-smt/  
mosesdecoder/blob/master/mert/  
sentence-bleu.cpp  
12https://github.com/Tiiger/bert_score  
13https://github.com/AIPHES/  
emnlp19-moverscore
7892