SlowGuess/NAACL_2021_OCR huggingface dataset:
2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy. Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984-4997, Online. Association for Computational Linguistics.
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language generation systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 35-41, Minneapolis, Minnesota. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Maja Popović. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
David M. W. Powers. 1998. Applications and explanations of Zipf's law. In New Methods in Language Processing and Computational Natural Language Learning.
Marcelo O.R. Prates, Pedro H. Avelar, and Luís C. Lamb. 2019. Assessing gender bias in machine translation: a case study with google translate. *Neural Computing and Applications*, pages 1-19.
C. J. Van Rijsbergen. 1979. Information Retrieval, 2nd edition. Butterworth-Heinemann, USA.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
pages 7881-7892, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Anastasia Shimorina. 2018. Human vs automatic metrics: on the importance of correlation design. CoRR, abs/1805.11474.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. Cambridge, MA.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of Machine Learning Research, volume 97, pages 5926-5936, Long Beach, California, USA. PMLR.
Miloš Stanojevic and Khalil Sima'an. 2014. BEER: BEtter evaluation as ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414-419, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4):1-38.
Steven Wegmann, Arlo Faria, Adam Janin, Korbinian Riedhammer, and Nelson Morgan. 2013. The TAO of ATWV: Probing the mysteries of keyword search performance. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 192-197. IEEE.
Hans Yang. 2020. XLM-UNMT-Models. https://github.com/Hansxsourse/XLM-UNMT-Models.
Ilya Zavorin, Aric Bills, Cassian Corey, Michelle Morrison, Audrey Tong, and Richard Tong. 2020. Corpora for cross-language information retrieval in six less-resourced languages. In Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020), pages 7-13, Marseille, France. European Language Resources Association.
George KingsleyZipf.1949.Human behaviour and the principle of least-effort.Cambridge MA edn. Addison-Wesley.
A Metrics Reproducibility
BLEU scores reported in this work are computed with the SACREBLEU library and have signature BLEU+case.mixed+lang.<xx>--<yy>+numrefs.1 +smooth.exp+tok.<TOK>+version.1.4.13, where <TOK> is zh for Chinese, and 13a for all other languages. MACROF<sub>1</sub> and MICROF<sub>1</sub> use the same tokenizer as BLEU. CHRF<sub>1</sub> is also obtained using SACREBLEU and has signaturechrF1+lang.<xx>--<yy>+numchars.6 +space,false +version.1.4.13. BLUERT scores are from the base model of Sellam et al. (2020), which is fine-tuned on WMT Metrics ratings data from 2015-2018. The BLEURT model is retrieved from https://storage.googleapis.com/bleurt-oss/bleurt-base-128.zip.
$\mathrm{MACROF}_1$  and  $\mathrm{MICROF}_1$  are computed using our fork of SACREBLEU as:
sacrebleu $REF -m macrof microf < $HYP.
B Agreement with WMT Human Judgments
Tables 9, 10, and 11 provide  $\tau$  between MT metrics and human judgments on WMT Metrics task 2017-2019.  $\star$ BLEU is based on pre-computed scores in WMT metrics package, whereas BLEU is based on our recalculation using SACREBLEU. Values marked with  $\times$  are not significant at  $\alpha = 0.05$ , and hence corresponding rows are excluded from the calculation of mean, median, and standard deviation.
Since  $\mathrm{MACROF}_1$  is the only metric that does not achieve statistical significance in the WMT 2019 EN-ZH setting, we carefully inspected it. Human scores for this setting are obtained without looking at the references by bilingual speakers (Ma et al., 2019), but the ZH references are found to have a large number of bracketed EN phrases, especially proper nouns that are rare types. When the text inside these brackets is not generated by an MT system,  $\mathrm{MACROF}_1$  naturally penalizes heavily due to the poor recall. Since other metrics assign lower importance to poor recall of such rare types, they achieve relatively better correlation to human scores than  $\mathrm{MACROF}_1$ . However, since the  $\tau$  values for EN-ZH are relatively lower than the other language pairs, we conclude that poor correlation of  $\mathrm{MACROF}_1$  in EN-ZH is due to poor quality references. Some settings did not achieve statistical significance due to a smaller sample set as there were fewer MT systems submitted, e.g. 2017 CS-EN.
*BLEU	BLEU	MACROF1	MICROF1	CHRF1
DE-CS	.855	.745	.964	.917	.982
DE-EN	.571	.655	.723	.695	.742
DE-FR	.782	.881	.927	.844	.915
EN-CS	.709	.954	.927	.927	.908
EN-DE	.540	.752	.741	.773	.824
EN-FI	.879	.818	.879	.848	.923
EN-GU	.709	.709	.600	.734	.709
EN-KK	.491	.527	.685	.636	.661
EN-LT	.879	.848	.970	.939	.881
EN-RU	.870	.848	.939	.879	.930
FI-EN	.788	.809	.909	.901	.875
FR-DE	.822	.733	.733	.764	.815
GU-EN	.782	.709	.855	.891	.945
KK-EN	.891	.844	.796	.844	.881
LT-EN	.818	.855	.844	.855	.833
RU-EN	.692	.729	.714	.780	.757
ZH-EN	.695	.695	.752	.676	.715
Median	.782	.752	.844	.844	.875
Mean	.751	.771	.821	.818	.841
SD	.124	.101	.112	.093	.095
EN-ZH	.606	.606	x.424	.595	.594
Wins	3	3	6	3	5
Table 9: WMT19 Metrics task: Kendall's  $\tau$  between metrics and human judgments.
C UNMT and SNMT Models
The UNMT models follow XLM's standard architecture and are trained with 5 million monolingual sentences for each language using a vocabulary size of 60,000. We train SNMT models for EN  $\leftrightarrow$  DE and select models with the most similar (or a slightly lower) BLEU as their UNMT counterparts on newstest2019. The DE-EN model selected is trained with 1 million sentences of parallel data and a vocabulary size of 64,000, and the EN-DE model selected is trained with 250,000 sentences of parallel data and a vocabulary size of 48,000. For EN  $\leftrightarrow$  FR and EN  $\leftrightarrow$  RO, we select SNMT models from submitted systems to WMT shared tasks that have similar or slightly lower BLEU scores to corresponding UNMT models, based on NewsTest2014 for EN  $\leftrightarrow$  FR and NewsTest2016 for EN  $\leftrightarrow$  RO.
Figure 1, which is a visualization of  $\mathrm{MACROF_1}$  for SNMT and UNMT models, shows that UNMT is generally better than SNMT on frequent types, however, SNMT outperforms UNMT on the rest leading to a crossover point in  $\mathrm{MACROF_1}$  curves. Since  $\mathrm{MACROF_1}$  assigns relatively higher weights to infrequent types than in BLEU, SNMT gains higher  $\mathrm{MACROF_1}$  than UNMT while both have approximately the same BLEU, as reported in Table 6.
A complete comparison of UNMT vs SNMT in different languages is in Table 12. A manual analysis of the ten sentences with the largest magnitude favoritism according to  $\mathrm{MACROF}_1$  and BLEU in
$\star$  BLEU BLEU  $\mathrm{MACROF}_1$  MICROF  $\mathbf{\Pi}_{1}$  CHRF
DE-EN	.828	.845	.917	.883	.919
EN-DE	.778	.750	.850	.783	.848
EN-ET	.868	.868	.934	.906	.949
EN-FI	.901	.848	.901	.879	.945
EN-RU	.889	.889	.944	.889	.930
EN-ZH	.736	.729	.685	.833	.827
ET-EN	.884	.900	.884	.878	.904
FI-EN	.944	.944	.889	.915	.957
RU-EN	.786	.786	.929	.857	.869
ZH-EN	.824	.872	.738	.780	.820
EN-CS	1.000	1.000	.949	1.000	.949
Median	.868	.868	.901	.879	.919
Mean	.858	.857	.875	.873	.902
SD	.077	.080	.087	.062	.052
TR-EN	×.200	×.738	×.400	×.316	×.632
EN-TR	×.571	×.400	.837	×.571	.849
CS-EN	×.800	×.800	×.600	×.800	×.738
Wins	1	2	3	2	6
Table 10: WMT18 Metrics task: Kendall's  $\tau$  between metrics and human judgments.
$\star$  BLEU BLEU  $\mathrm{MACROF}_1$  MICROFCHRF
DE-EN	.564	.564	.734	.661	.744
EN-CS	.758	.751	.767	.758	.878
EN-DE	.714	.767	.562	.593	.720
EN-FI	.667	.697	.769	.718	.782
EN-RU	.556	.556	.778	.648	.669
EN-ZH	.911	.911	.600	.854	.899
LV-EN	.905	.714	.905	.905	.905
RU-EN	.778	.611	.611	.722	.800
TR-EN	.911	.778	.674	.733	.907
ZH-EN	.758	.780	.736	.824	.732
Median	.758	.733	.735	.728	.791
Mean	.752	.713	.714	.742	.804
SD	.132	.110	.103	.097	.088
FI-EN	.867	.867	.733	.867	.867
EN-TR	.857	.714	.571	.643	.849
CS-EN	×1.000	×1.000	.667	.667	×.913
Wins	5	4	2	2	6
Table 11: WMT17 Metrics task: Kendall's  $\tau$  between metrics and human judgments.
Translation	SNMT	UNMT	SNMT Name
DE-EN NewsTest2019	32.7	33.9	Our Transformer
EN-DE NewsTest2019	24.0	24.0	Our Transformer
FR-EN NewsTest2014	31.1	31.2	OnlineA.0
EN-FR NewsTest2014	25.6	27.1	PROMT-Rule-based.3083
RO-EN NewsTest2016	30.8	29.6	Online-A.0
EN-RO NewsTest2016	31.2	31.0	uedin-pbmt.4362
Table 12: SNMT systems are selected such that their BLEU scores are approximately the same as the available pretrained UNMT models.
the FR-EN and RO-EN test sets is in Table 13 and Table 14. The complete texts of these sentences, their reference translations, and the system translations (including DE-EN mentioned in Sec 4), are shown in Tables 15, 16, 17, 18, 19, and 20.
Figure 1: SNMT vs UNMT  $\mathrm{MACROF_1}$  on the most frequent 500 types. UNMT outperforms SNMT on frequent types that are weighed heavily by BLEU however, SNMT is generally better than UNMT on rare types; hence, SNMT has a higher  $\mathrm{MACROF_1}$ .
δMACROF1	Fav	Analysis	δBLEU	Fav	Analysis
.044	S	S: synonym; U: untranslation, synonym	-.026	U	U: synonym; S: omitted adv, word order
-.038	U	U: no issues; S: synonym	.025	S	S: no issues; U: determiner, word order
.035	S	S: synonym; U: untranslation, synonym	.024	S	S: no issues; U: repetition, form
-.034	U	U: no issues; S: synonym; word_order	.021	S	S: verb, synonym; U: untranslation, noun, time, synonym
-.034	U	U: synonym; S: word_order, verb_ref	.021	S	S: synonym; U: synonym
-.033	U	U: no issues; S: synonym	-.021	U	U: omitted NER; S: synonym, word order
.033	S	S: word order; U: untranslation, NER, word order	-.021	U	U: untranslation; S: verb, word order
.032	S	S: synonym; U: number, omitted noun, un-translation, verb	-.021	U	U: synonym; S: extra preposition, synonym, word order
.030	S	S: adj; U: untranslation	.021	S	S: no issues; U: NER
.030	S	S: noun, synonym; U: noun, synonym	-.020	U	U: synonym; S: synonym, word order
Table 13: Analysis of the ten FR-EN test set segments with the most favoritism in SNMT (S) or UNMT (U), according to  $\mathrm{MACROF}_1$  (left) and BLEU (right). Fav is the favored system by metrics. Actual examples are shown in Appendix Tables 17 and 18.
δMACROF1	Fav	Analysis	δBLEU	Fav	Analysis
.131	S	S: word order; U: repetition, word order	.114	S	S: word order; U: repetition, word order
.063	S	S: noun, word order; U: repetition, untranslation, noun	.089	S	S: no issues; U: omitted noun, omitted time, NER
.062	S	S: extra, untranslation; U: untranslation, copy	-.072	U	U: country, untranslation; S: noun, word order
-.052	U	U: untranslation x 3, synonym; S: untranslation, synonym	-.045	U	U: synonym; S: synonym, word order
-.052	U	U: untranslation, NER, synonym; S: NER, synonym	-.041	U	U: untranslation; S: word order, subject
-.052	U	U: extra; S: untranslation	-.040	U	U: no issues; S: number, omitted preposition
-.050	U	U: adv; S: incoherent, adv	.039	S	S: extra, untranslation; U: untranslation, copy
-.050	U	U: active/passive voice, name; S: name	.036	S	S: no issues; U: extra verb
-.049	U	U: untranslation; S: untranslation, word order	-.035	U	U: repetition, untranslation, S: verb, synonym, word order
.048	S	S: no issues; U: NER	.034	S	S: synonym; U: untranslation
Table 14: Analysis of the ten RO-EN test set segments with the most favoritism in SNMT (S) or UNMT (U), according to  $\mathrm{MACROF}_1$  (left) and BLEU (right). Fav is the favored system by metrics. Actual examples are shown in Appendix Tables 19 and 20.
Table 15: Top 10 segments by  $\left| {\delta }_{\text{MACROF }1}\left( {i,{h}_{S},{h}_{U}}\right) \right|$  on DE-EN.
Table 17: Top 10 segments by  $\left| {\delta }_{\text{MACROF}1}\left( {i,{h}_{S},{h}_{U}}\right) \right|$  on FR-EN.
Table 18: Top 10 segments by  $\left| {\delta }_{\text{BLEU }}\left( {i,{h}_{S},{h}_{U}}\right) }\right|$  on FR-EN.