SlowGuess/NAACL_2022_OCR huggingface dataset:
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2022. Transparent human evaluation for image captioning. In Proc. of NAACL.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. 2021. GENIE: A leaderboard for human-in-the-loop evaluation of text generation.
Shun Kiyono, Takumi Ito, Ryuto Konno, Makoto Morishita, and Jun Suzuki. 2020. Tohoku-AIP-NTT at WMT 2020 news translation task. In Proc. of WMT.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. of ACL Demo and Poster Sessions.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proc. of EMNLP.
Wojciech Krysciński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proc. of EMNLP.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proc. of ACL.
Zuchao Li, Hai Zhao, Rui Wang, Kehai Chen, Masao Utiyama, and Eiichiro Sumita. 2020. SJTU-NICT's supervised and unsupervised neural machine translation systems for the WMT20 news translation task. In Proc. of WMT.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proc. of Text Summarization Branches Out.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Proc. of ECCV.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proc. of EMNLP.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proc. of WMT.
Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proc. of WMT.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proc. of ACL System Demonstrations.
Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2021. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. In Proc. of ACL.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proc. of ACL.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020b. Results of the WMT20 metrics shared task. In Proc. of WMT.
Fandong Meng, Jianhao Yan, Yijin Liu, Yuan Gao, Xi-anfeng Zeng, Qinsong Zeng, Peng Li, Ming Chen, Jie Zhou, Sifan Liu, and Hao Zhou. 2020. WeChat neural machine translation systems for WMT20. In Proc. of WMT.
Swaroop Mishra and Anjana Arunkumar. 2021. How robust are model rankings: A leaderboard customization approach for equitable evaluation. In Proc. of AAAI.
Alexander Molchanov. 2020. PROMT systems for WMT 2020 shared news translation task. In Proc. of WMT.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Caglar Güçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proc. of CoNLL.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proc. of NAACL.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proc. of WMT.
Csaba Oravecz, Katina Bontcheva, László Tihanyi, David Kolovratnik, Bhavani Bhaskar, Adrien Lardilleux, Szymon Klocek, and Andreas Eisele. 2020. eTranslation's submissions to the WMT 2020 news translation task. In Proc. of WMT.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling.* In *Proc. of NAACL Demonstrations*.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proc. of NAACL.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In Proc. of ICLR.
Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proc. of WMT.
Maja Popovic. 2017.  $\mathrm{chrF}++$ : words helping character n-grams. In Proc. of WMT.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proc. of WMT.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JLMR.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proc. of EMNLP.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. IJCV.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green AI. CACM.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proc. of EMNLP.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proc. of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proc. of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proc. of ACL.
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019. An entity-driven framework for abstractive summarization. In Proc. of EMNLP.
Tingxun Shi, Shiyu Zhao, Xiaopu Li, Xiaoxue Wang, Qian Zhang, Di Ai, Dawei Dang, Xue Zhengshan, and Jie Hao. 2020. OPPO's machine translation systems for WMT20. In Proc. of WMT.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of AMTA.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proc. of ACL.
Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proc. of EMNLP.
Robert Tibshirani. 1994. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proc. of WMT.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proc. of CVPR.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proc. of ACL.
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTer: Translation edit rate on character level. In Proc. of WMT.
Daimeng Wei, Hengchao Shang, Zhanglin Wu, Zhengzhe Yu, Liangyou Li, Jiaxin Guo, Minghan Wang, Hao Yang, Lizhi Lei, Ying Qin, and Shiliang Sun. 2020. HW-TSC's participation in the WMT 2020 news translation shared task. In Proc. of WMT.
Liwei Wu, Xiao Pan, Zehui Lin, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2020a. The Volctrans machine translation system for WMT20. In Proc. of WMT.
Shuangzhi Wu, Xing Wang, Longyue Wang, Fangxu Liu, Jun Xie, Zhaopeng Tu, Shuming Shi, and Mu Li. 2020b. Tencent neural machine translation systems for the WMT20 news translation task. In Proc. of WMT.
Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. In Proc. of AAAI.
Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In Proc. of EMNLP.
Lei Yu, Laurent Sartran, Po-Sen Huang, Wojciech Stokowiec, Domenic Donato, Srivatsan Srinivasan, Alek Andreev, Wang Ling, Sona Mokra, Agustin Dal Lago, Yotam Doron, Susannah Young, Phil Blunsom, and Chris Dyer. 2020. The DeepMind Chinese-English document translation system at WMT2020. In Proc. of WMT.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proc. of ICML.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. VinVL: Making visual representations matter in vision-language models. In Proc. of CVPR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. BERTScore: Evaluating text generation with BERT. In Proc. of ICLR.
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. Neural latent extractive document summarization. In Proc. of EMNLP.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In Proc. of AAAI.
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proc. of ACL.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences.
A Case Studies of Evaluation Practice
Fig. 4 depicts breakdowns of evaluation metrics used in the papers on machine translation and summarization from NAACL and ACL 2021. We examined all papers whose title contains "machine translation" and "summariization." We see the clear gap between generation modeling and evaluation research; most researchers do not take advantage of recent metrics that correlate better with human judgments.
B Participating Generators
Here we list the generators submitted in the initial BILLBOARDs.
B.1 WMT20 ZH-EN
Hyperparameter	Value
label smoothing	0.1
# max tokens	1024
dropout rate	0.1
encoder embedding dim	512
encoder ffn dim	2048
# encoder attn heads	8
decoder embedding dim	512
decoder ffn dim	2048
# decoder attn heads	8
max source positions	1024
max target positions	1024
Adam lrate	5 × 10-4
Adam β1	0.9
Adam β2	0.98
lr-scheduler	inverse square
warm-up lr	1 × 10-7
# warmup updates	4000
# max updates	600K
# GPUs	8
length penalty	0.6
We use all 16 submissions for the WMT20 ZHEN task (Barrault et al., 2020) $^{11}$  as well as our own three transformer baselines that were implemented in fairseq (Ott et al., 2019). Our baselines allow researchers to compare their translation models without resource-intensive techniques such as backtranslation (Sennrich et al., 2016a), model assembling, and deep encoders (Kasai et al., 2021a). Tables 4 and 5 list the hyperprameters. We generally follow the setting from Vaswani et al. (2017).
11https://www.statmt.org/wmt20/results.   
html.
Table 4: Transformer-base fairseq hyperparameters and setting.
Hyperparameter	Value
label smoothing	0.1
# max tokens	4096
dropout rate	0.1
encoder embedding dim	1024
encoder ffn dim	4096
# encoder attn heads	16
decoder embedding dim	1024
decoder ffn dim	4096
# decoder attn heads	16
max source positions	1024
max target positions	1024
Adam lrate	5 × 10-4
Adam β1	0.9
Adam β2	0.98
lr-scheduler	inverse square
warm-up lr	1 × 10-7
# warmup updates	4000
# max updates	600K
# GPUs	8
length penalty	0.6
Table 5: Transformer-large and transformer-large-ensemble faireseq hyperparameters and setting. Transformer-large-ensemble ensembles four transformer-large models with different random initializations.
We use newstest-2019 as the dev. set and the official training data. $^{12}$  We apply Moses tokenization (Koehn et al., 2007) and BPE with 32K operations (Sennrich et al., 2016b) to English text. We tokenize Chinese text with the Jieba package, $^{13}$  following Hassan et al. (2018). Separately from English, BPE with 32K operations is then applied to Chinese. The decoder input and output embeddings are tied. Moses detokenization is applied to get the final outputs in the last step. We make the three models and preprocessed train/dev. data publicly available. $^{14}$  Table 6 lists all generators and their automatic evaluation scores from the top-performing metric (ensemble in this case).
B.2 WMT20 EN-DE
Similar to WMT20 ZH-EN, we use all 14 submissions for the WMT20 EN-DE task along with our three transformer baselines. The same hyperparameters are chosen as in WMT20 ZH-EN (Tables 4 and 5). We preprocess both English and German text by the Moses tokenizer and joint BPE with 32K operations. All embeddings are shared. We apply the Moses tokenizer to get the final outputs.
12http://www.statmt.org/wmt20/ translation-task.html.
<sup>13</sup>https://github.com/fxsjy/jieba.
14https://github.com/jungokasai/billboard/tree/master/baselines.
Figure 4: Breakdowns of evaluation metrics used in the papers on machine translation and summarization from NAACL and ACL 2021. We examined all papers whose title contains "machine translation" and "summarization" and disregarded papers primarily on evaluation metrics. "QA" metrics use a QA system to evaluate summaries (e.g., Eyal et al., 2019). "Specialized" indicates specialized evaluation in a particular dimension, rather than the overall generation quality, such as document-level evaluations on contrastive sets (Voita et al., 2019).
Generator	Description	Score
Huoshan Translate	Wu et al. (2020a)	78.85
THUNLP	Not available	78.81
Huawei TSC	Wei et al. (2020)	78.79
DeepMind	Yu et al. (2020)	78.76
WeChat AI	Meng et al. (2020)	78.75
Tencent Translation	Wu et al. (2020b)	78.74
DiDi NLP	Chen et al. (2020)	78.66
OPPO	Shi et al. (2020)	78.59
Online-B	Not available	78.36
SJTU-NICT	Li et al. (2020)	78.27
trans-large-ensemble	§B.1	77.35
trans-large	§B.1	76.98
Online-A	Not available	76.86
trans-base	§B.1	76.79
dong-nmt	Not available	76.74
Online-G	Not available	76.44
zlabs-nlp	Not available	75.79
Online-Z	Not available	75.05
WMT Biomed Baseline	Bawden et al. (2020)	73.89
Table 6: WMT20 ZH-EN generators and reference papers. The score column indicates the score from the metric that currently correlates best with the human judgments (ensemble).
Table 7 shows the generators and their automatic evaluation scores from the top-performing metric (ensemble).
B.3 CNNDM Summarization
We submit all 26 models from Fabbri et al. (2021). Table 8 shows all models and their automatic evaluation scores from the top-performing metric (COMET).
Generator	Description	Score
Tohoku-AIP-NTT	Kiyono et al. (2020)	90.50
Tencent Translate	Wu et al. (2020b)	90.43
OPPO	Shi et al. (2020)	90.42
eTranslation	Oravecz et al. (2020)	90.39
Online-B	Not available	90.38
Huoshan Translate	Wu et al. (2020a)	90.32
AFRL	G Winnup and Anderson (2020)	90.16
Online-A	Not available	90.12
UEDIN	Germann (2020)	89.98
PROMT NMT	Molchanov (2020)	89.66
trans-large	§B.2	89.60
trans-large-ensemble	§B.2	89.59
trans-base	§B.2	89.35
Online-Z	Not available	89.26
Online-G	Not available	88.98
zlabs-nlp	Not available	88.65
WMT Biomed Baseline	Bawden et al. (2020)	88.23
Table 7: WMT20 EN-DE generators and reference papers. The score column indicates the score from the metric that currently correlates best with the human judgments (ensemble).
B.4 MSCOCO Image Captioning
We submit the four strong models from the literature (Kasai et al., 2022).<sup>16</sup> They share similar pipeline structure but vary in model architecture, (pre)training data, model size, and (pre)training objective. Table 9 shows the models with their papers and automatic scores from the top-performing metric (RefCLIP-S).
16https://github.com/jungokasai/THumB/ tree/master/mscoco. 17Model with CIDEr optimization, https://github. com.microsoft/Oscar/blob/master/VinVL_ MODEL_ZOO.md#Image-Captioning-on-COCO. 18Model with CIDEr optimization. 19Model with cross-entropy optimization, https: //vision-explorer.allenai.org/image_ captioning.
15https://github.com/Yale-LILY/SummEval.
Generator	Description	Score
Lead-3	First 3 sentences	-0.011
T5	Raffel et al. (2020)	-0.030
BART	Lewis et al. (2020)	-0.032
Pegasus-dynamic-mix	Zhang et al. (2020a)	-0.044
RNES	Wu and Hu (2018)	-0.049
Unified-ext-abs	Hsu et al. (2018)	-0.056
Pegasus-huge-news	Zhang et al. (2020a)	-0.056
REFRESH	Narayan et al. (2018)	-0.067
ROUGESal	Pasunuru and Bansal (2018)	-0.073
Human-H	Highlights	-0.075
NEUSUM	Zhou et al. (2018)	-0.083
BanditSum	Dong et al. (2018)	-0.083
LATENT	Zhang et al. (2018)	-0.099
Closed-book-decoder	Jiang and Bansal (2018)	-0.112
Multi-task-Ent-QG	Guo et al. (2018)	-0.117
Pointer-Generator	See et al. (2017)	-0.144
UniLM	Dong et al. (2019)	-0.151
Bottom-Up	Gehrmann et al. (2018)	-0.160
JEC	Xu and Durrett (2019)	-0.167
Fast-abs-rl	Chen and Bansal (2018)	-0.189
NeuralTD	Böhm et al. (2019)	-0.215
Improve-abs	KrySciński et al. (2018)	-0.329
BertSum-abs	Liu and Lapata (2019)	-0.341
STRASS	Bouscarrat et al. (2019)	-0.405
GPT-2-zero-shot	Ziegler et al. (2019)	-0.441
SENeca	Sharma et al. (2019)	-0.735
Table 8: CNNDM summarization generators and reference papers. They are from Fabbri et al. (2021), but we apply detokenization (Bird et al., 2009) and/or true-casing (Manning et al., 2014) to standardize the model outputs for better, reproducible evaluations. The score column indicates the score from the metric that currently correlates best with the human judgments (COMET).
Generator	Description	Score
VinVL-large17	Zhang et al. (2021)	83.78
VinVL-base18	Zhang et al. (2021)	83.45
Unified-VLP	Zhou et al. (2020)	82.59
Up-Down19	Anderson et al. (2018)	80.63
C Participating Metrics
Table 10 discusses details and configurations of the automatic metrics that we implement in our initial BILLBOARDs.
Table 9: MSCOCO image captioning generators and reference papers. The score column indicates the score from the metric that currently correlates best with the human judgments (RefCLIP-S).
Metric	Description	Refs.	Src.	Cont.
BLEU20	Papineni et al. (2002)	✓	X	X
ROUGE-321	Lin (2004)	✓	X	X
ROUGE-L	Lin (2004)	✓	X	X
METEOR	Banerjee and Lavie (2005)	✓	X	X
TER22	Snover et al. (2006)	✓	X	X
METEOR23	Banerjee and Lavie (2005)	✓	X	X
chrF24	Popović (2015)	✓	X	X
CIDEr25	Vedantam et al. (2015)	✓	X	X
SPICE	Anderson et al. (2016)	✓	X	X
CharacTER26	Wang et al. (2016)	✓	X	X
chrF++	Popović (2017)	✓	X	X
SummaQA27	Scialom et al. (2019)	X	✓	✓
BERTScore	Zhang et al. (2020b)	✓	X	✓
BLEURT28	Sellam et al. (2020)	✓	X	✓
COMET29	Rei et al. (2020)	✓	✓	✓
COMET-QE	Rei et al. (2020)	X	✓	✓
Prism-ref30	Thompson and Post (2020)	✓	X	✓
Prism-src	Thompson and Post (2020)	X	✓	✓
CLIP-S31	Hessel et al. (2021)	X	✓	✓
RefCLIP-S	Hessel et al. (2021)	✓	✓	✓
RefOnlyC	Kasai et al. (2022)	✓	X	✓
Table 10: Automatic metrics and their reference papers. The refs., src., and cont. columns indicate whether they use references, input source features, and pretrained contextual representations (e.g., BERT; Devlin et al., 2019), respectively.
D Additional Ensemble Metric Ablations
Seen in Table 11 are ablation studies for the ensemble metrics where one of the three selected metrics is removed at a time. Dropping one metric often has no impact on the correlation score, suggesting that these metrics are highly redundant and capture similar aspects of the output quality. BILLBOARDs encourage researchers to explore ways to diversify automatic evaluations by updating the ensemble metric every time a new metric is submitted.
ZH-EN	-0.61	COMET0.61	COMET-QE0.57	BLEURT0.61
EN-DE	-0.51	COMET0.52	COMET-QE0.52	Prism-ref0.52
CNNDM	-0.29	COMET0.23	COMET-QE0.31	BERTScore0.31
COCO	-0.45	RefCLIP-S0.44	RefOnlyC0.42	CIDEr0.43
Table 11: Correlations from ensemble ablation studies. One of the three selected metrics is removed at a time, and a new Lasso regression model is trained on the remaining metrics. The bigger the correlation drop is, the bigger the contribution is from the removed metric. COMET-QE is a referenceless metric.
E Additional Mixed-Effects Analysis
Table 12 presents fixed-effect coefficients that measure how much each automatic metric overrates machines over humans (§2.3). With some exceptions in CNNDM summarization, almost all automatic metrics underrate human generations (significantly positive coefficients). Table 13 swaps the roles of human-generated text, but we still see similar patterns: almost all metrics overrate machines over humans, but the problem is mitigated in COMETQE, a referenceless, quality estimation metric. This confirms that our findings hold independently of the design choice.
F Crowdworker vs. Rubric-based Expert Evaluations
Seen in Table 14 are examples where crowdworker evaluators (Barrault et al., 2020) and professional translators (Freitag et al., 2021) disagree: crowdworkers give lower scores to the human-generated translations than the machine-generated ones. The first case requires document-level context to properly evaluate. Document-level context and diversity in high-quality human translations can mislead crowdworkers.
ZH-EN	COMET-QE	Ensemble	COMET	BLEURT	BERTScore	CharacTER	MoverScore	METEOR
0.13±0.01	0.26±0.01	0.27±0.02	0.32±0.02	0.52±0.02	0.56±0.02	0.57±0.02	0.57±0.02
Prism-ref	chrF	TER	chrF++	ROUGE-3	BLEU	ROUGE-L	Prism-src
0.58±0.02	0.58±0.02	0.59±0.02	0.60±0.02	0.61±0.02	0.62±0.02	0.64±0.02	1.13±0.02
EN-DE	COMET-QE	Ensemble	COMET	MoverScore	chrF	chrF++	BLEU	CharacTER
-0.17±0.02	0.03±0.02	0.08±0.02	0.22±0.03	0.29±0.02	0.32±0.02	0.33±0.03	0.33±0.03
BERTScore	Prism-ref	TER	Prism-src
0.43±0.02	0.44±0.02	0.49±0.03	1.46±0.03
CNNDM	TER	COMET	Ensemble	BERTScore	MoverScore	COMET-QE	CharacTER	BLEURT
-0.58±0.14	-0.17±0.12	-0.16±0.12	-0.04±0.12	-0.03±0.11	0.02±0.11	0.14±0.15	0.25±0.12
SummaQA	ROUGE-L	BLEU	Prism-ref	chrF	chrF++	ROUGE-3	METEOR
0.27±0.10	0.33±0.13	0.37±0.11	0.38±0.12	0.43±0.13	0.45±0.13	0.49±0.11	0.53±0.12
COCO	CLIP-S	RefCLIP-S	CharacTER	chrF	ROUGE-3	chrF++	RefOnlyC	Ensemble
-0.04±0.05	0.09±0.06	0.13±0.07	0.18±0.07	0.22±0.07	0.23±0.07	0.24±0.06	0.24±0.06
SPICE	METEOR	BLEU	CIDEr	ROUGE-L	BERTScore	TER	MoverScore
0.25±0.07	0.32±0.07	0.39±0.07	0.43±0.06	0.44±0.07	0.45±0.06	0.45±0.07	0.51±0.05