SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 12: Fixed-effect coefficients  $\beta_0$  from the linear mixed-effects analysis that measures how much automatic metrics overrate machine text over human, as compared to human raters (§2.3).  $\beta_0 = 0$  is neutral, and statistical significance is indicated by red (positive) or blue text (negative). The subscripts indicate  $90\%$  confidence intervals. COMET-QE, Prism-src, SummaQA and CLIP-S are referenceless metrics. In both WMT20 ZH-EN and WMT20 EN-DE, Human-B is evaluated as human-generated translations. Human-A (WMT20 ZH-EN) and Human-A and Human-P (WMT20 EN-DE) are used as the reference set for reference-based metrics.
ZH-EN	COMET-QE	Ensemble	COMET	BLEURT	TER	BERTScore	ROUGE-3	Prism-ref
0.03±0.01	0.07±0.01	0.08±0.02	0.09±0.02	0.23±0.02	0.24±0.02	0.24±0.02	0.25±0.02
CharacTER	ROUGE-L	chrF	MoverScore	METEOR	chrFpp	BLEU	Prism-src
0.25±0.02	0.26±0.02	0.27±0.02	0.27±0.02	0.29±0.02	0.29±0.02	0.30±0.02	0.79±0.02
EN-DE	COMET-QE	Ensemble	COMET	MoverScore	Prism-ref	chrF	BERTScore	CharacTER
-0.09±0.02	-0.07±0.02	-0.06±0.03	0.02±0.02	0.18±0.02	0.20±0.02	0.21±0.02	0.22±0.02
chrF++	BLEU	TER	Prism-src
0.22±0.02	0.23±0.02	0.32±0.02	1.38±0.03
Table 13: Fixed-effect coefficients  $\beta_0$  from the linear mixed-effects analysis that measures how much automatic metrics overrate machine text over human, as compared to human raters (§2.3). The roles of human translations are swapped: Human-A is evaluated, and Human-B (WMT20 ZH-EN) and Human-B and Human-P (WMT20 EN-DE) are used as the reference. We still see similar patterns to Table 12: almost all automatic metrics overrate machines over humans, but the problem is less severe in the referenceless metric of COMET-QE.
WMT20 ZH-EN
Source	希望兴安省继续为白俄罗斯企业提供便利条件。	凭的是相机而动的时势驾驭。
Huoshan	It is hoped that Xing'an Province will continue to provide convenient conditions for Belarusian enterprises.	It is based on the current situation of the camera.
Human-A	He hoped that Hung Yen Province would continue to provide convenient conditions for Belarusian enterprises.	This relies on the ability to seize opportunities.
Human-B	He hoped that this could continue in the future.	It is based on the observation of various situations at different times.
Table 14: Examples where crowdsourced evaluators (Barrault et al., 2020) and professional translators (Freitag et al., 2021) disagree: crowdworkers give lower scores to the human-generated translations than the machine-generated ones. The first case requires document-level context to properly evaluate. 兴安省 is Hung Yen Province in Vietnam in this context, but there is entity ambiguity. (Xing'an Province that existed in the Republic of China.) The second one illustrates the diversity of human generations that misleads crowdworkers.