SlowGuess/ACL_2020_OCR huggingface dataset:
Considering BLEU (top plot in Figure 2), for insignificant BLEU differences, humans judge one system to be better than the other for half of these system pairs. This corresponds to a Type I error. It is of concern that BLEU cannot detect these differences. Worse, the difference in human scores has a very wide range. Conversely, when the BLEU score is significant but in the range 0–3, more than half of these systems are judged to be insignificantly different in quality (corresponding to a Type II error). For higher BLEU deltas, these errors diminish, however, even for a BLEU difference between 3 and 5 points, about a quarter of these system pairs are of similar quality. This paints a dour picture for the utility of BLEU as a tool for gatekeeping (i.e., to define a ‘minimum publishable unit’ in deciding paper acceptance on empirical grounds, through bounding the risk of Type II errors), as the unit would need to be wholly large to ensure only meaningful improvements are accepted. Were we seek to minimise Type I errors in the interests of nurturing good ideas, the thresh-
Figure 5: The agreement between metric errors over all 1362 system comparisons. The values in the diagonal indicate the total number of Type 1 and Type 2 errors for the metric. The off-diagonal cells show the total number of errors made by the row-metric where the column-metric is correct.
old would need to be so low as to be meaningless, effectively below the level required for acceptance of the bootstrap significance test.
The systems evaluated consist of a mix of systems submitted by researchers (mostly neural models) and anonymous online systems (where the MT system type is unknown). Even when we restrict the set of systems to only neural models submitted by researchers, the patterns of Type 1 and Type 2 errors remain the same (figure omitted for space reasons).
TER makes similar errors: TER scores can wrongly show that a system is much better than another when humans have judged them similar, or even worse, drawn the opposite conclusion.
CHRF, Y1S1-1 and ESIM have fewer errors compared to BLEU and TER. When these metrics mistakenly fail to detect a difference between systems, the human score difference is considerably lower than for BLEU. Accordingly, they should be used in place of BLEU. However the above argument is likely to still hold true as to their utility for gatekeeping or nurturing progress, in that the thresholds would still be particularly punitive or permissive, for the two roles, respectively.
Finally, Figure 5 looks at agreement between metric decisions when comparing MT systems. As expected, when BLEU or TER disagree with CHRF, ESIM, or YIsI-1, the former are more likely to be wrong. BLEU and TER have an  $80\%$  overlap in errors. The decisions of ESIM, a trained
neural model, diverge a little more from the other metrics. Overall, despite the variety of approaches towards the task, all five metrics have common biases: over half of all erroneous decisions made by a particular metric are made in common with all other metrics.
In this paper, we revisited the findings of the metrics task at WMT 2019, which flagged potential problems in the current best practises for assessment of evaluation metrics.
Pearson's correlation coefficient is known to be unstable for small sample sizes, particularly when the systems in consideration are very close in quality. This goes some way to explaining the findings whereby strong correlations between metric scores and human judgements evaporate when considering small numbers of strong systems. We show that the same can be true for any small set of similar quality systems, not just the top systems. This effect can partly be attributed to noise due to the small sample size, rather than true shortcomings in the metrics themselves. We need better methods to empirically test whether our metrics are less reliable when evaluating high quality MT systems.
A more serious problem, however, is outlier systems, i.e. those systems whose quality is much higher or lower than the rest of the systems. We found that such systems can have a disproportionate effect on the computed correlation of metrics. The resulting high values of correlation can then lead to false confidence in the reliability of metrics. Once the outliers are removed, the gap between correlation of BLEU and other metrics (e.g. CHRF, Y1S1-1 and ESIM) becomes wider. In the worst case scenario, outliers introduce a high correlation when there is no association between metric and human scores for the rest of the systems. Thus, future evaluations should also measure correlations after removing outlier systems.
Finally, the same value of correlation coefficient can describe different patterns of errors. Any single number is not adequate to describe the data, and visualising metric scores against human scores is the best way to gain insights into metric reliability. This could be done with scatter plots (e.g. Figure 3a) for each language pair, or Figure 5, which compresses this information into one graph.
Metrics are commonly used to compare two systems, and accordingly we have also investigated
the real meaning encoded by a difference in metric score, in terms of what this indicates about human judgements of the two systems. Most published work report BLEU differences of 1-2 points, however at this level we show this magnitude of difference only corresponds to true improvements in quality as judged by humans about half the time. Although our analysis assumes the Direct Assessment human evaluation method to be a gold standard despite its shortcomings, our analysis does suggest that the current rule of thumb for publishing empirical improvements based on small BLEU differences has little meaning.
Overall, this paper adds to the case for retiring BLEU as the de facto standard metric, and instead using other metrics such as CHRF, YIa1-1, or ESIM in its place. They are more powerful in assessing empirical improvements. However, human evaluation must always be the gold standard, and for continuing improvement in translation, to establish significant improvements over prior work, all automatic metrics make for inadequate substitutes.
To summarise, our key recommendations are:
- When evaluating metrics, use the technique outlined in Section 4.2 to remove outliers before computing Pearson's  $r$ .
- When evaluating MT systems, stop using BLEU or TER for evaluation of MT, and instead use CHRF, YIsI-1, or ESIM;
- Stop using small changes in evaluation metrics as the sole basis to draw important empirical conclusions, and make sure these are supported by manual evaluation.
We are grateful to the anonymous reviewers for their comments and valuable suggestions. This work was supported in part by the Australian Research Council.
Francis J Anscombe. 1973. Graphs in statistical analysis. The American Statistician, 27(1):17-21.
Vic Barnett and Toby Lewis. 1974. Outliers in Statistical Data. Wiley.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.
Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleks Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Néveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, USA.
Marina Fomicheva and Lucia Specia. 2019. Taking MT evaluation metrics to extremes: Beyond correlation with human judgments. Computational Linguistics, 45(3):515-558.
Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172-176, Doha, Qatar.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3-30.
Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266-274, Baltimore, Maryland, USA. Association for Computational Linguistics.
Boris Iglewicz and David Caster Hoaglin. 1993. How to detect and handle outliers, volume 16. Asq Press.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain.
Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764-766.
Chi-kiu Lo. 2019. YiSi — a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513, Florence, Italy.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799-2808, Florence, Italy.
Jason W Osborne and Amy Overbay. 2004. The power of outliers (and why researchers should always check for them). Practical Assessment, Research & Evaluation, 9(6):1-12.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, USA.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. Bleu: a method for automatic evaluation of machine translation. In Proceedings of
the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, USA.
Maja Popović. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393-401.
Peter J Rousseeuw and Mia Hubert. 2011. Robust statistics for outlier detection. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(1):73-79.
Aaron Smith, Christian Hardmeier, and Joerg Tiedemann. 2016. Climbing mont BLEU: The strange world of reachable high-BLEU translations. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 269-281.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas, pages 223-231.
Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Computational Linguistics and Intelligent Text Processing, pages 341-351, Berlin, Heidelberg. Springer Berlin Heidelberg.
Rand Wilcox. 2004. Inferences based on a skipped correlation coefficient. Journal of Applied Statistics, 31(2):131-143.
A The effect of removing outlier systems on the results of the WMT 19 metrics task
n	de-cs	de-fr	fr-de
All 11	-out 10	All 11	All 10	All 7
BEER	0.978	0.976	0.941	0.848	0.794
BLEU	0.941	0.922	0.891	0.864	0.821
CDER	0.864	0.734	0.949	0.852	0.794
CHARACTER	0.965	0.959	0.928	0.849	0.848
CHRF	0.974	0.970	0.931	0.864	0.796
CHRF+	0.972	0.967	0.936	0.848	0.785
EED	0.982	0.984	0.940	0.851	0.792
ESIM	0.980	0.986	0.950	0.942	0.825
HLEPORA_BASELINE	0.941	0.903	0.814	-	-
HLEPORB_BASELINE	0.959	0.951	0.814	-	-
NIST	0.954	0.944	0.916	0.862	0.800
PER	0.875	0.757	0.857	0.899	0.427
SACREBLE-BLEU	0.869	0.742	0.891	0.869	0.846
SACREBLE-CHRF	0.975	0.980	0.952	0.882	0.815
TER	0.890	0.787	0.956	0.895	0.673
WER	0.872	0.749	0.956	0.894	0.657
YISI-0	0.978	0.972	0.952	0.820	0.836
YISI-1	0.973	0.980	0.969	0.908	0.846
YISI-1_SRL	-	-	-	0.912	0.814
Source-based metrics:
IBM1-MORPHEME	0.355	0.009	0.509	0.625	0.357
IBM1-POS4GRAM	-	-	0.085	0.478	0.719
YISI-2	0.606	0.122	0.721	0.530	0.066
Table 3: Pearson correlation of metrics for the language pairs that do not involve English. For language pairs that contain outlier systems, we also show correlation after removing outlier systems. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold.
n	de-en	fi-en	gu-en	kk-en	lt-en	ru-en	zh-en
All 16	-out 15	All 12	All 11	-out 10	All 11	-out 9	All 11	-out 10	All 14	-out 13	All 15	-out 13
BEER	0.906	0.852	0.993	0.952	0.982	0.986	0.930	0.947	0.948	0.915	0.819	0.942	0.806
BERTR	0.926	0.897	0.984	0.938	0.995	0.990	0.829	0.948	0.959	0.971	0.933	0.974	0.911
BLEU	0.849	0.770	0.982	0.834	0.975	0.946	0.912	0.961	0.980	0.879	0.830	0.899	0.807
CDER	0.890	0.827	0.988	0.876	0.975	0.967	0.843	0.975	0.981	0.892	0.875	0.917	0.847
CHARACTER	0.898	0.852	0.990	0.922	0.978	0.953	0.833	0.955	0.963	0.923	0.828	0.943	0.845
CHRF	0.917	0.862	0.992	0.955	0.962	0.978	0.775	0.940	0.933	0.945	0.876	0.956	0.841
CHRF+	0.916	0.860	0.992	0.947	0.961	0.976	0.769	0.940	0.934	0.945	0.878	0.956	0.851
EED	0.903	0.853	0.994	0.976	0.988	0.980	0.779	0.929	0.930	0.950	0.872	0.949	0.840
ESIM	0.941	0.896	0.971	0.885	0.986	0.986	0.945	0.989	0.990	0.968	0.946	0.988	0.961
HLEPORA_BASELINE	0.975	0.855	0.947	0.879
HLEPORB_BASELINE	0.975	0.855	0.906	0.930	0.947	0.879
METEOR++_2.0(SYNTAX)	0.887	0.844	0.995	0.909	0.939	0.974	0.859	0.928	0.935	0.950	0.878	0.948	0.836
METEOR++_2.0(SYNTAX+COPY)	0.896	0.850	0.995	0.900	0.930	0.971	0.871	0.927	0.931	0.952	0.890	0.952	0.841
NIST	0.813	0.705	0.986	0.930	0.985	0.942	0.837	0.944	0.963	0.925	0.878	0.921	0.722
PER	0.883	0.808	0.991	0.910	0.948	0.737	0.533	0.947	0.933	0.922	0.880	0.952	0.884
PREP	0.575	0.452	0.614	0.773	0.967	0.776	0.817	0.494	0.397	0.782	0.685	0.592	0.111
SACREBLE-BLEU	0.813	0.794	0.985	0.834	0.975	0.946	0.912	0.955	0.967	0.873	0.813	0.903	0.807
SACREBLE-CHRF	0.910	0.852	0.990	0.952	0.937	0.969	0.750	0.935	0.923	0.919	0.874	0.955	0.846
TER	0.874	0.812	0.984	0.890	0.947	0.799	0.566	0.960	0.975	0.917	0.896	0.840	0.717
WER	0.863	0.803	0.983	0.861	0.926	0.793	0.579	0.961	0.981	0.911	0.885	0.820	0.716
WMDO	0.872	0.857	0.987	0.983	0.981	0.998	0.953	0.900	0.923	0.942	0.844	0.943	0.851
Y1SI-0	0.902	0.847	0.993	0.993	0.990	0.991	0.876	0.927	0.933	0.958	0.889	0.937	0.782
Y1SI-1	0.949	0.914	0.989	0.924	0.997	0.994	0.920	0.981	0.978	0.979	0.947	0.979	0.899
Y1SI-1_SRL	0.950	0.916	0.989	0.918	0.998	0.994	0.917	0.983	0.981	0.978	0.943	0.977	0.897
Source-based metrics:
IBM1-MORPHEME	0.345	0.223	0.740	0.487	0.638
IBM1-POS4GRAM	0.339	0.137
LASIM	0.247	0.334	0.310	0.260
LP	0.474	0.279	0.488	0.168
UNI	0.846	0.809	0.930	0.805	0.666
UNI+	0.850	0.805	0.924	0.808	0.669
Y1SI-2	0.796	0.612	0.642	0.566	0.820	0.324	0.662	0.442	0.346	0.339	0.708	0.940	0.622
Y1SI-2_SRL	0.804	0.630	0.947	0.675
Table 4: Pearson correlation of metrics for the to-English language pairs. For language pairs that contain outlier systems, we also show correlation after removing outlier systems. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold.
n	en-cs
All
11	en-de
22	en-fi
20	en-gu
12	en-kk
11	en-kt
11	en-lt
9	en-ru
12	en-zh
12
BEER	0.990	0.983	0.869	0.989	0.978	0.829	0.971	0.826	0.982
BLEU	0.897	0.921	0.419	0.969	0.943	0.737	0.852	0.576	0.989
CDER	0.985	0.973	0.849	0.978	0.957	0.840	0.927	0.668	0.985
CHARACTER	0.994	0.986	0.886	0.968	0.939	0.910	0.936	0.895	0.954
CHRF	0.990	0.979	0.881	0.986	0.972	0.841	0.972	0.900	0.981
CHRF+	0.991	0.981	0.883	0.986	0.970	0.848	0.974	0.907	0.982
EED	0.993	0.985	0.894	0.987	0.978	0.897	0.979	0.883	0.975
ESIM	-	0.991	0.928	0.957	0.926	-	0.980	0.900	0.989
HLEPORA_BASELINE	-	-	-	-	-	0.841	0.968	0.852	-
HLEPORB_BASELINE	-	-	-	-	-	0.841	0.968	0.852	0.980
NIST	0.896	0.321	0.246	0.971	0.936	0.786	0.930	0.611	0.993
PER	0.976	0.970	0.815	0.982	0.961	0.839	0.921	0.545	0.985
SACREBLE-BLEU	0.994	0.969	0.806	0.966	0.939	0.736	0.852	0.576	0.986
SACREBLE-CHRF	0.983	0.976	0.874	0.980	0.958	0.841	0.967	0.840	0.966
TER	0.980	0.969	0.841	0.981	0.960	0.865	0.940	0.547	0.994
WER	0.982	0.966	0.831	0.980	0.958	0.861	0.939	0.525	0.991
Y1SI-0	0.992	0.985	0.869	0.987	0.977	0.863	0.974	0.840	0.974
Y1SI-1	0.962	0.991	0.917	0.971	0.937	0.909	0.985	0.892	0.963
Y1SI-1_SRL	-	0.991	0.917	-	-	-	-	-	-
Source-based metrics:
IBM1-MORPHEME	0.871	0.870	0.198	0.084	0.254	-	-	0.810	-
IBM1-POS4GRAM	-	0.393	0.449	-	-	-	-	-	-
LASIM	-	0.871	0.007	-	-	-	-	-	0.823
LP	-	0.569	0.558	-	-	-	-	-	0.661
UNI	0.028	0.841	0.251	0.907	0.808	-	-	-	0.919
UNI+	-	-	-	-	-	-	-	-	0.918
USFD	-	0.224	0.301	-	-	-	-	-	0.857
USFD-TL	-	0.091	0.212	-	-	-	-	-	0.771
Y1SI-2	0.324	0.924	0.014	0.696	0.478	0.314	0.339	0.685	0.055
Y1SI-2_SRL	-	0.936	0.155	-	-	-	-	-	-
Table 5: Correlation of metrics for the from-English language pairs. For language pairs that contain outlier systems, we also show correlation after removing outlier systems. Values in bold indicate that the metric is not significantly outperformed by any other metric under the Williams Test.