SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 8 shows results for the scoring methods described in section 3.2 applied to the Massive configuration. Aggregating token probabilities using statistics other than mean gives small gains on some languages, but hurts on average. Regularizing with MC-dropout or subwords (SP-norm) leads to significant gains in some cases, with a slight overall increase over mean for SP-norm. We tuned confidence thresholds on WMT18 Metrics task data using a grid of 16 log-probability points in  $[-3,0]$ , which yielded optimal thresholds  $(-1,-0.6)$ . This
produced our best overall result, with systematic gains on  $en - xx$  pairs.
In this section we analyze various aspects of metric performance, confining our attention to the Massive model with mean scoring for consistency.
6.1 Performance across conditions
Subset	Avg
All	0.883
All - gu	0.893
Source-based evaluation	0.858
Source-based - gu	0.883
Reference-based evaluation	0.901
Reference-based - gu	0.901
Corpus ≥1M	0.839
Corpus <1M	0.924
No data	0.741
Different languages have different relations to our model, to the systems participating in the WMT task, and to the human scoring procedure used in the WMT19 data. Table 9 shows results for various conditions. Removing the language (gu) for which we have no training data improves average correlation substantially. The human evaluations for out-of-English language pairs involve comparing MT output to the source text; the evaluations for remaining pairs involve comparing it to reference translations. We see no boost from the language pairs for which source-based human evaluation was used (matching our setting), and in fact do somewhat worse on these pairs than the others, on average. Finally, we achieve better performance for lower-resource (< 1M parallel segments) language pairs than higher-resource pairs (with respect to the Prism-39 corpora), but poor average performance on the pairs (en-gu/gu-en) for which we had no training data.
6.2 Pairwise comparisons
Correlation statistics give an overall picture of metric performance, but do not directly reflect the frequent use case of deciding which of two systems is better. To measure this, we examined whether our metric agrees with human pairwise ranking decisions over all pairs of systems. Following (Mathur
et al., 2020), we apply the Wilcoxon ranksum test and paired t-test to detect when such decisions are significant according to human and metric scores respectively.
Table 9: Average Correlation for different subsets of languages.
Metric	Human-S	Human-NS
C (↑)	IC (↓)	NS	C (↑)	IC (↓)	NS
All Systems
BLEU	768	37	80	126	71	70
Prism	778	61	46	136	93	38
en-xx Systems
BLEU	411	25	53	38	25	26
Prism	421	36	32	39	29	21
xx-en Systems
BLEU	285	8	20	67	40	33
Prism	277	23	13	74	53	13
xx-yy Systems
BLEU	72	4	7	21	6	11
Prism	80	2	1	23	11	4
Table 10: WMT19 pairwise system level comparisons using the Massive configuration: Human-NS and Human-S means insignificant and significant differences according to human scores; C and IC stands for Correct and Incorrect ranking according to metric and human scores; NS represents insignificant differences according to the metric scores.
Table 10 shows ranking performance for Prism compared to BLEU, categorized according to language pair grouping. The general pattern across all groupings is that Prism is more decisive: it makes more significant decisions than BLEU, leading to higher rates of both correct and incorrect rankings. Among the 885 system pairs (across all languages) that are considered significantly different according to human judgment, Prism correctly ranks  $88\%$  with significantly different scores, compared to  $87\%$  for BLEU.
6.3 Quality of the evaluating model
How good is our multilingual MT system compared to the systems under evaluation? We generated translations of the test text for a subset of languages and compared the quality of the generated system outputs using BLEU. Figure 1 shows that our evaluating model achieves worse BLEU scores than many of the systems under evaluation, ranking around the median for most language pairs. Although Table 5 provides evidence that stronger systems produce better metrics, clearly it is not necessary to be among the top-ranked systems in order to generate a signal that is approximately as
reliable as BLEU.4
Figure 1: Quality across language pairs.
6.4 Proximity Bias
A potential pitfall in peer evaluation is bias toward one or more of the systems under evaluation. Clearly, the evaluating system will prefer its own output—how far from an evaluated system does it have to be in order to judge it fairly? Lacking access to the systems in the WMT19 task, we measure proximity using cross-BLEU score (using one output as hypothesis and the other one as reference translation) between the system output and the output generated by our Prism model. In the presence of bias, we would expect the metric to result in higher ranking for closer systems and lower ranking for farther systems (relative to human scores).
Figure 2: Relative Ranking of the closest and farthest systems under evaluation to the Prism system as measured by cross-BLEU.
Figure 2 shows the relative ranking of the closest and the farthest system to Prism (relative to human). Since the model makes mistakes in both
directions—ranks closest and farthest system both higher and lower than human—there is no evidence from this analysis that it exhibits a strong bias in favour of systems whose outputs are closer to its own. A potential explanation is that it is sufficiently far from most of the evaluated systems due to its multilingual training corpus. To verify this, we computed the average cross-BLEU for each evaluated system (relative to all others), and compared it to the same quantity for our system. Figure 3 shows that we are indeed an outlier system for most language pairs. The systems with lower cross-BLEU than Prism are mostly online or rule-based systems.[5]
Figure 3: Average Cross-BLEU for all evaluated systems and Prism.
6.5 Test-set Size
Size	Bleu	Prism
100	0.735	0.720
200	0.783	0.771
400	0.804	0.784
800	0.828	0.807
Table 11: Average correlations versus test-set size for the language pairs from Figure 2.
In principle, a major advantage of reference-free evaluation is that it can make use of arbitrarily large test sets, being constrained only by the amount of source-language data in the domain of interest. We hypothesize that this will improve metric performance by reducing sampling error. To test this hypothesis in the absence of larger human-scored test sets for WMT19, we sampled subsets of various sizes and measured average correlation. As shown
<sup>4</sup>It would be interesting to try to characterize the relation between system quality and metric strength more precisely, but in the absence of human judgments of our output quality, any such picture we could currently draw would be clouded by metric noise.
For Kazakh (kk), Prism-39 includes the WMT-15 dataset, resulting in higher cross-BLEU compared to other language pairs.
in Table 11, we observe a steady increase with test-size size. This provides persuasive, though not definitive, evidence that test sets beyond the scale of WMT19 would yield further improvements in accuracy for both metrics, a setting that would be more feasible for Prism than BLEU. Full curves are plotted in Figure 4 (See Appendix C).
In this paper, we have shed some light on the remarkable finding by Thompson and Post (2020) that a multilingual model trained on a large (but not enormous) general-domain corpus can be highly effective as an MT metric when used to score the outputs of other MT systems in the absence of reference translations. By scaling up the model and making small adjustments to tagging and scoring, we improve over the original results and achieve approximate parity with BLEU in correlation with human judgment on WMT19 data. We argue that this metric is a useful complement to reference-based metrics—including ones that are significantly more powerful than BLEU—due to its flexibility; and we provide evidence that scoring reliability can be further improved by using larger source-side-only test sets.
We find that the major determinant of success in peer evaluation is the quality of the evaluating model. However, there is no hard requirement that it be better than the models under evaluation: surprisingly, it can correctly rank models that outperform it on average. If we abstract away from quality, performance does not appear to be highly sensitive to the domain or the multilingual versus bilingual nature of the training corpus. Taken together, these results have the important practical implication that a single multilingual system such as ours could be broadly applicable for evaluating systems in a large number of language pairs (706 in our case), at different quality levels, and across a wide range of domains. In future work, we look forward to probing these results further, and determining whether alternative architectures or loss functions might be valuable in specializing an MT model for evaluating its peers.
We thank Julia Kreutzer, Ciprian Chelba, Aditya Siddhant, and the anonymous reviewers for their helpful and constructive comments.
Mikel Artetxe and Holger Schwenk. 2019. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197-3203, Florence, Italy. Association for Computational Linguistics.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 315-321, Geneva, Switzerland. COLING.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274, Online. Association for Computational Linguistics.
Ciprian Chelba, Junpei Zhou, Hideto Kazawa, Jeff Klingner, Mengmeng Niu, et al. 2020. Data troubles in sentence level confidence estimation for machine translation. arXiv preprint arXiv:2010.13856.
Kawin Ethayarajh and Dorsa Sadigh. 2020. BLEU neighbors: A reference-less approach to automatic evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 40-50, Online. Association for Computational Linguistics.
M. Fomicheva, Shuo Sun, L. Yankovskaya, F. Blain, Francisco Guzmán, M. Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555.
Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1-10, Florence, Italy. Association for Computational Linguistics.
Markus Freitag, George Foster, David Grangier, and Colin Cherry. 2020a. Human-paraphrased references improve neural machine translation.
Markus Freitag, David Grangier, and Isaac Caswell. 2020b. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Y Gal and Z Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In 33rd International Conference on Machine Learning, ICML 2016, volume 3, pages 1651-1660.
Larry Hedges and Ingram Olkin. 2014. Statistical methods for meta-analysis. Academic press.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Chi-kiu Lo. 2019. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513, Florence, Italy. Association for Computational Linguistics.
Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy. Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984-4997, Online. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707, Online. Association for Computational Linguistics.
Courtney Naples, Keisuke Sakaguchi, and Joel Tetreault. 2016. There's no comparison: Referenceless evaluation metrics in grammatical error correction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2109-2115, Austin, Texas. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung-Cheng Chiu, et al. 2019. Lingvo: a modular and scalable framework for sequence-to-sequence modeling. arXiv preprint arXiv:1902.08295.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazha-gan, and Yonghui Wu. 2020. Leveraging monolingual data with self-supervision for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2827-2835, Online. Association for Computational Linguistics.
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2430-2441, Online. Association for Computational Linguistics.
Lucia Specia and Kashif Shah. 2018. Machine Translation Quality Estimation: Applications and Future Perspectives. Springer International Publishing, Cham.
Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90-121, Online. Association for Computational Linguistics.
Antonio Toral. 2020. Reassessing claims of human parity and super-human performance in machine translation at WMT 2019. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 185-194, Lisboa, Portugal. European Association for Machine Translation.
Evan James Williams. 1959. Regression Analysis, volume 14. Wiley, New York.
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan
Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5370-5378. International Joint Conferences on Artificial Intelligence Organization.
Elizaveta Yankovskaya, Andre Tattar, and Mark Fishel. 2019. Quality estimation and translation metrics via pre-trained word and sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 101-105, Florence, Italy. Association for Computational Linguistics.
Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. 2016. The neural noisy channel. arXiv preprint arXiv:1611.02554.
Boliang Zhang, Ajay Nagesh, and Kevin Knight. 2020. Parallel corpus filtering via pre-trained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8545-8554, Online. Association for Computational Linguistics.
Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1656–1671, Online. Association for Computational Linguistics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Linguistics.
A Outlier Systems
lang	Outliers
de-cs	CAiRE.6949
de-en	online-X.0
de-fr	-
en-cs	-
en-de	online-X.0, en_de_task.6790
en-fi	apertium-fin-eng-unconstrained-en-fi.6448
en-gu	-
en-kk	NICT.6550, DBMS-KU_ENKK.6730
en-lt	-
en-ru	NICT.6563
en-zh	-
fi-en	-
fr-de	MSRA.MADL.6893, eTranslation.6262, online-X.0
gu-en	Ju_Saarland.6525
kk-en	UMD.6736, DBMS-KU_KKEN.6726
lt-en	online-X.0
ru-en	NICT.6561
zh-en	online-X.0, Apprentice-c.6706
B WMT 2019 System-Level results for all language pairs
Table 12: Outlier systems using MAD filtering in WMT19.
Metric	Avg	en-cs	en-de	en-fi	en-gu	en-kk	en-lt	en-ru	en-zh
BLEU	0.911	0.994	0.806	0.939	0.737	0.575	0.986	0.946	0.802
CHRF	0.933	0.983	0.871	0.964	0.843	0.829	0.969	0.989	0.799
UNI+	0.808	-	-	-	-	-	-	0.746	-
Yisi-2	0.487	0.324	-	0.478	0.314	0.685	0.055	0.134	-0.097
Prism	0.861	0.865	0.754	0.858	0.444	0.789	0.908	0.903	0.793
Prism-trg2xx	0.853	0.867	0.717	0.876	0.365	0.811	0.936	0.902	0.778
Prism-src2xx	0.858	0.871	0.730	0.878	0.400	0.813	0.939	0.901	0.789
Metric	de-en	fi-en	gu-en	kk-en	lt-en	ru-en	zh-en	de-cs	de-fr	fr-de
BLEU	0.794	0.985	0.975	0.912	0.967	0.812	0.808	0.743	0.891	0.846
CHRF	0.852	0.991	0.946	0.836	0.930	0.877	0.831	0.981	0.957	0.833
UNI+	0.805	0.924	-	-	-	0.669	-	-	-	-
Yisi-2	0.612	0.642	0.820	0.662	0.346	0.708	0.622	0.122	0.721	0.62
Prism	0.829	0.941	0.915	0.724	0.985	0.769	0.826	0.987	0.889	0.269
Prism-trg2xx	0.798	0.943	0.911	0.683	0.979	0.752	0.830	0.989	0.882	0.212
Prism-src2xx	0.796	0.942	0.893	0.709	0.978	0.739	0.828	0.991	0.882	0.203
Table 13: Baseline results. All numbers are system-level correlations. Avg gives averages over all language pairs.
Data	Avg	en-cs	en-de	en-fi	en-gu	en-kk	en-lt	en-ru	en-zh
Prism-39	0.858	0.871	0.730	0.878	0.400	0.813	0.939	0.901	0.789
WMT-15	0.840	0.825	0.530	0.815	0.423	0.909	0.914	0.845	0.774
Prism-13	0.863	0.869	0.779	0.891	0.379	0.820	0.925	0.901	0.795
Prism-39 + WMT-15	0.867	0.862	0.653	0.854	0.446	0.860	0.932	0.880	0.789
Adding monolingual data
Prism-39 + Wiki-39	0.832	0.839	0.649	0.854	0.446	0.823	0.917	0.877	0.757
WMT-15 + WMT-15-Mono	0.869	0.855	0.646	0.826	0.848	0.913	0.940	0.867	0.793
Prism-39 + WMT-15-Mono	0.851	0.869	0.766	0.871	0.487	0.850	0.945	0.892	0.754
Bilingual Models	0.726	0.695	0.867	0.769
Data	de-en	fi-en	gu-en	kk-en	lt-en	ru-en	zh-en	de-cs	de-fr	fr-de
Prism-39	0.796	0.942	0.893	0.709	0.978	0.739	0.828	0.991	0.882	0.203
WMT-15	0.815	0.954	0.918	0.509	0.986	0.841	0.835	0.970	0.851	0.116
Prism-13	0.845	0.943	0.911	0.745	0.983	0.749	0.836	0.985	0.863	0.124
Prism-39 + WMT-15	0.802	0.950	0.921	0.734	0.986	0.851	0.810	0.993	0.881	0.170
Adding monolingual data
Prism-39 + Wiki-39	0.742	0.934	0.907	0.684	0.975	0.680	0.836	0.982	0.822	0.117
WMT-15 + WMT-15-Mono	0.842	0.956	0.976	0.584	0.987	0.838	0.824	0.903	0.890	0.238
Prism-39 + WMT-15-Mono	0.803	0.944	0.952	0.680	0.952	0.722	0.754	0.972	0.875	0.233
Bilingual Models	0.801	0.862	0.650	0.826
Table 14: Effect of training data. Significant improvement over baseline "Prism-39" systems are underlined.
Metric	Avg	en-cs	en-de	en-fi	en-gu	en-kk	en-lt	en-ru	en-zh
BLEU	0.911	0.994	0.806	0.939	0.737	0.575	0.986	0.946	0.802
Big	0.808	0.791	0.541	0.833	0.381	0.785	0.898	0.861	0.698
Prism	0.858	0.871	0.730	0.878	0.400	0.813	0.939	0.901	0.789
Massive	0.883	0.900	0.819	0.899	0.423	0.820	0.953	0.923	0.819
Metric	de-en	fi-en	gu-en	kk-en	lt-en	ru-en	zh-en	de-CS	de-fr	fr-de
BLEU	0.794	0.985	0.975	0.912	0.967	0.812	0.808	0.743	0.891	0.846
Big	0.702	0.926	0.891	0.649	0.970	0.640	0.819	0.989	0.827	0.040
Prism	0.796	0.942	0.893	0.709	0.978	0.739	0.828	0.991	0.882	0.203
Massive	0.840	0.948	0.906	0.751	0.981	0.789	0.834	0.991	0.906	0.301
Table 15: Effect of Model capacity.
Aggregation Method	Avg	en-cs	en-de	en-fi	en-gu	en-kk	en-lt	en-ru	en-zh
Mean	0.883	0.900	0.819	0.899	0.423	0.820	0.953	0.923	0.819
Std-dev	0.882	0.913	0.778	0.900	0.448	0.755	0.950	0.929	0.780
Median	0.870	0.849	0.868	0.884	0.408	0.862	0.945	0.908	0.849
Min	0.872	0.925	0.765	0.907	0.489	0.623	0.945	0.939	0.722
MC-dropout (Mean)	0.877	0.936	0.826	0.904	0.432	0.699	0.929	0.940	0.881
MC-dropout (Std-dev)	0.855	0.890	0.800	0.894	0.417	0.809	0.947	0.920	0.803
SP-Norm (Mean)	0.884	0.903	0.814	0.895	0.407	0.872	0.949	0.921	0.839
Confidence threshold	0.886	0.941	0.828	0.966	0.569	0.696	0.987	0.940	0.774
Aggregation Method	de-en	fi-en	gu-en	kk-en	lt-en	ru-en	zh-en	de-cs	de-fr	fr-de
Mean	0.840	0.948	0.906	0.751	0.981	0.789	0.834	0.991	0.906	0.301
Std-dev	0.831	0.970	0.946	0.712	0.987	0.769	0.809	0.989	0.907	0.276
Median	0.851	0.895	0.885	0.761	0.970	0.788	0.870	0.973	0.888	0.341
Min	0.829	0.973	0.902	0.644	0.990	0.757	0.794	0.978	0.900	0.280
MC-dropout (Mean)	0.813	0.956	0.485	0.189	0.941	0.834	0.821	0.974	0.923	0.330
MC-dropout (Std-dev)	0.837	0.946	0.903	0.743	0.981	0.786	0.837	0.992	0.901	0.286
SP-Norm (Mean)	0.834	0.948	0.902	0.795	0.980	0.801	0.833	0.990	0.906	0.318
Confidence threshold	0.823	0.930	0.906	0.670	0.970	0.716	0.765	0.971	0.935	0.386
Table 16: Scoring methods. Significant improvements over baseline Mean systems are underlined.
C Correlation versus test-set size
Figure 4: Correlation as test-size size increases, for BLEU (left panel) and Prism (right panel). Each point is the average correlation over 10 random draws of subsets of the given size.