SlowGuess/NAACL_2021_OCR huggingface dataset:
Sweta Agrawal<sup>1</sup>, George Foster<sup>2</sup>, Markus Freitag<sup>2</sup>, Colin Cherry<sup>2</sup>
sweagraph@umd.edu,{fosterg, freitag, colincherry}@google.com
Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual model can achieve state of the art results when used as a reference-free metric. We experiment with various modifications to this model, and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weaknesses of the approach, and find that it is surprisingly robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities.
Traditional automatic metrics for machine translation (MT), such as BLEU (Papineni et al., 2002), score MT output by comparing it to one or more reference translations. This has several disadvantages. First, high-quality reference translations are expensive to create. This means that in practice, evaluation is usually carried out with relatively small, carefully curated test corpora. The need for careful preparation limits the number of domains for which an MT system can be conveniently assessed, and small test-set sizes can make it difficult to draw robust conclusions (Card et al., 2020). Second, enshrining ground truth in a small number of references (usually just one) is inherently problematic, since valid translations can vary along many dimensions; Freitag et al. (2020b) demonstrate that different (correct) references for the same test set can result in different system rankings according to the same reference-based metric. Finally, scoring the similarity between an MT hypothesis and a reference translation involves recognizing the extent to which they are mutual paraphrases. When gross discrepancies exist, this is a relatively easy problem for which surface-level metrics can provide a reliable signal, but capturing the subtle errors typical
of high-quality MT is more difficult, and it is not clear whether it is substantially easier than scoring the similarity between texts in different languages.
These problems can be avoided by looking only at the source text when assessing MT output. There is evidence that this is the best practice for human evaluation (Toral, 2020). Moreover, it has recently been investigated for automatic metrics as well (Yankovskaya et al., 2019; Lo, 2019; Zhao et al., 2020; Ma et al., 2019). Such reference-free metrics are flexible and scalable, but since they are essentially performing the same task as an MT model, they raise a circularity concern: if we can reliably score MT output, why wouldn't we use the scoring model to produce better output? One answer to this is practical: the scoring model might be too large to deploy, or it might not easily support efficient inference (Yu et al., 2016). A more interesting answer is that a scoring model could be set up to provide a signal that is complementary to the systems under evaluation. That is, it might be capable of correctly ranking competing MT hypotheses even when its own preferred hypothesis is worse on average than those of the systems it is evaluating. In our experiments we find that this can indeed be the case.
In recent work, Thompson and Post (2020) showed that a single multilingual MT model trained on 39 languages can achieve excellent paraphrase recognition when used in zero-shot mode to compare MT output with reference sentences in the same language. On the WMT 2019 metrics task, their method (Prism) beat or tied all previous reference-based metrics on all languages. Although it was not the main focus of their work, Prism achieved a new state-of-the-art as a reference-free metric, simply scoring target given source text using an MT model, in a post-competition comparison to the 2019 "Quality Estimation as a metric" shared task (Ma et al., 2019).
<sup>1</sup>Except Gujarati, which was absent from their training corpus.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1158-1171 June 6-11, 2021. ©2021 Association for Computational Linguistics
Our aim in this paper is to characterize the conditions under which the Prism approach—using one MT system to perform peer evaluation on other systems—can be successful: what properties does the evaluating system need to have, how powerful should it be, and how close can it be to the systems under evaluation? We focus on system-level evaluation, which we believe is the most compelling use case for reference-free methods, targeting a broad characterization that complements the potentially more precise picture furnished by reference-based metrics for a specific test corpus. We first replicate the correlation with human judgment results from Thompson and Post (2020) on WMT 2019, using the same corpora and architecture. Next, we examine several alternative design decisions in an attempt to improve Prism and further our understanding. These include the effects of varying training corpora (domain, number of languages, use of monolingual data); model capacity (scaling up and down from the original architecture); and different methods for regularizing token-level probabilities (Monte-Carlo dropout, subword sampling) and for combining them into system-level scores (summary statistics over tokens, confidence thresholds over sentences). Finally, we analyze the results of our best model, measuring how its performance depends on various factors: language pair and human-judgment methodology, output quality, proximity to the systems under evaluation, and size of the test set.
We demonstrate improvements over the original Prism metric due to model capacity and different methods for combining probabilities; surprisingly, we find little gain from adjusting the domain or languages in the original multilingual corpus (although we show that a competition-grade English-German system outperforms the generic multilingual system). We find that the evaluating MT system's output quality is generally correlated with its performance as a metric, although we corroborate the surprising finding from Thompson and Post (2020) that it is not necessary to be the best—our system is middle-of-the-road or worse according to BLEU across most WMT 2019 languages. We measure the proximity between our system and the systems under evaluation and find no evidence that this is a source of bias. Despite using no references, our model achieves approximate parity with BLEU both in system-level correlation with human judgment, and when used for pairwise comparisons.
Reference-free evaluation is widely used for many NLP tasks such as grammatical error correction (Napoles et al., 2016), dialog (Sinha et al., 2020; Mehri and Eskenazi, 2020) and text generation (Ethayarajh and Sadigh, 2020). There has been recent interest in reference-free evaluation for MT, which was a joint track between the WMT 2019 metrics task (Ma et al., 2019) and quality estimation task (Fonseca et al., 2019). Reference-free metrics competed head-to-head with standard metrics, and generally did worse. However, the results from the best reference-free systems, UNI+ (Yankovskaya et al., 2019) and YiSi-2 (Lo, 2019) were surprisingly close to the standard metric scores on the language pairs for which they were evaluated.
UNI+ computes word-level embeddings for source and MT output sentences using pre-trained multilingual BERT and LASER (Artetxe and Schwenk, 2019) models, then feeds averaged vectors to a neural classifier trained to predict human scores from previous MT metrics tasks. YiSi-2 is similar, except that it works in an unsupervised fashion, computing similarities between mBERT embeddings for aligned source and target words, and returning an F-measure statistic. In more recent work, Zhao et al. (2020) adopt a similar approach based on mBERT, aligning representations from multilingual embedding spaces before computing distances with MoverScore (Zhao et al., 2019), and adding a GPT-based target-side language model. The current state-of-the-art in reference-free evaluation for MT is represented by the Prism approach (Thompson and Post, 2020) which we extend here.
It is worth distinguishing reference-free evaluation from two related tasks that share formal similarities. The first is quality or confidence estimation (Blatz et al., 2004; Specia and Shah, 2018; Chelba et al., 2020), which aims to score the fitness of MT output for a downstream application. This is typically supervised, although a recent approach (Fomicheva et al., 2020) dispenses with the need to learn from human annotations, as do most of the approaches we study in this paper. Quality estimation is most usefully applied at the sentence level, and it can make use of powerful "glass-box" features which capture the internals of an MT system. In contrast, reference-free evaluation is most naturally applied at the system (test-set) level, and ideally should make no assumptions about the sys
tems under evaluation. The second task is parallelcorpus mining (Zhang et al., 2020; Yang et al., 2019), which aims to identify valid translations at various levels of granularity. Its scoring aspect is similar to reference-free evaluation, but it is applied to a different input distribution, attempting to identify human-generated translation pairs rather than scoring MT outputs for a given human-generated source text.
We aim to generate a quality score  $s(X,Y) = \sum_{x,y} s(x,y)$  for source and target texts  $X, Y$  which consist of segment (nominally, sentence) pairs  $x, y$ . We assume no document or ordering information among segments, and do not directly evaluate scores for individual segment pairs. All methods we consider make use of token-level log-probabilities from a standard autoregressive neural MT system:  $\log p(y_t | y_{<t}, x)$ , where  $y = y_1 \ldots y_T$ . We experimented with reverse probabilities  $p(x|y)$ , but like Thompson and Post (2020) found these gave no advantage, and do not include them in our reported results. The following sections describe our model architecture, scoring techniques, and evaluation methodology.
Our baseline NMT model uses a standard Transformer architecture identical to that of Thompson and Post (2020) (up to toolkit differences), trained on the same multilingual corpus. To encourage language-agnostic encoder representations for zero-shot scoring, the baseline uses target-language tags at the beginning of each target sentence (Johnson et al., 2017). Since we do not require such representations for reference-free evaluation, we also tried introducing the tags earlier, at the beginning of each source sentence. We vary training corpora and model capacity as described in section 4.1, but otherwise make no changes to the model.
3.2 Scoring
We investigated various techniques for deriving segment-level scores  $s(x, y)$ : regularization, different methods for aggregating token-level probabilities, and segment-level confidence thresholds.
To obtain smoother scores, we used Monte-Carlo dropout (Gal and Ghahramani, 2016) and subword
regularization (Kudo, 2018). These involve estimates of the form:
$$
\log p (y | x) = \sum_ {k = 1} ^ {K} \log p _ {k} (y | x) / K,
$$
where  $p_k(y|x)$  is a probability estimate that depends on the smoothing method. For MC-dropout, it is obtained by dropping neural connections with probability  $\alpha$ . For subword regularization,  $p_k(y|x) = p(\tilde{y}_k|\tilde{x}_k)$ , where  $\tilde{x}_k$  and  $\tilde{y}_k$  are randomly-sampled alternative subword segmentations of  $x$  and  $y$ . Note that MC-dropout decomposes over tokens, yielding smoother per-token probabilities; subword regularization does not, since it does not preserve tokenization.
Aggregating token-level log-probabilities
Given a sequence of token probabilities  $\log p(y_t | y_{<t}, x), t = 1 \dots T$ , we derive segment-level scores  $s(x, y)$  using various statistics. Following Thompson and Post (2020), we sum to obtain segment log-probabilities or average to obtain mean token-wise log-probabilities. To eliminate the effect of outliers, we tried the median instead of the mean. To test the opposite intuition, we also tried the minimum. Finally, to reflect overall consistency, we compute standard deviation.
Confidence Thresholds
Quality scores implicitly reflect the presence or absence of errors in MT output. In some cases, model probabilities provide strong evidence for or against the existence of errors, but in other cases the model may be agnostic. To capture this intuition, we used the following mapping to obtain segment scores:
$$
s (x, y) = \left\{ \begin{array}{l l} - 1, & \log p (y | x) / T <   l \\ + 1, & \log p (y | x) / T > h \\ 0, & \text {e l s e} \end{array} \right.
$$
To set the thresholds  $(l, h)$  we used a coarse grid search on development data.
We evaluate reference-free metric scores on data from the WMT19 metrics task (Ma et al., 2019), consisting of outputs from different MT systems
2We perform an approximate search for the 10-best subword segmentations, then sample from this list with probability proportional to a unigram estimate  $q^{\alpha}(\tilde{x} |x)$
for 18 language pairs. For each language pair, we compute a metric score for each system, then use correlation with the provided human scores to assess the quality of our metric.<sup>3</sup> Following Ma et al. (2019) we measure correlation using Pearson's coefficient, and use Williams' test (Williams, 1959) to compute the significance of correlation differences, with a p-value  $< 0.05$ .
Ma et al. (2019) note that correlation scores are unrealistically high for many language pairs, and suggest using only the best  $k$  systems for small values of  $k$ . However, Mathur et al. (2020) show that this results in noisy and unreliable estimates. We adopt their suggestion to instead remove outlier systems whose scores have large deviations from the median according to the formula:
$$
\frac {| h - \tilde {h} |}{1 . 4 8 3 \times \operatorname {m e d i a n} _ {h} (| h - \tilde {h} |)} > 2. 5,
$$
where  $h$  is a system-level human score, and  $\tilde{h}$  is the median score across all systems for a given language pair.
To summarize a metric's performance across a set of language pairs, we report the weighted average of its Pearson correlations across languages. We first apply the Fisher Z-transformation to normalize raw language-specific correlations, then weight by the number of MT systems per language (post outlier filtering), then invert the Fisher Z-transformation and take the mean (Hedges and Olkin, 2014).
We used four training corpora. Prism-39 consists of noise-filtered multi-way parallel data curated by Thompson and Post (2020), extracted primarily from Wikimatrix, Global Voices, EuroParl, SE-Times, and United Nations, consisting of 99.8M sentence pairs in 39 languages, including direct parallel data for 706 language pairs. Wiki-39-Mono consists of monolingual data extracted from the multilingual Wikipedia corpus for the languages available in Prism-39. WMT-15 is the parallel
training data provided for the WMT 2019 News Translation Task (Barrault et al., 2019), augmented with 5 languages from previous WMT years—Estonian (et), Spanish (es), Latvian (lt), Hindi (hi) and Turkish (tr). All language pairs are to/from English except French-German. Sizes range from 60 million sentence pairs for English-Czech to 10k pairs for English-Gujarati (Table 4). Finally, WMT-15-Mono is the monolingual data provided alongside WMT-15.
Test data is from the WMT 2019 Metrics Task (Ma et al., 2019), consisting of system outputs on news-domain text for all 18 language pairs included in the task: English (en) to/from Czech (cs), German (de), Finnish (fi), Gujarati (gu), Kazakh (kk), Lithuanian (lt), Russian (ru), and Chinese (zh), excluding cs-en. There are three other language pairs not including English: de-CS, de-fr and fr-de. The average number of systems per language is 12, and the average test-set size is 1,633.
4.2 MT Systems
Scale	Params	Layers	Hidden	Heads	Model
Big	473M	6	8192	16	1024
Prism	900M	8	12288	20	1280
Massive	1.8B	8	16384	32	2048
Table 1: Model configurations used in our experiments.
We used the Lingvo toolkit (Shen et al., 2019), to train Transformer sequence-to-sequence models of various sizes as shown in Table 1, where the baseline Prism configuration matches that of Thompson and Post (2020). We use AdaFactor optimization with a learning rate of 1.0 and batch size of  $\sim 8000$  samples. Our shared vocabulary comprises 64k subwords.
This section presents our main results. All correlations in the tables below are for system-level scores, after outlier systems have been discarded for each language pair. For brevity, we report average correlations, normalized and weighted as described in section 3.3; full results are provided in Appendix B. Unless otherwise stated, all methods score system outputs using average log probabilities normalized by segment length.
3Human annotators assign segment-level scores on a  $0 - 100$  scale which are averaged across segments, then normalized to correct for annotator differences, then averaged across annotators to produce system-level scores. For out-of-English language pairs, annotations are made by comparison to the source text, which directly corresponds to our setting; for other pairs, they are made by comparing to reference translations.
Metric	All	en-xx	xx-en	xx-yy
BLEU	0.911	0.917	0.921	0.838
CHRF	0.933	0.937	0.919	0.954
UNI+*	0.808	0.746	0.822	-
Yisi-2*	0.487	0.272	0.646	0.489
Prism	0.861	0.814	0.887	0.911
Prism-trg2xx	0.853	0.812	0.872	0.907
Prism-src2xx	0.858	0.822	0.871	0.914
Table 2 shows key WMT19 baseline results for reference-based metrics (top two lines), reference-free metrics (next three lines), and our reimplementation of the Prism model (bottom lines). We achieve slightly better results for source-side tagging (Prism-src2xx), and on average match the original Prism results that use target-side tagging with this configuration, which we adopt for further experiments. The en-xx results are affected negatively by the inclusion of en-gu, which is absent from the Prism-39 corpus and has low correlation (0.400); however, interestingly, results for gu-en are on par with other language pairs, presumably due to the prevalence of English in the corpus.
5.2 Training data
Table 2: Baseline results. All numbers are average system-level correlations. *Average is computed over language pairs for which the corresponding metric had a submission in the WMT19 Metrics task.
Data	All	en-xx	xx-en	xx-yy
Prism-39	0.858	0.822	0.871	0.914
WMT-15	0.840	0.776	0.890	0.854
Prism-13	0.863	0.828	0.888	0.888
Prism-39 + WMT-15	0.867	0.811	0.896	0.923
Adding monolingual data
Prism-39 + Wiki-39	0.832	0.792	0.859	0.869
WMT-15 + WMT-15-Mono	0.870	0.839	0.910	0.818
Prism-39 + WMT-15-Mono	0.851	0.831	0.863	0.874
Table 3 gives results for training on different corpora described in section 4.1. The first four lines correspond to different multilingual training corpora, beginning with the Prism-39 model from the previous section. We see no gain on average from using the provided WMT-15 training corpora, despite possibly better domain fit and generally larger sizes for the language pairs in the test set (Table 4).
We speculate that this is due to preprocessing as we made no effort to clean or filter the WMT-15 corpus. This hypothesis is supported by the Prism-13 results, where we trained on the language pairs in Prism-39 that overlapped with the WMT-15 corpus, achieving slightly better average performance. Combining Prism-39 and WMT-15 improves further, yielding a relatively small but statistically significant average gain over pure Prism-39, at the cost of lower performance for the en-xx language pairs.
Table 3: Effect of training data. Significant improvement over baseline "Prism-39" systems are underlined.
LP	Prism-39	WMT-15
en-zh	1.49	64.33
en-fr	3.52	40.44
en-ru	2.25	38.49
en-cs	0.65	25.98
en-es	4.40	15.18
de-fr	0.65	9.82
en-fi	0.28	6.58
en-de	1.36	4.50
en-et	0.22	2.17
en-lv	0.09	0.63
en-lt	0.16	0.63
en-kk	0.20	0.22
en-gu	0.00	0.01
de-cs	0.37	0.00
Table 4: Corpus size for overlapping language pairs from Prism-39 and WMT-15 (in millions of segments): WMT-15 has more parallel data available for all languages except de-cs, where no parallel corpora is available in the WMT-15 dataset.
Inspired by improvements for low-resource languages from monolingual data (Siddhant et al., 2020), we used the MASS denoising objective to add general-domain monolingual data (Wiki-39) to Prism-39 and in-domain data (WMT-15-Mono) to both Prism-39 and WMT-15 (Table 6 for a comparison on the relative sizes of the monolingual corpora). Overall, the general-domain data hurts correlation significantly, while in-domain helps significantly, but only for WMT-15. As expected, monolingual data tends to help lower-resource languages (gu, kk, lt) most, with a particularly large gain for xx-en with WMT-15 + WMT-15-Mono. However, the correlation for xx-yy language-pairs degrades significantly, which we attribute to the en-centric nature of the WMT-15 dataset.
Data	en-de	en-lt	en-ru	en-zh	de-en	lt-en	ru-en	zh-en
BLEU	0.806	0.986	0.946	0.802	0.794	0.985	0.812	0.808
Prism-39	0.730	0.939	0.901	0.789	0.796	0.978	0.739	0.828
Bilingual Models	0.726	0.695	0.867	0.769	0.801	0.862	0.650	0.826
Competition-grade	0.913	-	-	-	-	-	-	-
Table 5: Bilingual vs multilingual models for scoring.
LP	WMT-15-Mono	Wiki-39
de	275.69	59.93
en	199.90	130.79
fr	160.93	48.52
lt	106.19	4.85
ru	80.14	46.27
cs	72.15	9.44
fi	18.84	7.88
kk	13.82	3.34
gu	4.64	0.00
zh	2.15	21.79
et	51.68	3.03
es	43.81	36.17
hi	23.61	2.37
lv	10.20	1.36
tr	9.65	5.42
Table 6: Corpus size for overlapping languages from WMT-15-Mono and Wiki-39 dataset (in millions). The last five languages are not a part of WMT'19 Metrics evaluation task but were included when training the multilingual MT system.
5.3 Bilingual Systems
Can we use bilingual MT systems for peer evaluation? We chose four representative language pairs from Prism-39 and trained "Big" models (see Table 1) in eight directions, with dedicated 64k subword vocabularies. Table 5 shows that for medium and high resource languages (de, ru, and zh), the bilingual model performs comparably to the multilingual model. However, for the low resource language "lt", the multilingual model is significantly better. As with the results elsewhere in this section, this suggests that correlation tends to follow the pattern one would expect if we were mainly interested in model quality. This is corroborated by the results in the last line of the table, where we compare a competition-grade model for en-de (Freitag et al., 2020a), similar to the winning submission from WMT19, to our models. The competition-grade model achieves a much better correlation and also improves on BLEU by a wide margin.
5.4 Model Capacity
Metric	All	en-xx	xx-en	xx-yy
BLEU	0.911	0.917	0.921	0.838
Big	0.808	0.745	0.838	0.885
Prism	0.858	0.822	0.871	0.914
Massive	0.883	0.858	0.890	0.927
Motivated by the link between correlation and model quality, we varied model capacity according to the settings in Table 1, using the Prism-39 training corpus. The results in Table 7 show a clear pattern of gains with increasing capacity. The Massive configuration does best overall, achieving statistical parity with BLEU on average.
5.5 Scoring Methods
Table 7: Effect of Model capacity.
Aggregation Method	All	en-xx	xx-en	xx-yy
Mean	0.883	0.858	0.890	0.927
Std-dev	0.882	0.847	0.903	0.919
Median	0.870	0.859	0.876	0.887
Min	0.872	0.840	0.895	0.896
MC-dropout (Mean)	0.877	0.847	0.888	0.926
SP-Norm (Mean)	0.884	0.861	0.892	0.924
Confidence threshold	0.886	0.898	0.858	0.910
Table 8: Scoring methods. Significant improvements over baseline Mean systems are underlined.