SlowGuess/ACL_2020_OCR huggingface dataset:
Emanuele Bugliarello $^{c}$  Sabrina J. Mielke $^{f}$  Antonios Anastasopoulos $^{g}$  Ryan Cotterell $^{h,g}$  Naoaki Okazaki $^{g}$
<sup>c</sup>University of Copenhagen <sup>f</sup>Johns Hopkins University <sup>a</sup>Carnegie Mellon University <sup>b</sup>University of Cambridge <sup>c</sup>ETH Zürich <sup>d</sup>Tokyo Institute of Technology
emanuele@di.ku.dk, sjmielke@jhu.edu, aanastas@cs.cmu.edu, rcotterell@inf.ethz.ch, okazaki@c.titech.ac.jp
The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.
Machine translation (MT) is one of the core research areas in natural language processing. Current state-of-the-art MT systems are based on neural networks (Sutskever et al., 2014; Bahdanau et al., 2015), which generally surpass phrase-based systems (Koehn, 2009) in a variety of domains and languages (Bentivogli et al., 2016; Toral and Sánchez-Cartagena, 2017; Castilho et al., 2017; Bojar et al., 2018; Barrault et al., 2019). Using phrase-based MT systems, various controlled studies to understand where the translation difficulties lie for different language pairs were conducted (Birch et al., 2008; Koehn et al., 2009). However, comparable studies have yet to be performed for neural machine translation (NMT). As a result, it is still unclear whether all translation directions are equally easy (or hard) to model for NMT. This paper hence aims at filling this gap: Ceteris paribus,
Figure 1: Left: Decomposing the uncertainty of a sentence as mutual information plus language-inherent uncertainty: mutual information (MI) corresponds to just how much easier it becomes to predict  $T$  when you are given  $S$ . MI is symmetric but the relation between  $\mathrm{H}(S)$  and  $\mathrm{H}(T)$  can be arbitrary. Right: estimating cross-entropies using models  $q_{\mathrm{MT}}$  and  $q_{\mathrm{LM}}$  invalidates relations between bars, except that  $\mathrm{H}_q(\cdot) \geq \mathrm{H}(\cdot)$ . XMI, our proposed metric, is no longer purely a symmetric measure of language, but now an asymmetric measure that mostly highlights models' shortcomings.
is it easier to translate from English into Finnish or into Hungarian? And how much easier is it? Conversely, is it equally hard to translate Finnish and Hungarian into another language?
Based on BLEU (Papineni et al., 2002) scores, previous work (Belinkov et al., 2017) suggests that translating into morphologically rich languages, such as Hungarian or Finnish, is harder than translating into morphologically poor ones, such as English. However, a major obstacle in the cross-lingual comparison of MT systems is that many automatic evaluation metrics, including BLEU and METEOR (Banerjee and Lavie, 2005), are not cross-lingually comparable. In fact, being a function of  $n$ -gram overlap between candidate and reference translations, they only allow for a fair comparison of the performance between models when translating into the same test set in the same target language. Indeed, one cannot and should not draw conclusions about the difficulty of translating a source language into different target languages purely based on BLEU (or METEOR) scores.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1640-1649 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In response, we propose cross-mutual information (XMI), a new metric towards cross-linguistic comparability in NMT. In contrast to BLEU, this information-theoretic quantity no longer explicitly depends on language, model, and tokenization choices. It does, however, require that the models under consideration are probabilistic. As an initial starting point, we perform a case study with a controlled experiment on 21 European languages. Our analysis showcases XMI's potential for shedding light on the difficulties of translation as an effect of the properties of the source or target language. We also perform a correlation analysis in an attempt to further explain our findings. Here, in contrast to the general wisdom, we find no significant evidence that translating into a morphologically rich language is harder than translating into a morphologically impoverished one. In fact, the only significant correlate of MT difficulty we find is source-side type-token ratio.
2 Cross-Linguistic Comparability through Likelihoods, not BLEU
Human evaluation will always be the gold standard of MT evaluation. However, it is both time-consuming and expensive to perform. To help researchers and practitioners quickly deploy and evaluate new systems, automatic metrics that correlate fairly well with human evaluations have been proposed over the years (Banerjee and Lavie, 2005; Snover et al., 2006; Isozaki et al., 2010; Lo, 2019). BLEU (Papineni et al., 2002), however, has remained the most common metric to report the performance of MT systems. BLEU is a precision-based metric: a BLEU score is proportional to the geometric average of the number of  $n$ -grams in the candidate translation that also appear in the reference translation for  $1 \leq n \leq 4$ .
In the context of our study, we take issue with two shortcomings of BLEU scores that prevent a cross-linguistically comparable study. First, it is not possible to directly compare BLEU scores across languages because different languages might express the same meaning with a very different number of words. For instance, agglutinative languages like Turkish often use a single word to express what other languages have periphrastic constructions for. To be concrete, the expression "I will have been programming" is five words in En
glish, but could easily have been one word in a language with sufficient morphological markings; this unfairly boosts BLEU scores when translating into English. The problem is further exacerbated by tokenization techniques as finer granularities result in more partial credit and higher  $n$  for the  $n$ -gram matches (Post, 2018). In summary, BLEU only allows us to compare models for a fixed target language and tokenization scheme, i.e. it only allows us to draw conclusions about the difficulty of translating different source languages into a specific target one (with downstream performance as a proxy for difficulty). Thus, BLEU scores cannot provide an answer to which translation direction is easier between any two source-target pairs.
In this work, we address this particular shortcoming by considering an information-theoretic evaluation. Formally, let  $\nu_{\mathrm{S}}$  and  $\nu_{\mathrm{T}}$  be source- and target-language vocabularies, respectively. Let  $S$  and  $T$  be source- and target-sentence-valued random variables for languages S and T, respectively; then  $S$  and  $T$  respectively range over  $\nu_{\mathrm{S}}^{*}$  and  $\nu_{\mathrm{T}}^{*}$ . These random variables  $S$  and  $T$  are distributed according to some true, unknown probability distribution  $p$ . The cross-entropy between the true distribution  $p$  and a probabilistic neural translation model  $q_{\mathrm{MT}}(\mathbf{t}|\mathbf{s})$  is defined as:
$$
\begin{array}{l} \mathrm {H} _ {q _ {\mathrm {M T}}} (T \mid S) = \tag {1} \\ - \sum_ {\mathbf {t} \in \mathcal {V} _ {\mathrm {T}} ^ {*}} \sum_ {\mathbf {s} \in \mathcal {V} _ {\mathrm {S}} ^ {*}} p (\mathbf {t}, \mathbf {s}) \log_ {2} q _ {\mathrm {M T}} (\mathbf {t} \mid \mathbf {s}) \\ \end{array}
$$
Since we do not know  $p$ , we cannot compute eq. (1). However, given a held-out data set of sentence pairs  $\{(\mathbf{s}^{(i)},\mathbf{t}^{(i)})\}_{i = 1}^{N}$  assumed to be drawn from  $p$ , we can approximate the true cross-entropy as follows:
$$
\begin{array}{l} \mathrm {H} _ {q _ {\mathrm {M T}}} (T \mid S) \approx \tag {2} \\ - \frac {1}{N} \sum_ {i = 1} ^ {N} \log_ {2} q _ {\mathrm {M T}} (\mathbf {t} ^ {(i)} \mid \mathbf {s} ^ {(i)}) \\ \end{array}
$$
In the limit as  $N\to \infty$  , eq. (2) converges to eq. (1).
We emphasize that this evaluation does not rely on language tokenization provided that the model  $q_{\mathrm{MT}}$  does not (Mielke, 2019). While common in the evaluation of language models, cross-entropy evaluation has been eschewed in machine translation research since (i) not all MT models are probabilistic and (ii) we are often interested in measuring the quality of the candidate translation our model actually produces, e.g. under approximate decoding. However, an information-theoretic evaluation
1BLEU also corrects for reference coverage and includes a length penalty, but we focus on the high-level picture.
is much more suitable for measuring the more abstract notion of which language pairs are hardest to translate to and from, which is our purpose here.
3 Disentangling Translation Difficulty and Monolingual Complexity
We contend that simply reporting cross-entropies is not enough. A second issue in performing a controlled, cross-lingual MT comparison is that the language generation component (without translation) is not equally difficult across languages (Cotterell et al., 2018). We claim that the difficulty of translation corresponds more closely to the mutual information  $\mathrm{MI}(S;T)$  between the source and target language, which tells us how much easier it becomes to predict  $T$  when  $S$  is given (see Figure 1). But what is the appropriate analogue of mutual information for cross-entropy? One such natural generalization is a novel quantity that we term cross-mutual information, defined as:
$$
\operatorname {X M I} (S \rightarrow T) = \mathrm {H} _ {q _ {\mathrm {L M}}} (T) - \mathrm {H} _ {q _ {\mathrm {M T}}} (T \mid S) \tag {3}
$$
where  $\mathrm{H}_{q_{\mathrm{LM}}}(T)$  denotes the cross-entropy of the target sentence  $T$  under the model  $q_{\mathrm{LM}}$ . As in §2, this can, analogously, be approximated by the cross-entropy of a separate target-side language model  $q_{\mathrm{LM}}$  over our held-out data set:
$$
\begin{array}{l} \operatorname {X M I} (S \rightarrow T) \approx \tag {4} \\ - \frac {1}{N} \sum_ {i = 1} ^ {N} \log_ {2} \frac {q _ {\mathrm {L M}} (\mathbf {t} ^ {(i)})}{q _ {\mathrm {M T}} (\mathbf {t} ^ {(i)} \mid \mathbf {s} ^ {(i)})} \\ \end{array}
$$
which, again, becomes exact as  $N\to \infty$ . In practice, we note that we mix different distributions  $q_{\mathrm{LM}}(\mathbf{t})$  and  $q_{\mathrm{MT}}(\mathbf{t}\mid \mathbf{s})$  and, thus,  $q_{\mathrm{LM}}(\mathbf{t})$  is not necessarily representable as a marginal: there need not be any distribution  $\tilde{q} (\mathbf{s})$  such that  $q_{\mathrm{LM}}(\mathbf{t}) = \sum_{\mathbf{s}\in \mathcal{V}_{\mathrm{S}}^{*}}q_{\mathrm{MT}}(\mathbf{t}\mid \mathbf{s})\cdot \tilde{q} (\mathbf{s})$ . While  $q_{\mathrm{MT}}$  and  $q_{\mathrm{LM}}$  can, in general, be any two models, we exploit the characteristics of NMT models to provide a more meaningful, model-specific estimate of XMI. NMT architectures typically consist of two components: an encoder that embeds the input text sequence, and a decoder that generates translated output text. The latter acts as a conditional language model, where the source-language sentence embedded by the encoder drives the target-language generation. Hence, we use the decoder of  $q_{\mathrm{MT}}$  as our  $q_{\mathrm{LM}}$  to accurately estimate the difficulty of translation for a given architecture in a controlled way.
In summary, by looking at XMI, we can effectively decouple the language generation component, whose difficulties have been investigated by Cotterell et al. 2018 and Mielke et al. 2019, from the translation component. This gives us a measure of how rich and useful the information extracted from the source language is for the target-language generation component.
In order to measure which pairs of languages are harder to translate to and from, we make use of the latest release v7 of Europarl (Koehn, 2005): a corpus of the proceedings of the European Parliament containing parallel sentences between English (en) and 20 other European languages: Bulgarian (bg), Czech (cs), Danish (da), German (de), Greek (e1), Spanish (es), Estonian (et), Finnish (fi), French (fr), Hungarian (hu), Italian (it), Lithuanian (lt), Latvian (lv), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk), Slovene (sl) and Swedish (sv).
Pre-processing steps In order to precisely effect a fully controlled experiment, we enforce a fair comparison by selecting the set of parallel sentences available across all 21 languages in Europarl. This fully controls for the semantic content of the sentences; however, we cannot adequately control for translationese (Stymne, 2017; Zhang and Toral, 2019). Our subset of Europarl contains 190,733 sentences for training, 1,000 unique, random sentences for validation and 2,000 unique, random sentences for testing. For each parallel corpus, we jointly learn byte-pair encodings (BPE; Sennrich et al., 2016) for the source and target languages, using 16,000 merge operations. We use the same vocabularies for the language models.
Setup In our experiments, we train Transformer models (Vaswani et al., 2017), which often achieve state-of-the-art performance on MT for various language pairs. In particular, we rely on the PyTorch (Paszke et al., 2019) re-implementation of the Transformer model in the fairseq toolkit (Ott et al., 2019). For language modeling, we use the decoder from the same architecture, training it at the sentence level, as opposed to commonly used fixed-length chunks. We train our systems using label smoothing (LS; Szegedy et al., 2016; Meister et al.,
2For English, we arbitrarily chose the English portion of the en-bg vocabulary.
∅ → en	bg	cs	da	de	el	es	et	fi	fr	hu	it	lt	lv	nl	pl	pt	ro	sk	sl	sv	avg
BLEU	47.4	42.4	46.3	44.0	50.0	50.6	39.3	38.2	44.9	38.4	40.8	37.6	40.3	38.3	39.8	48.3	50.5	44.2	45.3	43.7	43.5
XMI(∅ → en)	102.3	97.0	99.7	96.5	105.3	103.8	92.8	92.1	97.0	92.5	92.1	89.2	94.2	86.5	91.9	102.5	106.1	99.8	100.1	96.9	96.9
HqLM(en)	154.2	154.2
HqMT(en | ∅)	51.8	57.2	54.5	57.7	48.9	50.4	61.4	62.0	57.2	61.6	62.1	65.0	60.0	67.7	62.3	51.7	48.1	54.4	54.1	57.3	57.3
en → ∅	bg	cs	da	de	el	es	et	fi	fr	hu	it	lt	lv	nl	pl	pt	ro	sk	sl	sv	avg
BLEU	46.3	34.7	45.0	36.3	45.5	50.2	27.7	30.5	45.7	30.3	37.9	31.0	34.6	34.9	30.5	46.7	44.2	39.8	41.5	41.3	38.73
XMI(enti ∅)	106.2	102.8	103.3	104.0	111.0	108.1	100.2	98.0	99.7	99.1	95.3	96.0	99.3	90.4	98.3	105.2	112.4	105.8	107.9	100.1	102.1
HqLM(∅)	156.5	164.0	152.7	167.6	163.7	159.3	162.5	158.6	154.9	166.6	158.6	159.2	156.4	159.7	163.4	159.3	160.5	157.7	158.2	153.1	159.6
HqMT(∅ | en)	50.3	61.2	49.4	63.6	52.7	51.3	62.4	60.6	55.1	67.5	63.3	63.1	57.0	69.3	65.1	54.1	48.1	51.9	50.3	53.0	57.5
Table 1: Test scores, from and into English, Europarl, visualized in Figure 2 and Figure 3.
Figure 2: Some correlations between metrics in Table 1, into and from English. More correlations in Figure 4.
2020) as it has been shown to prevent models from over-confident predictions, which helps to regularize the models. We report cross-entropies  $\left(\mathrm{H}_{q_{\mathrm{MT}}}, \mathrm{H}_{q_{\mathrm{LM}}}\right)$ , XMI, and BLEU scores obtained using SACREBLEU (Post, 2018). Finally, in a similar vein to Cotterell et al. (2018), we multiply cross-entropy values by the number of sub-word units generated by each model to make our quantities independent of sentence lengths (and divide them by the total number of sentences to match our approximations of the true distributions). See App. A for experimental details.
We train 40 systems, translating each language into and from English. The models' performance in terms of BLEU scores, and the cross-mutual information (XMI) and cross-entropy values over the test sets are reported in Table 1 with significant values marked in App. B.
Translating into English When translating into the same target language (in this case, English), BLEU scores are, in fact, comparable, and can be used as a proxy for difficulty. We can then conclude, for instance, that Lithuanian  $(1\mathrm{t})$  is the hardest language to translate from, while Spanish  $(\in s)$  is the easiest. In this scenario, given the good correlation of BLEU scores with human evaluations, it is desirable that XMI correlates well with BLEU. This behavior is indeed apparent in the blue points in the left part of Figure 2, confirming the efficacy of XMI in evaluating the difficulty of translation while still being independent of the target language generation component.
Translating from English Despite the large gaps between BLEU scores in Table 1, one should not be tempted to claim that it is easier to translate into English than from English for these languages as often hinted at in previous work (e.g., Belinkov et al., 2017). As we described above, different target languages are not directly comparable, and we actually find that XMI is slightly higher, on average, when translating from English, indicating that it is actually easier, on average, to transfer information correctly in this direction. For instance, translation from English to Finnish is shown to be easier than from Finnish to English, despite the large gap
$^{3}$ Signature: BLEU+c.mixed+#.1+s.exp+tok.13a+v.1.2.12.  $^{4}$ Due to resource limitations, we chose these tasks because most of the information available in the web is in English (https://w3techs.com/technologies/overview/content_language) and effectively translating it into any other language would reduce the digital language divide (http://labs.theguardian.com/digital-language-divide/). Besides, translating into English gives most people access to any local information.
Figure 3:  $\mathrm{H}_{q_{\mathrm{LM}}}(T)$ , decomposed into  $\mathrm{XMI}(S \to T)$ , the information that the system successfully transfers, and  $\mathrm{H}_{q_{\mathrm{MT}}}(T \mid S)$ , the uncertainty that remains in the target language, all measured in bits. Note that in  $\mathrm{XMI}(S \to T)$  the translation is from the left to the right argument.
Metric	Pearson	Spearman
word number ratio	0.2988 (0.0611)	0.3570 (0.0237)
TTRsrc	-0.5196 (0.0006)	-0.5136 (0.0007)
TTRtgt	0.1651 (0.3086)	0.3355 (0.0343)
dTTR	-0.4427 (0.0042)	-0.4660 (0.0024)
word overlap ratio	0.1383 (0.3949)	0.1731 (0.2853)
Table 2: Correlation coefficients (and  $p$ -values) between XMI and data-related features.
in BLEU scores. This suggests that the former model is heavily penalized by the target-side language model; this is likely because Finnish has a large number of inflections for nouns and verbs. Another interesting example is given by Greek (e1) and Spanish (es) in Table 1, where, again, the two tasks achieve very different BLEU scores but similar XMI. In light of the correlation with BLEU when translating into English, this shows us that Greek is just harder to language-model, corroborating the findings of Mielke et al. (2019). Moreover, Figure 2 clearly shows that, as expected, XMI is not as well correlated with BLEU when translating from English, given that BLEU scores are not cross-lingually comparable.
Correlations with linguistic and data features Last, we conduct a correlation study between the translation difficulties as measured by XMI and the linguistic and data-dependent properties of each translation task, following the approaches of Lin et al. (2019) and Mielke et al. (2019). Table 2 lists Pearson's and Spearman's correlation coefficients for data-dependent metrics, where bold values indicate statistically significant results  $(p < 0.05)$  after Bonferroni correction  $(p < 0.0029)$ . Interestingly, the only features that significantly correlate with our metric are related to the type-to-token ratio (TTR) for the source language and the distance
between source and target TTRs. This implies that a potential explanation for the differences in translation difficulty lies in lexical variation. For full correlation results, refer to App. D.
In this work, we propose a novel information-theoretic approach, XMI, to measure the translation difficulty of probabilistic MT models. Differently from BLEU and other metrics, ours is language- and tokenization-agnostic, enabling the first systematic and controlled study of crosslingual translation difficulties. Our results show that XMI correlates well with BLEU scores when translating into the same language (where they are comparable), and that higher BLEU scores in different languages do not necessarily imply easier translations. In future work, we plan to extend this analysis across more translation pairs, more diverse languages and multiple domains, as well as investigating the effect of translationese or source-side grammatical errors (Anastasopoulos, 2019).
The authors are thankful to the anonymous reviewers for their valuable feedback. The second-to-last author acknowledges a Facebook Fellowship and discussions with Tiago Pimentel. This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 801199, the National Science Foundation under grant 1761548, and by "Research and Development of Deep Learning Technology for Advanced Multilingual Speech Translation," the Commissioned Research of National Institute of Information and Communications Technology (NICT), Japan.