SlowGuess/ACL_2020_OCR huggingface dataset:
Wei Zhao†, Goran Glavas‡, Maxime PeyrardΦ, Yang Gao*, Robert WestΦ, Steffen Eger†
† Technische Universität Darmstadt ‡ University of Mannheim, Germany  
Φ EPFL, Switerland * Royal Holloway University of London, UK  
{zhao, eger}@aiphes.tu-darmstadt.de  
goran@informatik.uni-mannheim.de, yang.gao@rhul.ac.uk  
{maxime.peyrard, robert.west}@epfl.ch
Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual similarity. In this paper, we concern ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations, which represents a natural adversarial setup for multilingual encoders. Reference-free evaluation holds the promise of web-scale comparison of MT systems. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations, namely, (a) a semantic mismatch between representations of mutual translations and, more prominently, (b) the inability to punish "translationese", i.e., low-quality literal translations. We propose two partial remedies: (1) post-hoc re-alignment of the vector spaces and (2) coupling of semantic-similarity based metrics with target-side language modeling. In segment-level MT evaluation, our best metric surpasses reference-based BLEU by 5.7 correlation points. We make our MT evaluation code available. $^{1}$
A standard evaluation setup for supervised machine learning (ML) tasks assumes an evaluation metric which compares a gold label to a classifier prediction. This setup assumes that the task has clearly defined and unambiguous labels and, in most cases, that an instance can be assigned few labels. These assumptions, however, do not hold for natural language generation (NLG) tasks like machine trans
lation (MT) (Bahdanau et al., 2015; Johnson et al., 2017) and text summarization (Rush et al., 2015; Tan et al., 2017), where we do not predict a single discrete label but generate natural language text. Thus, the set of labels for NLG is neither clearly defined nor finite. Yet, the standard evaluation protocols for NLG still predominantly follow the described default paradigm: (1) evaluation datasets come with human-created reference texts and (2) evaluation metrics, e.g., BLEU (Papineni et al., 2002) or METEOR (Lavie and Agarwal, 2007) for MT and ROUGE (Lin and Hovy, 2003) for summarization, count the exact "label" (i.e.,  $n$ -gram) matches between reference and system-generated text. In other words, established NLG evaluation compares semantically ambiguous labels from an unbounded set (i.e., natural language texts) via hard symbolic matching (i.e., string overlap).
The first remedy is to replace the hard symbolic comparison of natural language "labels" with a soft comparison of texts' meaning, using semantic vector space representations. Recently, a number of MT evaluation methods appeared focusing on semantic comparison of reference and system translations (Shimanaka et al., 2018; Clark et al., 2019; Zhao et al., 2019). While these correlate better than  $n$ -gram overlap metrics with human assessments, they do not address inherent limitations stemming from the need for reference translations, namely: (1) references are expensive to obtain; (2) they assume a single correct solution and bias the evaluation, both automatic and human (Dreyer and Marcu, 2012; Fomicheva and Specia, 2016), and (3) limitation of MT evaluation to language pairs with available parallel data.
Reliable reference-free evaluation metrics, directly measuring the (semantic) correspondence between the source language text and system translation, would remove the need for human references and allow for unlimited MT evaluations: any
1https://github.com/AIPHES/ ACL20-Reference-Free-MT-Evaluation
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1656-1671 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
monolingual corpus could be used for evaluating MT systems. However, the proposals of reference-free MT evaluation metrics have been few and far apart and have required either non-negligible supervision (i.e., human translation quality labels) (Specia et al., 2010) or language-specific preprocessing like semantic parsing (Lo et al., 2014; Lo, 2019), both hindering the wide applicability of the proposed metrics. Moreover, they have also typically exhibited performance levels well below those of standard reference-based metrics (Ma et al., 2019).
In this work, we comparatively evaluate a number of reference-free MT evaluation metrics that build on the most recent developments in multilingual representation learning, namely cross-lingual contextualized embeddings (Devlin et al., 2019) and cross-lingual sentence encoders (Artetxe and Schwenk, 2019). We investigate two types of cross-lingual reference-free metrics: (1) Soft token-level alignment metrics find the optimal soft alignment between source sentence and system translation using Word Mover's Distance (WMD) (Kusner et al., 2015). Zhao et al. (2019) recently demonstrated that WMD operating on BERT representations (Devlin et al., 2019) substantially outperforms baseline MT evaluation metrics in the reference-based setting. In this work, we investigate whether WMD can yield comparable success in the reference-free (i.e., cross-lingual) setup; (2) Sentence-level similarity metrics measure the similarity between sentence representations of the source sentence and system translation using cosine similarity.
Our analysis yields several interesting findings. (i) We show that, unlike in the monolingual reference-based setup, metrics that operate on contextualized representations generally do not outperform symbolic matching metrics like BLEU, which operate in the reference-based environment. (ii) We identify two reasons for this failure: (a) firstly, cross-lingual semantic mismatch, especially for multi-lingual BERT (M-BERT), which construes a shared multilingual space in an unsupervised fashion, without any direct bilingual signal; (b) secondly, the inability of the state-of-the-art cross-lingual metrics based on multilingual encoders to adequately capture and punish "translationese", i.e., literal word-by-word translations of the source sentence—as translationese is an especially persistent property of MT systems, this problem is particularly troubling in our context of reference-free MT evaluation. (iii) We show that by executing an additional weakly-supervised cross-lingual
re-mapping step, we can to some extent alleviate both previous issues. (iv) Finally, we show that the combination of cross-lingual reference-free metrics and language modeling on the target side (which is able to detect "translationese"), surpasses the performance of reference-based baselines.
Beyond designating a viable prospect of web-scale domain-agnostic MT evaluation, our findings indicate that the challenging task of reference-free MT evaluation is able to expose an important limitation of current state-of-the-art multilingual encoders, i.e., the failure to properly represent corrupt input, that may go unnoticed in simpler evaluation setups such as zero-shot cross-lingual text classification or measuring cross-lingual text similarity not involving "adversarial" conditions. We believe this is a promising direction for nuanced, fine-grained evaluation of cross-lingual representations, extending the recent benchmarks which focus on zero-shot transfer scenarios (Hu et al., 2020).
Manual human evaluations of MT systems undoubtedly yield the most reliable results, but are expensive, tedious, and generally do not scale to a multitude of domains. A significant body of research is thus dedicated to the study of automatic evaluation metrics for machine translation. Here, we provide an overview of both reference-based MT evaluation metrics and recent research efforts towards reference-free MT evaluation, which leverage cross-lingual semantic representations and unsupervised MT techniques.
Reference-based MT evaluation. Most of the commonly used evaluation metrics in MT compare system and reference translations. They are often based on surface forms such as  $n$ -gram overlaps like BLEU (Papineni et al., 2002), SentBLEU, NIST (Doddington, 2002), chrF++ (Popovic, 2017) or METEOR++(Guo and Hu, 2019). They have been extensively tested and compared in recent WMT metrics shared tasks (Bojar et al., 2017a; Ma et al., 2018a, 2019).
These metrics, however, operate at the surface level, and by design fail to recognize semantic equivalence lacking lexical overlap. To overcome these limitations, some research efforts exploited static word embeddings (Mikolov et al., 2013b) and trained embedding-based supervised metrics on sufficiently large datasets with available human judgments of translation quality (Shimanaka
et al., 2018). With the development of contextual word embeddings (Peters et al., 2018; Devlin et al., 2019), we have witnessed proposals of semantic metrics that account for word order. For example, Clark et al. (2019) introduce a semantic metric relying on sentence mover's similarity and the contextualized ELMo embeddings (Peters et al., 2018). Similarly, Zhang et al. (2019) describe a reference-based semantic similarity metric based on contextualized BERT representations (Devlin et al., 2019). Zhao et al. (2019) generalize this line of work with their MoverScore metric, which computes the mover's distance, i.e., the optimal soft alignment between tokens of the two sentences, based on the similarities between their contextualized embeddings. Mathur et al. (2019) train a supervised BERT-based regressor for reference-based MT evaluation.
Reference-free MT evaluation. Recently, there has been a growing interest in reference-free MT evaluation (Ma et al., 2019), also referred to as "quality estimation" (QE) in the MT community. In this setup, evaluation metrics semantically compare system translations directly to the source sentences. The attractiveness of automatic reference-free MT evaluation is obvious: it does not require any human effort or parallel data. To approach this task, Popovic et al. (2011) exploit a bag-of-word translation model to estimate translation quality, which sums over the likelihoods of aligned word-pairs between source and translation texts. Specia et al. (2013) estimate translation quality using language-agnostic linguistic features extracted from source lanuage texts and system translations. Lo et al. (2014) introduce XMEANT as a crosslingual reference-free variant of MEANT, a metric based on semantic frames. Lo (2019) extended this idea by leveraging M-BERT embeddings. The resulting metric, YiSi-2, evaluates system translations by summing similarity scores over words pairs that are best-aligned mutual translations. YiSi-2-SRL optionally combines an additional similarity score based on the alignment over the semantic structures (e.g., semantic roles and frames). Both metrics are reference-free, but YiSi-2-SRL is not resource-lean as it requires a semantic parser for both languages. Moreover, in contrast to our proposed metrics, they do not mitigate the misalignment of cross-lingual embedding spaces and do not integrate a target-side language model, which we identify to be crucial components.
Recent progress in cross-lingual semantic similarity (Agirre et al., 2016; Cer et al., 2017) and unsupervised MT (Artetxe and Schwenk, 2019) has also led to novel reference-free metrics. For instance, Yankovskaya et al. (2019) propose to train a metric combining multilingual embeddings extracted from M-BERT and LASER (Artetxe and Schwenk, 2019) together with the log-probability scores from neural machine translation. Our work differs from that of Yankovskaya et al. (2019) in one crucial aspect: the cross-lingual reference-free metrics that we investigate and benchmark do not require any human supervision.
Cross-lingual Representations. Cross-lingual text representations offer a prospect of modeling meaning across languages and support cross-lingual transfer for downstream tasks (Klementiev et al., 2012; Rückle et al., 2018; Glavaš et al., 2019; Josifoski et al., 2019; Conneau et al., 2020). Most recently, the (massively) multilingual encoders, such as multilingual M-BERT (Devlin et al., 2019), XLM-on-RoBERTa (Conneau et al., 2020), and (sentence-based) LASER, have profiled themselves as state-of-the-art solutions for (massively) multilingual semantic encoding of text. While LASER has been jointly trained on parallel data of 93 languages, M-BERT has been trained on the concatenation of monolingual data in more than 100 languages, without any cross-lingual mapping signal. There has been a recent vivid discussion on the cross-lingual abilities of M-BERT (Pires et al., 2019; K et al., 2020; Cao et al., 2020). In particular, Cao et al. (2020) show that M-BERT often yields disparate vector space representations for mutual translations and propose a multilingual remapping based on parallel corpora, to remedy for this issue. In this work, we introduce re-mapping solutions that are resource-leaner and require easy-to-obtain limited-size word translation dictionaries rather than large parallel corpora.
3 Reference-Free MT Evaluation Metrics
In the following, we use  $\mathbf{x}$  to denote a source sentence (i.e., a sequence of tokens in the source language),  $\mathbf{y}$  to denote a system translation of  $\mathbf{x}$  in the target language, and  $\mathbf{y}^{\star}$  to denote the human reference translation for  $\mathbf{x}$ .
3.1 Soft Token-Level Alignment
We start from the MoverScore (Zhao et al., 2019), a recently proposed reference-based MT evaluation
metric designed to measure the semantic similarity between system outputs (y) and human references  $(\mathbf{y}^{\star})$ . It finds an optimal soft semantic alignments between tokens from y and  $\mathbf{y}^{\star}$  by minimizing the Word Mover's Distance (Kusner et al., 2015). In this work, we extend the MoverScore metric to operate in the cross-lingual setup, i.e., to measure the semantic similarity between  $n$ -grams (unigram or bigrams) of the source text x and the system translation y, represented with embeddings originating from a cross-lingual semantic space.
First, we decompose the source text  $\mathbf{x}$  into a sequence of  $n$ -grams, denoted by  $\boldsymbol{x}_n = (\mathrm{x}_1^n,\dots ,\mathrm{x}_m^n)$  and then do the same operation for the system translation  $\mathbf{y}$ , denoting the resulting sequence of  $n$ -grams with  $\boldsymbol{y}_n$ . Given  $\boldsymbol{x}_n$  and  $\boldsymbol{y}_n$ , we can then define a distance matrix  $\pmb{C}$  such that  $C_{ij} = \| E(\mathrm{x}_i^n) - E(\mathrm{y}_j^n)\|_2$  is the distance between the  $i$ -th  $n$ -gram of  $\boldsymbol{x}$  and the  $j$ -th  $n$ -gram of  $\boldsymbol{y}$ , where  $E$  is a cross-lingual embedding function that maps text in different languages to a shared embedding space. With respect to the function  $E$ , we experimented with cross-lingual representations induced (a) from static word embeddings with RCSLS (Joulin et al., 2018)) (b) with M-BERT (Devlin et al., 2019) as the multilingual encoder; with a focus on the latter. For M-BERT, we take the representations of the last transformer layer as the text representations.
WMD between the two sequences of  $n$ -grams  $\pmb{x}^n$  and  $\pmb{y}^n$  with associated  $n$ -gram weights  $^2$  to  $\pmb{f}_{\pmb{x}^n}\in \mathbb{R}^{|\pmb{x}^n|}$  and  $\pmb {f}_{\pmb{y}^n}\in \mathbb{R}^{|\pmb{y}^n|}$  is defined as:
$$
m (\boldsymbol {x}, \boldsymbol {y}) := \operatorname {W M D} \left(\boldsymbol {x} ^ {n}, \boldsymbol {y} ^ {n}\right) = \min  _ {\boldsymbol {F}} \sum_ {i j} C _ {i j} \cdot F _ {i j},
$$
s.t.  $F\mathbf{1} = f_{\pmb{x}^n}$  ，  $F^{\top}\mathbf{1} = f_{\pmb{y}^{n}}$
where  $\pmb{F} \in \mathbb{R}^{|\pmb{x}^n| \times |\pmb{y}^n|}$  is a transportation matrix with  $F_{ij}$  denoting the amount of flow traveling from  $\mathbf{x}_i^n$  to  $\mathbf{y}_j^n$ .
3.2 Sentence-Level Semantic Similarity
In addition to measuring semantic distance between  $\mathbf{x}$  and  $\mathbf{y}$  at word-level, one can also encode them into sentence representations with multilingual sentence encoders like LASER (Artetxe and Schwenk, 2019), and then measure their cosine distance
$$
m (\boldsymbol {x}, \boldsymbol {y}) = 1 - \frac {E (\boldsymbol {x}) ^ {\top} E (\boldsymbol {y})}{\| E (\boldsymbol {x}) \| \cdot \| E (\boldsymbol {y}) \|}.
$$
3.3 Improving Cross-Linguual Alignments
Initial analysis indicated that, despite the multilingual pretraining of M-BERT (Devlin et al., 2019) and LASER (Artetxe and Schwenk, 2019), the monolingual subspaces of the multilingual spaces they induce are far from being semantically well-aligned, i.e., we obtain fairly distant vectors for mutual word or sentence translations. To this end, we apply two simple, weakly-supervised linear projection methods for post-hoc improvement of the cross-lingual alignments in these multilingual representation spaces.
Notation. Let  $D = \{(w_{\ell}^{1}, w_{k}^{1}), \ldots, (w_{\ell}^{n}, w_{k}^{n})\}$  be a set of matched word or sentence pairs from two different languages  $\ell$  and  $k$ . We define a remapping function  $f$  such that any  $f(E(w_{\ell}))$  and  $E(w_{k})$  are better aligned in the resulting shared vector space. We investigate two resource-lean choices for the re-mapping function  $f$ .
Linear Cross-lingual Projection (CLP). Following related work (Schuster et al., 2019), we re-map contextualized embedding spaces using linear projection. Given  $\ell$  and  $k$ , we stack all vectors of the source language words and target language words for pairs  $D$ , respectively, to form matrices  $X_{\ell}$  and  $X_{k} \in \mathbb{R}^{n \times d}$ , with  $d$  as the embedding dimension and  $n$  as the number of word or sentence alignments. The word pairs we use to calibrate M-BERT are extracted from EuroParl (Koehn, 2005) using FastAlign (Dyer et al., 2013), and the sentence pairs to calibrate LASER are sampled directly from EuroParl. Mikolov et al. (2013a) propose to learn a projection matrix  $W \in \mathbb{R}^{d \times d}$  by minimizing the Euclidean distance between the projected source language vectors and their corresponding target language vectors:
$$
\min  _ {\boldsymbol {W}} \| \boldsymbol {W} \boldsymbol {X} _ {\ell} - \boldsymbol {X} _ {k} \| _ {2}.
$$
Xing et al. (2015) achieve further improvement on the task of bilingual lexicon induction (BLI) by constraining  $\mathbf{W}$  to an orthogonal matrix, i.e., such that  $\mathbf{W}^{\top}\mathbf{W} = \mathbf{I}$ . This turns the optimization into the well-known Procrustes problem (Schönemann, 1966) with the following closed-form solution:
$$
\hat {\boldsymbol {W}} = \boldsymbol {U} \boldsymbol {V} ^ {\intercal}, \boldsymbol {U} \boldsymbol {\Sigma} \boldsymbol {V} ^ {\intercal} = \operatorname {S V D} (\boldsymbol {X} _ {\ell} \boldsymbol {X} _ {k} ^ {\intercal})
$$
2We follow Zhao et al. (2019) in obtaining  $n$ -gram embeddings and their associated weights based on IDF.
$^{3}$ LASER is jointly trained on parallel corpora of different languages, but in resource-lean language pairs, the induced embeddings from mutual translations may be far apart.
4While LASER requires large parallel corpora in pretraining, we believe that fine-tuning/calibrating the embeddings post-hoc requires fewer data points.
We note that the above CLP re-mapping is known to have deficits, i.e., it requires the embedding spaces of the involved languages to be approximately isomorphic (Søgaard et al., 2018; Vulić et al., 2019). Recently, some re-mapping methods that reportedly remedy for this issue have been suggested (Glavaš and Vulić, 2020; Mohiuddin and Joty, 2020). We leave the investigation of these novel techniques for our future work.
Universal Language Mismatch-Direction (UMD) Our second post-hoc linear alignment method is inspired by the recent work on removing biases in distributional word vectors (Dev and Phillips, 2019; Lauscher et al., 2019). We adopt the same approaches in order to quantify and remedy for the "language bias", i.e., representation mismatches between mutual translations in the initial multilingual space. Formally, given  $\ell$  and  $k$ , we create individual misalignment vectors  $E(w_{\ell}^{i}) - E(w_{k}^{i})$  for each bilingual pair in  $D$ . Then we stack these individual vectors to form a matrix  $Q \in \mathbb{R}^{n \times d}$ . We then obtain the global misalignment vector  $\pmb{v}_{B}$  as the top left singular vector of  $Q$ . The global misalignment vector presumably captures the direction of the representational misalignment between the languages better than the individual (noisy) misalignment vectors  $E(w_{\ell}^{i}) - E(w_{k}^{i})$ . Finally, we modify all vectors  $E(w_{\ell})$  and  $E(w_{k})$ , by subtracting their projections onto the global misalignment direction vector  $v_{B}$ :
$$
f (E (w _ {\ell})) = E (w _ {\ell}) - \cos (E (w _ {\ell}), v _ {B}) v _ {B}.
$$
Language Model BLEU scores often fail to reflect the fluency level of translated texts (Edunov et al., 2019). Hence, we use the language model (LM) of the target language to regularize the crosslingual semantic similarity metrics, by coupling our cross-lingual similarity scores with a GPT language model of the target language (Radford et al., 2018). We expect the language model to penalize translationese, i.e., unnatural word-by-word translations and boost the performance of our metrics.5
In this section, we evaluate the quality of our MT reference-free metrics by correlating them with human judgments of translation quality. These quality
judgments are based on comparing human references and system predictions. We will discuss this discrepancy in  $\S 5.3$ .
Word-level metrics. We denote our word-level alignment metrics based on WMD as MOVERSCORE-NGRAM + ALIGN(EMBEDDING), where ALIGN is one of our two post-hoc cross-lingual alignment methods (CLP or UMD). For example, MOVER-2 + UMD(M-BERT) denotes the metric combining MoverScore based on bigram alignments, with M-BERT embeddings and UMD as the post-hoc alignment method.
Sentence-level metric. We denote our sentence-level metrics as: COSINE + ALIGN(EMBEDDING). For example, COSINE + CLP(LASER) measures the cosine distance between the sentence embeddings obtained with LASER, post-hoc aligned with CLP.
We collect the source language sentences, their system and reference translations from the WMT17-19 news translation shared task (Bojar et al., 2017b; Ma et al., 2018b, 2019), which contains predictions of 166 translation systems across 16 language pairs in WMT17, 149 translation systems across 14 language pairs in WMT18 and 233 translation systems across 18 language pairs in WMT19. We evaluate for X-en language pairs, selecting X from a set of 12 diverse languages: German (de), Chinese (zh), Czech (cs), Latvian (lv), Finnish (fi), Russian (ru), and Turkish (tr), Gujarati (gu), Kazakh (kk), Lithuanian (lt) and Estonian (et). Each language pair in WMT17-19 has approximately 3,000 source sentences, each associated to one reference translation and to the automatic translations generated by participating systems.
We compare with a range of reference-free metrics: ibm1-morpheme and ibm1-pos4gram (Popovic, 2012), LASIM (Yankovskaya et al., 2019), LP (Yankovskaya et al., 2019), YiSi-2 and YiSi-2-srl (Lo, 2019), and reference-based baselines BLEU (Papineni et al., 2002), SentBLEU (Koehn et al., 2007) and  $\mathrm{ChrF} + +$  (Popovic, 2017) for MT evaluation (see §2). The main results are reported on WMT17. We report the results obtained on WMT18 and WMT19 in the Appendix.
The code of these unsupervised metrics is not released, thus we compare to their official results on WMT19 only.
5We linearly combine the cross-lingual metrics with the LM scores using a coefficient of 0.1 for all setups. We choose this value based on initial experiments on one language pair.