Venue: NAACL 2018

# Automated Paraphrase Lattice Creation

for HyTER Machine Translation Evaluation

Marianna Apidianaki\({}^{*\dagger}\), Guillaume Wisniewski\({}^{\dagger}\), Anne Cocos\({}^{*}\) &Chris Callison-Burch\({}^{*}\)

###### Abstract

We propose a variant of a well-known machine translation (MT) evaluation metric, htera.htera, which exploits reference translations enriched with meaning equivalent expressions. The original hter metric relied on hand-crafted paraphrase networks which restricted its applicability to new data. We test, for the first time, htera with automatically built paraphrase lattices. We show that although the metric obtains good results on small and carefully curated data with both manually and automatically selected substitutes, it achieves medium performance on much larger and noisier datasets, demonstrating the limits of the metric for tuning and evaluation of current MT systems.

## 1 Introduction

Human translators and MT systems can produce multiple plausible translations for input texts. To reward meaning-equivalent but lexically divergent translations, MT evaluation metrics exploit synonyms and paraphrases, or multiple references Papineni et al. (2002); Doddington (2002); Denkowski and Lavie (2010); Lo et al. (2012). The hter metric Dreyer and Marcu (2012) relies on massive reference networks encoding an exponential number of correct translations for parts of a given sentence, proposed by human annotators. The manually built networks attempt to encode the set of all correct translations for a sentence, and hter rewards high quality hypotheses by measuring their minimum edit distance to the set of possible translations.

Hytar spurred a lot of enthusiasm but the need for human annotations heavily reduced its applicability to new data. We propose to use an embedding-based lexical substitution model Melamud et al. (2015) for building this type of reference networks and test, for the first time, the metric with automatically generated lattices (hereafter hytera). We show that hytera strongly correlates with hipter with hand-crafted lattices, and approximates the hter score Snover et al. (2006) as measured using post-edits made by human annotators. Furthermore, we generate lattices for standard datasets from a recent WMT Metrics Shared Task and perform the first evaluation of hyter on large and noisier datasets. The results show that it still remains an interesting solution for MT evaluation, but highlight its limits when used to evaluate recent MT systems that make far less errors of lexical choice than older systems.

## 2 The Original HyTER Metric

The hter metric Dreyer and Marcu (2012) computes the similarity between a translation hypothesis and a reference lattice that compactly encodes millions of meaning-equivalent translations. Formally hter is defined as:

\[\textsc{hyter}\left(x,\mathcal{Y}\right)=\operatorname*{arg\,min}_{y\in \mathcal{Y}}\frac{\mathrm{LS}(x,y)}{\mathrm{len}(y)} \tag{1}\]

where \(\mathcal{Y}\) is a set of references that can be encoded as a finite state automaton such as the one represented in Figure 1, \(x\) is a translation hypothesis and LS is the standard Levenshtein distance, defined as the minimum number of substitutions, deletions and insertions required to transform \(x\) into \(y\). We use, in all our experiments, our own im

Figure 1: An English reference sentence enriched with substitutes selected by the embedding-based lexical substitution model.



plementation of Hyter1 that relies on the OpenFST framework (Allauzen et al., 2007). Contrary to the original hyter implementation, we do not consider permutations when transforming \(x\) into \(y\) as previous results (cf. Table 3 in (Dreyer and Marcu, 2012)) have shown that permutations have only very little impact while significantly increasing the computational complexity of hyter computation.2 We also use an exact search rather than a A* search to minimize Equation (1).

Footnote 1: The code is available at [https://bitbucket.org/gwisniewski/hytera/](https://bitbucket.org/gwisniewski/hytera/)

Footnote 2: Note that as permutations of interest can be compactly encoded in a fine-state graph (Kumar and Byrne, 2005), the MOVE operation can be easily considered in our code by applying the substitutions to the permutation lattice rather than to the sentence.

The hjter metric has already been successfully used in MT evaluation but only with handcrafted lattices. To the best of our knowledge, this is the first time it is tested with lattices built automatically.

## 3 Automatic Lattice Creation

We propose an alternative to the costly manual annotation of reference translations which exploits an embedding-based model of lexical substitution proposed by Melamud et al. (2015) (called AddCos). The original AddCos implementation selects substitutes for words in context from the whole vocabulary. Here, we restrict candidate substitutes to paraphrases of words in the Paraphrase Database (PPDB) XXL package (Ganitkevitch et al., 2013).3

Footnote 3: ppdp paraphrases come into packages of different sizes (going from s to XXXL): small packages contain highprecision paraphrases while larger ones have high coverage. All are available from paraphrase.org

AddCos quantifies the fit of substitute word \(s\) for target word \(t\) in context \(C\) by measuring the semantic similarity of the substitute to the target, and the similarity of the substitute to the context:

\[{\rm AddCos}(s,t,C)=\frac{\cos(s,t)+\sum_{c\in C}\cos(s,c)}{|C|+1} \tag{2}\]

The vectors \(s\) and \(t\) are word embeddings of the substitute and target generated by the _skip-gram with negative sampling_ model (Mikolov et al., 2013b,a).4 The context \(C\) is the set of context embeddings for words appearing within a fixed-width window of the target \(t\) in a sentence (we use a window width of 1). The embeddings \(c\) are context embeddings generated by _skip-gram_.5 In our implementation, we train 300-dimensional word and context embeddings over the 4B words in the Annotated Gigaword (AGiga) corpus (Napoles et al., 2012) using the gensim word2vec package Mikolov et al. (2013b,a; Rehurek and Sojka, 2010).6

Footnote 4: For the moment, we focus on individual content words. In future work, we plan to also annotate longer text segments in the references with multi-word PPDB paraphrases.

Each content word token in a sentence is expanded to include all its possible substitutes selected by AddCos in this specific context, and the lattice can take any path from the expanded start token to the expanded end token. We filter the paraphrase candidates according to: a) their PPDB2.0 score, an _out-of-context_ measure of paraphrase confidence which denotes the strength of the relation between the paraphrase and the target word (hereafter, PPDBSc) Pavlick et al. (2015); b) the substitution score assigned to paraphrases in context by the AddCos model (hereafter, AddCosSc), which shows whether the paraphrase is a good fit for the target word _in a specific context_.7 Figure 1 shows the four highest ranked paraphrases proposed by AddCos for words in the English reference sentence: _Matt Damon downplays diversity in filmmaking_. The sentences _Matt Damon underestimates richness in cinematography_ and _Matt Damon belittles pluralism in cinema_. are included among the 48 references encoded in this lattice.

Footnote 7: Our implementation of AddCos with PPDB substitutes can be found at [https://github.com/acocos/lexsub_addocs](https://github.com/acocos/lexsub_addocs)

## 4 Evaluating Hjter with Automatic Substitutions

We assess the quality of hytera to evaluate the quality of MT output both at the sentence and the system level. We first use the setting of Dreyer and Marcu (2012), in Section SS 5.1, to compare the score estimated by hyter and hytera to hjter scores. In Section SS 5.2, we explore whether hytera can reliably predict human translation quality scores from the WMT16 Metrics Shared Task.

 



## 5 Comparing hIter and Hytera

### Open MT NIST Evaluation

To evaluate the performance of hurer, Dreyer and Marcu (2012) examine whether it can approximate the hter score (Snover et al., 2006) that measures the number of edits required to change a system output into its post-edition. hter scores are a good estimate of translation quality and usefulness, but require each translation hypothesis to be corrected by a human annotator. Dreyer and Marcu (2012) show that it can be closely approximated by hter scores. In this section, we reproduce their experiments with hytera to see whether it is possible to use automatically-built rather than hand-crafted references to approximate her scores.

DataFollowing Dreyer and Marcu (2012), we consider a subset of the 'progress set' used in the 2010 Open MT NIST evaluation.8 This corpus is made of 102 Arabic and 102 Chinese sentences. Each sentence is automatically translated into English by five MT systems and these translation hypotheses are post-edited by experienced LDC annotators, allowing us to compute hrer scores. The corpus also contains four references and meaningful annotations which allow for direct comparison to the original hyter.

Footnote 8: The corpus is available from LDC under reference ldc2014t09.

Experimental SettingWe build meaning-equivalent lattices by applying the lexical substitution method described in Section 3 to each of the four references associated with a sentence, and considering the union of the resulting lattices. We report results for two kinds of lattices: lattices encoding all lexical substitutes available for a word in PPDB (allPars) and lattices of substitutes with PPDBSc\(>\)2.3 (allParsFiltered) and AddCoSc\(\geq\)0. As expected, the allPars lattices are much larger than the manual and the filtered lattices (cf. Table 1). In all our experiments, all corpora are down-cased and tokenized using standard Moses scripts. hter scores are computed using terp(Snover et al., 2009).

Sentence Level EvaluationTable 2 reports the correlation between hyter, hytera and hter at the sentence level. We also include as a baseline the correlation with the sentence-level Bleu score, estimated by the arithmetic mean of 1 to 4-gram precisions.9

Footnote 9: More precisely: \({}_{\text{SBleu}}=\frac{1}{4}\cdot\sum_{i=1}^{4}p_{i}\) where \(p_{i}\) is the number of \(i\)-grams that appears both in the reference and in the hypothesis divided by the number of \(i\)-grams in the reference.

In all cases, there is a high correlation between hyter, hytera and hter, significantly higher than the correlation between bleu and hter. This observation shows that replacing the hand-crafted lattices with automatically built ones has only a moderate impact on the hyter metric quality: automatic lattices result in a small drop of the correlation when evaluating hypotheses translated from Chinese, and slightly improve it for the Arabic to English condition. Overall hytera scores are highly correlated with hyter scores (\(\rho=0.766\) for Arabic and \(\rho=0.756\) for Chinese). More importantly, considering the filtered lattices allows to significantly reduce computation time compared to the allPars ones without hurting the quality estimation capacity of the metric.

System Level EvaluationFigure 2 shows how the five MT systems are ranked by the different metrics we consider, when translating from Arabic to English. All metrics rank the systems in the same order, except from hyter with allParsFiltered that only inverts two systems. Note that the tested systems were selected by NIST to cover a variety of system architectures (statistical, rule-based, hybrid) and performances (Dreyer and Marcu, 2012), which makes distinction between them an easy task for all metrics. The benefits of using a metric like hyter, which focuses on the word level, are much clearer in the sentence-based evaluation (Table 2).

### WMT Metrics Evaluation

In our second set of experiments, we explore the ability of hytera to predict direct human judgments at the sentence level using the setting of the WMT16 Metrics Shared Task (Bojar et al., 2016). We measure the correlation between adver

\begin{table}
\begin{tabular}{r c c} \hline \hline  & ar2en & zh2en \\ \hline manual & \(9,454,542\) & \(7.8\times 10^{9}\) \\ allPars & \(1.8\times 10^{27}\) & \(8.5\times 10^{27}\) \\ allParsFiltered & \(10,803\) & \(3.3\times 10^{20}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Median number of references generated by each method.



equacy scores collected on Amazon Mechanical Turk following the method advocated by Graham et al. (2016) and the translation quality estimated by applying hirera to the official WMT reference. Table 3 reports the results achieved by hytera on the six language pairs of the WMT16 Shared Task and its rank among the other metrics tested in the competition.

Hytera obtains medium performance on the WMT16 dataset, which is much larger and noisier than the dataset used for evaluation in Dreyer and Marcu (2012): it is made, for each language, of 560 translations sampled from outputs of all systems taking part in the WMT15 campaign. It is important to note that the hter scores used in the initial hIter evaluation were produced by experienced LDC annotators, while the WMT16 Direct Assessment (DA) adequacy judgments were collected from non-experts through crowd-sourcing Bojar et al. (2016). Hytera achieves higher performance than the sentBLEU baseline in four language pairs (cs/de/ru/tr-en). It obtains slightly lower correlation than sentBLEU for fi-en and ro-en, the language pairs in which correlation was lower for all metrics.

Among the metrics tested at the WMT16 shared task we find combination metrics, and metrics that have been tuned on a development dataset. The metric that performs best for most languages in the segment-level WMT16 evaluation, dpmfcomb, combines 57 individual metrics Yu et al. (2015). Similarly, the second highest ranked metric, MetricsF, combines bleu, Meteor, the alignment-based metric UPF-Combalt Fomicheva et al. (2016), and fluency features. The beer metric, found in fifth position, is a trained evaluation metric with a linear model that combines features capturing character n-grams and permutation trees Stanojevic and Sima'an (2015).

We report the rank of hytera among all metrics (single and combined), and among the single ones. It is important to note that hytera needs no tuning, is straightforward to use and very fast to compute, especially with filtered lattices (on average 6s).

The lower performance of the metric on this dataset is also due to the different nature of the MT systems tested. While in the Dreyer and Marcu (2012) evaluation, the systems came from the 2010 Open MT NIST evaluation and were selected to cover a variety of architectures and performances, the systems that participated in WMT15 are, for the large part, neural MT systems Bojar et al. (2015). As reported by Bentivogli et al. (2016), Neural MT systems make at least 17% fewer lexical choice errors than phrase-based systems, which limits the potential of hytera, primarily focused on capturing correct lexical choice.

## 6 Conclusion

We have proposed a method for automatic paraphrase lattice creation which makes the hter metric applicable to new datasets. We provide the first evaluation of hter on data from a recent

\begin{table}
\begin{tabular}{l c c} \hline \hline  & ar → en & zh → en \\ \hline HyTER & 0.667 & 0.672 \\ HyTERA(allPars) & 0.673 & 0.616 \\ HydTRA(allPersFiltered) & 0.671 & 0.605 \\ Bleu & -0.563 & -0.556 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Correlation, calculated by Spearman’s \(\rho\), between the scores of translation hypotheses estimated by hyter and hter.

Figure 2: Five Arabic to English MT systems scored by different metrics.

 

WMT Metrics Shared task. We show that although the metric achieves high correlation with human judgments of translation quality on small and carefully curated data, with both manual and automatically constructed paraphrase networks, it obtains medium performance on recent WMT data. The lower performance is mainly due to the noisier nature of the data and to the higher quality lexical choices made by current neural MT systems, compared to phrase-based and transfer systems, which limits the potential of the metric for system evaluation and tuning.

In its current form, the paraphrase substitution mechanism supports only lexical substitutions. It would be straightforward to extend the AddCos method to handle multi-word paraphrases by training embeddings for multi-word phrases, keeping in mind that longer substitutions might require restructuring the produced sentences to preserve grammaticality.

## 7 Acknowledgments

We would like to thank Markus Dreyer for sharing with us the original HyTER code.

This work has been supported by the French National Research Agency under project ANR-16-CE33-0013. This material is based in part on research sponsored by DARPA under grant number FA8750-13-2-0017 (the DEFT program) and HR0011-15-C-0115 (the LORELEI program). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA and the U.S. Government.

## References

* Allauzen et al. (2007) Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri. 2007. OpenFst: A General and Efficient Weighted Finite-State Transducer Library. In _Proceedings of the Ninth International Conference on Implementation and Application of Automata, (CIAA 2007)_. Springer, volume 4783 of _Lecture Notes in Computer Science_, pages 11-23.
* Bentivogli et al. (2016) Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase-Based Machine Translation Quality: a Case Study. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Austin, Texas, pages 257-267.
* Bojar et al. (2015) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hobamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. Lisbon, Portugal, pages 1-46.
* Bojar et al. (2016) Ondrej Bojar, Yvette Graham, Amir Kamran, and Milos Stanojevic. 2016. Results of the WMT16 Metrics Shared Task. In _Proceedings of the First Conference on Machine Translation_. Berlin, Germany, pages 199-231.
* Denkowski and Lavie (2010) Michael Denkowski and Alon Lavie. 2010. Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_. Los Angeles, California, USA, pages 250-253.
* Doddington (2002) George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-occurrence Statistics. In _Proceedings of the Second International Conference on Human Language Technology Research_. San Diego, California, pages 138-145.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & hytera & sentBleu & WMT All & WMT Single Metrics \\ \hline  & \(r\) & \(r\) & best \(r\) & rank/15 & best \(r\) & rank/13 \\ \hline cs \(\rightarrow\) en & .599 & .557 & .713 & 10 & .671 & 8 \\ de \(\rightarrow\) en & .498 & .448 & .601 & 6 & .591 & 4 \\ fi \(\rightarrow\) en & .476 & .484 & .598 & 7 & .554 & 5 \\ ro \(\rightarrow\) en & .483 & .499 & .662 & 9 & .639 & 7 \\ ru \(\rightarrow\) en & .525 & .502 & .618 & 10 & .618 & 8 \\ tr \(\rightarrow\) en & .540 & .532 & .663 & 10 & .627 & 8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Pearson correlation between hytera and human judgments at the segment level on WMT16 Metrics Shared Task data on different language pairs. We compare to the scores of the sentBLEU baseline. We report the best correlation achieved by the participating metrics and the rank of hytera among all 15 participants, and among the single 13 metrics left after excluding combined ones.

 

Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-Equivalent Semantics for Translation Evaluation. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Montreal, Canada, pages 162-171.
* Fomicheva et al. (2016) Marina Fomicheva, Noria Bel, Lucia Specia, Iria da Cunha, and Anton Malinovskyi. 2016. CobaltF: A Fluent Metric for MT Evaluation. In _Proceedings of the First Conference on Machine Translation_. Berlin, Germany, pages 483-490.
* Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Atlanta, Georgia, pages 758-764.
* Graham et al. (2016) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. _Natural Language Engineering_ FirstView:1-28.
* Kumar and Byrne (2005) Shankar Kumar and William Byrne. 2005. Local Phrase Reordering Models for Statistical Machine Translation. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_. Vancouver, British Columbia, Canada, pages 161-168.
* Lo et al. (2012) Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully Automatic Semantic MT Evaluation. In _Proceedings of the Seventh Workshop on Statistical Machine Translation_. Montreal, Canada, pages 243-252.
* Melamud et al. (2015) Oren Melamud, Omer Levy, and Ido Dagan. 2015. A Simple Word Embedding Model for Lexical Substitution. In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_. Denver, Colorado, pages 1-7.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_ .
* Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. pages 3111-3119.
* Napoles et al. (2012) Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In _Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX)_. Montreal, Canada, pages 95-100.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In _Proceedings of 40th Annual Meeting of the Association for Computational Linguistics_. Philadelphia, Pennsylvania, USA, pages 311-318.
* Pavlick et al. (2015) Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Beijing, China, pages 425-430.
* Rehrurek and Sojka (2010) Radim Rehrurek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_. Valletta, Malta, pages 45-50.
* Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In _Proceedings of AMTA_. Cambridge, MA, USA, pages 223-231.
* Snover et al. (2009) Matthew Snower, Nitin Madnani, Bonnie J. Dorr, and Richard Schwartz. 2009. Fluency, Adequacy, or HTER?: Exploring Different Human Judgments with a Tunable MT Metric. In _Proceedings of the Fourth Workshop on Statistical Machine Translation_. StatMt '09, pages 259-268.
* Stanojevic and Sima'an (2015) Milos Stanojevic and Khalil Sima'an. 2015. BEER 1.1: ILLC UxA submission to metrics and tuning task. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. Lisbon, Portugal, pages 396-401.
* Yu et al. (2015) Hui Yu, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015. CASICT-DCU Participation in WMT2015 Metrics Task. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. Lisbon, Portugal, pages 417-421.