A Poor Man’s Translation Memory Using Machine Translation Evaluation Metrics
Michel Simard †Atsushi Fujita ‡† National Research Council Canada
michel.simard@nrc.ca‡Future University Hakodate
fujita@fun.ac.jp

Evaluation Methodology

Motivations
Much recent work on post-editing and computer-assisted translation (CAT) ignores the fact that mosttranslators are already using CAT tools, the most common of which are TM systems. In the shortterm, new tools are not going to supplant TMs, but rather complement them.
I The problem for researchers who wish to experiment with TM technology is that existing systems are
big and unwieldy: their basic functionality is often concealed behind a thick layer of GUI, and theiralgorithms are not documented (they are trade secrets). This makes it difficult to use commercialsystems in a research setting, or even to reverse-engineer their functionality.
I The obvious alternative is to re-implement the TM functionality from scratch.

ITranslation Memory
Conceptually, a Translation Memory consists of:I a database D, containing pairs hs, ti, where s is source-language segment of text (typically asentence) and t is its translation in the target language;I a similarity function f ; andI a filtering threshold αGiven a new sentence to translate q (the query), the core TM functionality consists in finding the best
match for q in D, i.e. the pair hŝ, t̂i with maximum similarity x = f (q, ŝ); if x ≥ α, then the system
outputs the target-language counterpart t̂ of ŝ, otherwise nothing.
Similarity Function f measures the similarity between two source-language strings. Typically, it
produces a value between 0 and 1, where 0 means “completely different” and 1 means “identical”; α
can then be in the range [0, 1]. It is generally acknowledged that commercial TM systems use variantsof the Levenshtein distance, e.g.:count edits(q, s)
fLevenshtein(q, s) = 1 − min 1,|q|

We opt for the evaluation approach proposed in Simard & Isabelle (2009) , in which TM systems areevaluated as if they were MT systems: test sentences are submitted to the TM, with the filtering
threshold α set to zero, thus effectively inhibiting output filtering. The target segments of the best matches
are then compared to the reference translations, using standard MT evaluation metrics. In practice, in thisstudy, we use the same metrics that were used as similarity functions.
Data
We perform experiments to assess theperformance of each MT evaluation metricas TM similarity function. Experiments
were done using English, French, German
and Spanish data, drawn from Europarl
v.6 (Koehn, 2005), the OPUS corpus(Tiedemann, 2009) and the JRC-Acquis
v.2.2 (Steinberger, 2006). From eachcorpus, we randomly sampled 1000 pairsof segments, to be used as test data; therest was used to build translation
memories.

Implementing the core TM functionality requires that we come up with a similarity function. As it turns
out, one sub-field of MT research that has churned out many such functions is MT evaluation: Many (ifnot all) of the MT evaluation metrics proposed in recent years rely on measuring the similarity between
a machine translation output and one or more reference translations. In this study, we examine fivedifferent evaluation metrics:I WER – Word-error rate is based on word-level Levenshtein distance. As far as anyone knows, thisis essentially what is used in commercial TM systems, and serves as baseline for this study.
I BLEU – Papineni et al. (2002) : based on n-gram precision, it implements the idea of accounting
separately for adequacy (low-order n-grams) and fluency (high-order n-grams).
I NIST – Doddington (2002) : Adds a notion of IR-style relevance to the mix.
We also consider Meteor, under two different conditions:
I VMeteor (“Vanilla” Meteor) – Banerjee & Lavie (2005): considers lexical recall, while
de-emphasizing match length.
I Meteor – Denkowski & Lavie (2011) : linguistic resources (stemmer, WordNet, paraphrases) bring us
closer to semantic similarity.

Language

Europarl

en-de
en-es
en-fr
de-en
es-en
fr-en
en-de
en-es
en-fr
de-en
es-en
fr-en
en-fr
en-fr

EMEA
JRC-Acquis

en-fr
en-es
en-de
ECB
en-fr
en-es
en-de
EMEA
en-fr
JRC-Acquis en-fr

Evaluation Metric
WER BLEUNIST VMeteor Meteor
WER BLEU
BLEU Meteor Meteor
WER BLEUNIST VMeteor VMeteor
WER BLEUNIST VMeteor VMeteor
WER BLEUNISTMeteor Meteor
WER VMeteor Meteor Meteor Meteor
WER BLEUNISTMeteor Meteor
WER BLEU
BLEU VMeteor VMeteor
WER BLEU
BLEU VMeteor VMeteor
WER BLEU
BLEU VMeteor VMeteor
WER BLEU
BLEU Meteor Meteor
WER VMeteor VMeteor Meteor Meteor
WER BLEU
BLEU Meteor Meteor
WER BLEU
BLEU VMeteor VMeteor
WER BLEU
BLEU VMeteor VMeteor

Length ratios between source language queryand TM best match indirectly impacts targetlanguage ratio as well. It thus plays a potentially
important role in measured performance.
Similarity Function Source Ratio Target Ratio|ŝ|/|q||t̂|/|r |WER
0.85 ± 0.18 0.88 ± 0.58
BLEU
1.01 ± 0.60 1.04 ± 0.91
NIST
1.03 ± 0.15 1.06 ± 0.63
VMeteor1.31 ± 0.85 1.43 ± 2.01
Meteor
1.36 ± 0.85 1.48 ± 2.07

hŝ, t̂i = arg max fX (q, s)

Performance differs the most at
high-coverage levels, i.e. when TM
outputs are proposed even for
low-similarity matches. In a real-life
TM application, weakly matchingsegments are seldom useful: This is
the kind of material that the translator
typically does not want to see.
I in the low-coverage areas, where
only the best matching segments fromthe TM are retained, all metrics display
very comparable performances.
I80
60
40
200020

40

60

80

100

Coverage [%words]

Custom Paraphrase Tables for Meteor

When WER is used to measureperformance, WER always
comes out as the best similarity
function;
I BLEU generally exhibits similar
behaviour.
I VMeteor and Meteor always
prefer one of the Meteor family;both metrics also always agreewith one another, usually
preferring VMeteor when English
is the source language andMeteor when English is target.
I NIST has low self-esteem: this isbecause local optimization(finding the best match for eachsentence) doesn’t guarantee a
global maximum.

One way of optimizing the performance of Meteor as a TM similarity function is to provide itwith domain-specific paraphrases.
I CMeteor (“Custom” Meteor): Using the method of Fujita et al. (2012), we extract
paraphrases from each TM to create domain-specific paraphrase tables, and use thesewith Meteor instead of the standard tables (no parameter tuning).
I In practice, domain-specific paraphrases do not lead to measurable gains or losses in
performance: the in-domain paraphrases theoretically allow finding more useful matches
in the TM, but the translation of these are often also realized as target-language
domain-specific paraphrases, which are not properly acknowledged by the evaluation
metrics.
Example 1: QueryMeteor
CMeteor
WER
BLEU
NIST
VMeteorThis is the process we are commencing.
I suggest that we perhaps continue the work we have started.
This is the point at which we must start.
This is the stage we are at.
This is the stage we are at.
This is the stage we are at.
We are in the process of revising this regulation.

Example 2: Query

A lysodren patient card is included at the end of thisleaflet.
At the end of this leaflet.
Detailed instructions for subcutaneous injection are provided at the end of this leaflet.
Listed at the end of this leaflet.
Ingredients are listed at the end of this leaflet.
Listed at the end of this leaflet (see section 6).
At the end of this leaflet.

MeteorCMeteorWERBLEUNISTVMeteor
Conclusions

15012090603000306090120150

Length of query

hs,ti∈D

BLEU and NIST tend to produce TM best matches whose source segment length is very close to that ofthe query.
I WER naturally favors segments that are much shorter than the query. On the target side, shorter TM
matches such as those produced by the WER similarity function will be penalized at evaluation time by
the BLEU and NIST brevity penalties.
I Although the Meteor metrics (Meteor and VMeteor) de-emphasize length-similarity, they tend to produce
source segments that are much longer than the query, which will naturally be penalized by
precision-based evaluation metrics.
I

http://nrc-cnrc.gc.ca

100

TM query:match size ratios

We implement an exhaustive search strategy : for every pair hs, ti in the TM D, measure thesimilarity between q and s, using MT evaluation metric X ’s similarity function:

query q is used in place of the reference translation and s is used in place of the machine translation
output.
I BLEU and NIST do not behave well when applied to single sentences: we use smoothed versions ofthese functions, as in Lin & Och (2004).
I NIST and WER do not produce values strictly between 0 and 1, their value needs to be normalized.
While it is sometimes possible to use public domain MT evaluation software directly (e.g. Meteor,VMeteor), it is often easier and more efficient to reimplement the distance functions based on thepublished descriptions (BLEU, WER and NIST).

EuroparlTM (“Train”)
Testsegments words words
1.8M
50.4M 28 817
1.8M
49.2M 28 365
1.7M48.0M 26 715
194k
5.7M 30 471
114k
3.1M 28 054
111k
3.0M 27 426753k
9.1M 16 514
329k
6.9M 19 260

IImplementation

ILanguage

This table reports which similarity function f (q, s) performs best, according to each MT evaluation metric.
Who wins the race depends heavily on who is keeping the score!

ECB

MT Evaluation Metrics

CorpusResults

CorpusTM filtering has a direct impact on performance: As threshold α is set higher, less queriesfind matches in the TM, but performance on the filtered material improves.

WERImplement the core Translation Memory (TM) functionality: given a source-language sentence, findthe best matching translation in a bilingual corpus. Our goal is to build a reasonable research
prototype, focusing on output quality rather than time/space efficiency.

TM performance VS coverage

Lengh of best match

The Goal

MT evaluation metrics can be used effectively as translation memory similarity functions.
Each metric has its own characteristics and potential benefits, but evaluation isproblematic.
I Metrics based on n-gram precision such as BLEU and NIST are less computationallyexpensive than classic edit-distance-based metrics such as WER, or metrics that relyon linguistic resources, such as Meteor. In practice, they are easy to implement andproduce results comparable to WER, especially in high-similarity situations, where it
counts for real-life TM usage.
I Customizing linguistic resources such as paraphrase tables could help in better
leveraging the contents of the TM when appropriate metrics are used, such as Meteor orTERp (Snover et al. 2009). Extracting domain-specific paraphrases is one possible
avenue, but in a TM perspective, it would be interesting to extend similarity to othersemantic relations besides synonymy, e.g. antonymy, hyponymy, etc.
I When evaluating the performance of TM systems using MT evaluation metrics, ingeneral, we find that whichever metric is used as TM similarity function will likely obtain
the best score under that evaluation metric. This suggests that existing MT evaluation
metrics are not appropriate for evaluating TM performance. In fact, it is unclear
whether it is actually possible to measure TM performance in an unbiased way usingfully automatic methods. Human-based evaluation may well be the only credible
alternative, and is what we plan to resort to in future experiments.

http://paraphrasing.org/˜fujita/