A Study of Translation Edit Rate with Targeted Human Annotation
Matthew Snover and Bonnie Dorr
Institute for Advanced Computer Studies
University of MarylandCollege Park, MD 20742{snover,bonnie}@umiacs.umd.edu

Richard Schwartz, Linnea Micciulla, and John Makhoul
BBN Technologies
10 Moulton Street
Cambridge, MA 02138{schwartz,lmicciul,makhoul}@bbn.com

Abstract
We examine a new, intuitive measurefor evaluating machine-translation output
that avoids the knowledge intensivenessof more meaning-based approaches, andthe labor-intensiveness of human judgments. Translation Edit Rate (TER) measures the amount of editing that a human would have to perform to changea system output so it exactly matches areference translation. We show that thesingle-reference variant of TER correlates
as well with human judgments of MT
quality as the four-reference variant of
BLEU. We also define a human-targeted
TER (or HTER) and show that it yields
higher correlations with human judgments
than BLEU—even when BLEU is given
human-targeted references. Our results indicate that HTER correlates with humanjudgments better than HMETEOR andthat the four-reference variants of TER
and HTER correlate with human judgments as well as—or better than—a second human judgment does.

1

Introduction

Due to the large space of possible correct translations, automatic machine translation (MT) has
proved a difficult task to evaluate. Human judgments of evaluation are expensive and noisy. Many
automatic measures have been proposed to facilitate fast and cheap evaluation of MT systems, themost widely used of which is BLEU (Papineni etal., 2002), an evaluation metric that matches ngrams from multiple references. A variant of thismetric, typically referred to as the “NIST” metric,
was proposed by Doddington (Doddington, 2002).
Other proposed methods for MT evaluation include
METEOR (Banerjee and Lavie, 2005), which uses
unigram matches on the words and their stems,
and a linear combination of automatic MT evaluation methods along with meaning-based features
for identifying paraphrases (Russo-Lassner et al.,2005).1
We define a new, more intuitive measure of“goodness” of MT output—specifically, the numberof edits needed to fix the output so that it semantically matches a correct translation. In a less expensive variant, we attempt to avoid the knowledge intensiveness of more meaning-based approaches, andthe labor-intensiveness of human judgments. We
also seek to achieve higher correlations with humanjudgments by assigning lower costs to phrasal shifts
than those assigned by n-gram-based approaches
such as BLEU.
Recently the GALE (Olive, 2005) (Global Autonomous Language Exploitation) research program
introduced a new error measure called Translation
Edit Rate (TER)2 that was originally designed to1

One variant of the meaning-based approach incorporates
the translation error rate described in this paper. We adopt a
simpler evaluation paradigm that requires no meaning-based
features, but still achieves correlations that are better than theexisting standard, BLEU.
2Within the GALE community, the TER error measure isreferred to as Translation Error Rate , derived from the Word
Error Rate (WER) metric in the automatic speech recognition
community. The name is regrettable for its implication that itcount the number of edits (including phrasal shifts)
performed by a human to change a hypothesis sothat it is both fluent and has the correct meaning.
This was then decomposed into two steps: defining
a new reference and finding the minimum numberof edits so that the hypothesis exactly matches oneof the references. This measure was defined suchthat all edits, including shifts, would have a costof one. Finding only the minimum number of edits, without generating a new reference is the measure defined as TER; finding the minimum of edits
to a new targeted references is defined as humantargeted TER (or HTER). We investigate both measures and also present a procedure to create targetedreferences where a fluent speaker of the target language creates a new reference translation targeted
for this system output by editing the hypothesis until it is both fluent and has the same meaning as thereference(s).
The next section describes work related to ourMT evaluation approach. Following this we define
TER and its human-targeted variant, HTER. We thenpresent our experimental design and results of ourexperiments, comparing TER and HTER to BLEU
and METEOR (and their human-targeted variants,
HBLEU and HMETEOR). We compare these measures against human judgments of the fluency andadequateness of the system output. Finally, we conclude with a summary of results and future work.

2

Related Work

The first attempts at MT evaluation relied on purely
subjective human judgments (King, 1996). Laterwork measured MT error by post editing MT output and counting the number of edits, typically measured in the number of keystrokes to convert thesystem output into a “canonical” human translation(Frederking and Nirenburg, 1994). Attempts havebeen made to improve MT performance by automatic post-editing techniques (Knight and Chander,1994). Post editing measures have also been shown
effective for text summarization evaluation (Mani etal., 2002) and natural language generation (Sripada
et al., 2004).
is the definitive MT measure. The authors make no such claim,and have adopted the name Translation Edit Rate for use in thispaper and the wider community.

When developing MT systems, a purely automatic measure of accuracy is preferred for rapid
feedback and reliability. Purely human based evaluation metrics fail in this regard and have largely been
replaced by purely automatic MT evaluations. Automatic MT evaluation has traditionally relied upon
string comparisons between a set of reference translations and a translation hypothesis. The quality ofsuch automatic measures can only be determined bycomparisons to human judgments. One difficultyin using these automatic measures is that their output is not meaningful except to compare one systemagainst another.
BLEU (Papineni et al., 2002) calculates the scoreof a translation by measuring the number of ngrams, of varying length, of the system output that
occur within the set of references. This measure has
contributed to the recent improvement in MT systems by giving developers a reliable, cheap evaluation measure on which to compare their systems.
However, BLEU is relatively unintuitive and relies
upon a large number of references and a large number of sentences in order to correlate with humanjudgments.
METEOR (Banerjee and Lavie, 2005) is an evaluation measure that counts the number of exact word
matches between the system output and reference.
Unmatched words are then stemmed and matched.
Additional penalities are assessed for reordering thewords between the hypothesis and reference. Thismethod has been shown to correlate very well withhuman judgments.
An MT scoring measure that uses the notion of
maximum matching string (MMS) has been demonstrated to yield high correlations with human judges(Turian et al., 2003). The MMS method is similar to
the approach used by TER, in that it only allows astring to be matched once, and also permits string
reordering. The MMS approach explicitly favors
long contiguous matches, whereas TER attempts tominimize the number of edits between the reference
and the hypothesis. TER assigns a lower cost tophrasal shifts than MMS, and does not explicitly favor longer matching strings.

3

Definition of Translation Edit RateTER is defined as the minimum number of edits
needed to change a hypothesis so that it exactly
matches one of the references, normalized by theaverage length of the references. Since we are concerned with the minimum number of edits needed tomodify the hypothesis, we only measure the numberof edits to the closest reference (as measured by the
TER score). Specifically:TER =# of edits
average # of reference words

Possible edits include the insertion, deletion, andsubstitution of single words as well as shifts of word
sequences. A shift moves a contiguous sequenceof words within the hypothesis to another location
within the hypothesis. All edits, including shifts ofany number of words, by any distance, have equalcost. In addition, punctuation tokens are treated as
normal words and mis-capitalization is counted asan edit. 3
Consider the reference/hypothesis pair below,
where differences between the reference and hypothesis are indicated by upper case:
REF: SAUDI ARABIA denied THIS WEEK
information published in the
AMERICAN new york times
HYP: THIS WEEK THE SAUDIS denied
information published in the
new york times

Here, the hypothesis (HYP) is fluent and means
the same thing (except for missing “American”) as
the reference (REF). However, TER does not consider this an exact match. First, we note that thephrase “this week” in the hypothesis is in a “shifted”
position (at the beginning of the sentence rather thanafter the word “denied”) with respect to the hypothesis. Second, we note that the phrase “Saudi Arabia”
in the reference appears as “the Saudis” in the hypothesis (this counts as two separate substitutions).
Finally, the word “American” appears only in thereference.

If we apply TER to this hypothesis and reference,
the number of edits is 4 (1 Shift, 2 Substitutions, and4= 31%. BLEU
1 Insertion), giving a TER score of 13
also yields a poor score of 32.3% (or 67.7% whenviewed as the error-rate analog to the TER score) on
the hypothesis because it doesn’t account for phrasal
shifts adequately.
Clearly these scores do not reflect the acceptability of the hypothesis, but it would take humanknowledge to determine that the hypothesis semantically matches the reference. A solution to this, using
human annotators is discussed in Section 4.
Optimal calculation of edit-distance withmove operations has been shown to be NPComplete(Shapira and Storer, 2002), causing us to
use the following approximation to calculate TER
scores. The number of edits for TER is calculated
in two phases. The number of insertions, deletions,and substitutions is calculated using dynamic
programming. A greedy search is used to find theset of shifts, by repeatedly selecting the shift thatmost reduces the number of insertions, deletions
and substitutions, until no more beneficial shifts
remain. Note that a shift that reduces the number of
insertions, deletions, substitutions by just one has no
net reduction in cost, due to the cost of 1 for the shiftitself. However, in this case, we still adopt the shift,because we find that the alignment is more correct
subjectively and often results in slightly lower edit
distance later on. Then dynamic programmingis used to optimally calculate the remaining edit
distance using a minimum-edit-distance (where
insertions, deletions and substitutions all have cost1). The number of edits is calculated for all of thereferences, and the best (lowest) score is used. The
pseudo-code for calculating the number of edits isshown in Algorithm 1.
The greedy search is necessary to select the setof shifts because an optimal sequence of edits (withshifts) is very expensive to find. In order to further reduce the space of possible shifts, to allow for
efficient computation, several other constraints are
used:

3

It is possible that this measure could be improved by a more
careful setting of edit weights, but this paper is concerned with
exploring the current TER measure in use. The choice for shifts
to be cost one, regardless of length or distance, seems somewhat
arbitrary and other cost measures for shifts might yield higher
correlations with human judgments.

1. The shifted words must match the referencewords in the destination position exactly.
2. The word sequence of the hypothesis in theAlgorithm 1 Calculate Number of Edits
input: HYPOTHESIS h
input: REFERENCES R
E←∞
for all r ∈ R doh0 ← he←0repeatFind shift, s, that most reduces min-editdistance(h0 , r)if s reduces edit distance thenh0 ← apply s to h0
e←e+1
end ifuntil No shifts that reduce edit distance remaine ← e+ min-edit-distance(h0 , r)if e < E thenE←e
end ifend forreturn E
original position and the corresponding reference words must not exactly match.
3. The word sequence of the reference that corresponds to the destination position must be misaligned before the shift.
As an example, consider the following reference/hypothesis pair:
REF: a b c d e f
HYP: a

d e

c
b c f

The words “b c” in the hypothesis can be shiftedto the left to correspond to the words “b c” in thereference, because there is a mismatch in the current
location of “b c” in the hypothesis, and there is amismatch of “b c” in the reference. After the shift
the hypothesis is changed to:
REF: a b c d e f c
HYP: a b c d e f

The minimum-edit-distance algorithm is O(n2 )in the number of words. Therefore we use a beam
search, which reduces the computation to O(n), sothat the evaluation code works efficiently on longsentences.
TER as defined above, only calculates the numberof edits between the best reference and the hypoth-esis. It most accurately measures the error rate ofa hypothesis when that reference is the closest possible reference to the hypothesis. While predetermined references can be used to measure the errorrate, the most accurate results require custom references generated with the assistance of a human annotator.

4

Human-targeted Translation Edit Rate

As stated earlier, the acceptability of a hypothesis is
not entirely indicated by the TER score, which ignores notions of semantic equivalence. This section
describes an approach that employs human annotation to make TER be a more accurate measure oftranslation quality. 4
Our human-in-the-loop evaluation, or HTER (for
Human-targeted Translation Edit Rate), involves aprocedure for creating targeted references. In order to accurately measure the number of edits necessary to transform the hypothesis into a fluent target language (English in our experiments) sentencewith the same meaning as the references, one must
do more than measure the distance between the hypothesis and the current references. Specifically, amore successful approach is one that finds the closest possible reference to the hypothesis from thespace of all possible fluent references that have thesame meaning as the original references.
To approximate this, we use human annotators,
who are fluent speakers of the target language, togenerate a new targeted reference. We start with
an automatic system output (hypothesis) and one ormore pre-determined, or untargeted, reference translations. They could generate the targeted referenceby editing the system hypothesis or the original reference translation. We find that most editors edit
the hypothesis until it is fluent and has the same
meaning as the untargeted reference(s). We thencompute the minimum TER (using the technique described above in Section 3) using this single targetedreference as a new human reference. The targetedreference is the only human reference used for the
purpose of measuring HTER. However, this reference is not used for computing the average reference
length.5
4

The human-in-the-loop variant of TER is the one that will
be used in the GALE MT Evaluations this year.
5
The targeted reference is not used to compute the averageWithin this approach it is possible to reduce thecost for development within a system. Specifically,
it is not necessary to create new references for each
run; for many systems, most translations do notchange from run to run. New targeted references
only need to be created for those sentences whosetranslations have changed. Moreover, we probably
only need to create new references on sentences witha significantly increased edit rate (since the last run).
The human annotation tool that we used for our
experiments displays all references and the system
hypothesis. In the main window, the annotation toolshows where the hypothesis differs from the best reference, as determined by applying TER. The tool
also shows the minimum number of edits for the“reference in progress.” In addition, the surroundingreference sentences in the document are also shownto give the annotator additional context. We foundthat annotators took an average of 3 to 7 minutes per
sentence to provide a good targeted reference. The
time was relatively consistent over the 4 annotators,
but we believe this time could be reduced by a better
annotation tool.
An example set of references, hypothesis andresulting human-targeted reference are shown
below:
Ref 1:
Ref 2:

Hyp:

Targ:

The expert, who asked not to be
identified, added, "This depends
on the conditions of the bodies."
The experts who asked to remain
unnamed said, "the matter isrelated to the state of thebodies."
The expert who requested
anonymity said that "thesituation of the matter is linked
to the dead bodies".
The expert who requested
anonymity said that "the matteris linked to the condition of thedead bodies".

Note that the the term “dead bodies,” which is acorrect translation from the Arabic, disagrees withboth references, but is present in the hypothesis andtargeted reference. The annotator can reach suchconclusions by using the surrounding context andworld knowledge.
reference length, as this would change the denominator in the
TER calculation, and crafty annotators could favor long targetedreferences in order to minimize HTER.

5

Experimental Design

In our experiments, we used the results of two MT
systems, from MTEval 2004, which we call S1 andS2 . According to the distribution requirements set
by NIST for the shared metrics data, these systems remain anonymous. According to the MTEval 2004 metrics, S1 is one of the lower performing systems and S2 is one of the best systems. We
used 100 sentences from the MTEval 2004 Arabic
evaluation data set. Each sentence had four reference translations, which we will henceforth refer to
as untargeted references, that were used to calculate BLEU scores in the MTEval 2004 evaluation.
The sentences were chosen randomly from the set ofsentences that had also been annotated with humanjudgments of fluency and adequacy for the MTEval
2004 evaluation.6
In order to create the targeted references for
HTER, four monolingual native English annotators
corrected the system output. Two annotators wereassigned to each sentence from each system.7 Annotators were coached on how to minimize the edit
rate.8 while preserving the meaning of the reference translations. Bilingual annotators could possibly give more accurate results, since they would
be better able to understand the source sentence
meaning, but are much more expensive to hire andwould likely take more time to complete the task.
Given that four references translations are available
to convey the meaning, monolingual annotators area cheaper, more readily available alternative.
If the annotators were told to simply edit the system output, without any attempt to minimize HTER,
they would lack incentive to find a good targeted reference and would favor transforming the system output using the least effort possible (e.g., by copying
one of the original references). In this case muchhigher HTER scores result, which do not then cor6
We used the community standard human judgments from
MTEval 2004. For time and cost reasons we did not generate
our own independent human judgments.
7
Two annotators were used in this study so that issues ofannotator consistency could be examined. In practice one annotator per sentence would be adequate to create the targetedreferences.
8
The coaching given to the annotators in order to minimize
HTER, consisted mostly of teaching them which edits were considered by TER. The annotators also consulted with each otherduring training to compare various techniques they found tominimize HTER scores.

respond to calculating the minimum number of edits
required to transform the system output into a fluent
sentence with the same meaning as the references.
After the initial generation of the targeted references, another pass of annotation was performed toensure that the new targeted references were sufficiently accurate and fluent. During this second
pass, other annotators, other HTER annotators who
worked on different data, checked (and corrected) all
targeted references for fluency and meaning without
exposure to the system output. On average, this second pass changed 0.63 words per sentence. This correction pass raised the average HTER score as theannotators had typically erred in favor of the system output. The corrections in this pass consisted
almost entirely of fixing small errors, that were aresult of annotator oversight, such as verb or tense
agreement.

6

Results

Table 1 shows that the HTER (i.e., TER with one
human-targeted reference) reduces the edit rate by
33% relative to TER with 4 untargeted references.
Substitutions were reduced by the largest factor, presumably because words were often judged synonymous. In both TER and HTER, the majority of theedits were substitutions and deletions. Because TER
and HTER are edit-distance metrics, lower numbers indicate better performance. In previous pilot studies with more experienced annotators, HTER
yielded an even higher reduction of 50%.9
Condition
TER: 4 UnTarg Refs
HTER: 1 Targ Ref

Ins
4.6
3.5

Del
12.0
10.5

Sub
25.8
14.6

Shift
7.2
4.9

Total
49.6
33.5

Table 1: Untargeted (TER) and Human-targeted(HTER) Results: Average of S1 and S2
In an analysis of shift size and distance, we foundthat most shifts are short in length (1 word) and are
by less than 7 words. We also did a side-by-side
9

The annotators in this study were recent additions to ourannotation group, as opposed to the veteran annotators we usedin our pilot study. In addition the annotators in this study wereplaced under more strict time constraints, encouraging them not
to spend more than five minutes on a single sentence. This
tradeoff of annotation speed and annotation quality is important
to consider as HTER is considered for use in community-wide
MT evaluations.

comparison of HTER with BLEU and METEOR(see Table 2)10 and found that human-targeted references lower edit distance overall but the TER measure is aided more than BLEU by targeted references: HTER yields a reduction of 33% whereas
HBLEU yields a reduction of 28%. It is possiblethat performance of HBLEU is biased, as the targeted references were designed to minimize TER
scores rather than BLEU scores. In the case of
BLEU and METEOR (and the human-targeted variants), we must subtract the score from 1 to get numbers that are comparable to TER (and its humantargeted variant). That is, lower numbers indicate
better scores for all three measures.
We also did a study of the correlations among
TER, HTER, BLEU, HBLEU, METEOR, HMETEOR and Human Judgments, as shown in Table 3.
The table shows the Pearson Coefficients of correlation between each of the evaluation metrics that wemeasured. Note that the correlations along the diagonal are not always 1, as several of these are theaverages of multiple correlations.
TER, BLEU and METEOR are abbreviated as T,
B and M, respectively. T(1) refers to the applicationof TER with only one untargeted reference. (The reported correlation refers to an average of the 4 correlations, one for each reference.)
B(1) and M(1) are analogously computed for
BLEU and METEOR, respectively. T(4), B(4), andM(4) refer to the score computed using all 4 untargeted references for TER, BLEU, and METEOR, respectively. HT, HB and HM refer to the applicationof TER, BLEU, and METEOR, respectively, with
only one human-targeted reference. (The reported
correlation refers to an average of the 2 correlations,
one for each human-targeted reference.)
Human Judgments refer to the average of fluency
and adequacy judgments from both human judges.
Sentences from both systems were used for a totalof 200 data points—no significant differences were
found when only one of the systems was used. The
values for the evaluation measures decrease for better values, whereas the values for human judgments
increase for better values; however, for clarity, we
report only the magnitude, not the sign of the corre10

The 4 untargeted references were not used in calculating
the HTER, HBLEU, HMETEOR metrics.

Condition
BLEU: 4 UnTarg Refs
HBLEU: 1 Targ Ref
METEOR: 4 UnTarg Refs
HMETEOR: 1 Targ Ref
TER: 4 UnTarg Refs
HTER: 1 Targ Ref

S1
73.5
62.2
46.0
33.9
53.2
39.7

S2
62.1
45.0
39.2
22.1
46.0
27.2

Average
67.8
53.6
42.1
28.0
49.6
33.5

Table 2: Untargeted and Human-targeted Scores: BLEU, HBLEU, METEOR, HMETEOR, TER, HTER
Measure
T(1)
T(4)
HT
B(1)
B(4)
HB
M(1)
M(4)
HM
HJ

T(1)
0.737
0.792
0.606
0.473
0.518
0.502
0.555
0.586
0.482
0.390

T(4)
1.000
0.789
0.521
0.606
0.624
0.652
0.727
0.618
0.539

HT
0.929
0.457
0.565
0.794
0.623
0.675
0.802
0.630

B(1)
0.709
0.737
0.535
0.479
0.488
0.433
0.325

B(4)
1.000
0.687
0.553
0.596
0.545
0.391

HB
0.919
0.607
0.643
0.761
0.579

M(1)
0.845
0.888
0.744
0.493

M(4)
1.000
0.806
0.550

HM
0.945
0.602

HJ
1.000

Table 3: Correlations among TER (T), BLEU (B), METEOR (M), human variants (HT, HB, HM), andHuman Judgments (HJ)
lations.
The correlation between HTER (HT) and Human
Judgments (HJ) was very high with a Pearson coefficient of 0.630, exceeding the correlation of allother metrics. T(1) and T(4) are both well correlated with HT (r=0.606 and r=0.789, respectively),
and are ideal for system development where the costof HTER is prohibitive. In addition, T(1) is shownto correlate as well with human judgments as B(4)(r=0.390 and r=0.391, respectively), indicating that
equally valuable results can be gained with TER using one fourth of the number of references—even
without human targeting. While M(1) and M(4)correlate better with Human Judgments than T(1)and T(4), respectively, neither M(1) nor M(4) (northe human-targeted HM) correlate as well as withhuman judgments as HTER (0.493/0.550/0.602 vs.
0.630).
In an analysis of standard deviation due to annotator differences (Table 4), we observed that TER ismuch less sensitive to the number of references than
BLEU and also that the standard deviation decreased
somewhat with targeted references. To computevariance among annotators, we first compute thevariances from the mean for each sentence. Then wethen took the average (weighted by length) acrosssentences and took the square root. Table 5 shows
the means for each annotator. The standard deviation of these numbers is 2.8%.
We also examined correlations between the two
human judges and the evaluation metrics (see Table 6). HJ-1 and HJ-2 refer to the two sets of humanjudgments, each of which is the average of fluency
and adequacy. Correlating the two sets of humanjudgments against each other shows a correlation of
only 0.478, less than the correlation of the averageof the judges with HTER, or even the correlation of
HTER with either individual human judgment set(r=0.506 and r=0.575). In fact, even the TER(4)measure (with untargeted references) correlates almost as well with each of the human judgments(0.461 and 0.466) as each of the humans against
each other (0.478).
That HTER correlates better with average humanjudgments than individual human judgments correlate with each other may seem paradoxical at first.
This is due to the fact that the individual judgments
actually have a high degree of disagreement. The average of many individual judgments actually forms
the better benchmark of performance. Much of thedilemma arises in the difficulty of assigning a subjective numerical judgment to a translation, and the

B:
B:
B:
HB:
M:
M:
M:
HM:
T:
T:
T:
HT:

Condition
4 UnTarg Refs
3 of 4 UnTarg Refs
1 of 4 UnTarg Refs
1 of 2 Targ Refs
4 UnTarg Refs
3 of 4 UnTarg Refs
1 of 4 UnTarg Refs
1 of 2 Targ Refs
4 UnTarg Refs
3 of 4 UnTarg Refs
1 of 4 UnTarg Refs
1 of 2 Targ Refs

Mean
67.8
71.0
83.2
53.6
42.1
43.3
49.6
28.0
49.6
51.0
57.0
33.5

Std Dev
5.7
8.9
10.1
3.4
7.9
6.7
3.4
8.1
6.8

Table 4: Standard Deviation Due to Annotator Differences for TER (T), BLEU (B), METEOR (M), andhuman variants (HT, HB, HM)
Annotator
1
2
3
4

Mean HTER
31.5
36.4
31.1
29.9

Table 5: Variance Among Annotators
Measure
HJ-1
HJ-2

T(1)
0.332
0.339

T(4)
0.461
0.466

HT
0.506
0.575

B(1)
0.270
0.288

B(4)
0.341
0.331

HB
0.461
0.532

M(1)
0.446
0.403

M(4)
0.502
0.446

HM
0.511
0.525

HJ-1
1.000
0.478

HJ-2
1.000

Table 6: Correlations among TER (T), BLEU (B), METEOR (M), human variants (HT, HB, HM), andIndividual Human Judgments
high degree of variability it entails. Given its correlation to the average human judgment, HTER seems
to be a possible substitute for such subjective humanjudgments.

7

Conclusions and Future Work

We have described a new measure for the ’goodness’of MT, HTER, which is less subjective than pure human judgments. This method is expensive, in that it
requires approximately 3 to 7 minutes per sentence
for a human to annotate. While the studies presented
here use four reference translations to create a targeted reference, fewer references would likely beadequate, reducing the cost of the method, relative
to methods that require many reference translations.
HTER is not suitable for use in the development cycle of an MT system, although it could be employedon a periodic basis, but does appear to be a possible substitute for subjective human judgments of MT
quality.
We have shown that TER is adequate for research
purposes as it correlates reasonably well with humanjudgments and also with HTER. However it gives anoverestimate of the actual translation error rate. Targeted references mitigates this issue. When compared with TER with 4 untargeted references, theedit rate with HTER was reduced 33%.
In addition, HTER makes fine distinctions amongcorrect, near correct, and bad translations: correcttranslations have HTER = 0 and bad translations
have high HTER, around 1.
In our correlation experiments, we showed that
BLEU and TER are highly correlated, and that
HTER is more highly correlated to human judgments than BLEU or HBLEU. Although METEOR
using untargeted references is more highly correlated than TER using untargeted references, humantargeted HTER correlates with human judgments
better than METEOR, or its human-targeted variant (HMETEOR). Future studies might benefit fromalso examining correlations with the MMS evaluation metric(Turian et al., 2003).
The correlations shown were only on single sentences; correlations on document length segments

should also be explored. In addition, the HTER
numbers do vary, depending on the training, skill,and effort of the annotators. In a previous pilot
study, the reduction from TER to HTER was 50%
instead of 33%.
TER is easy to explain to people outside of the
MT community (i.e., the amount of work needed tocorrect the translations). Both TER and HTER appear to be good predictors of human judgments oftranslation quality. In addition, HTER may represent a method of capturing human judgments abouttranslation quality without the need for noisy subjective judgments. The automatic TER score with
4 references correlates as well with a single humanjudgment as another human judgment does, while
the scores with a human in the loop, such as HTER,
correlate significantly better with a human judgment
than a second human judgment does. This confirmsthat if humans are to be used to judge the qualityof MT output, this should be done by creating a newreference and counting errors, rather than by making
subjective judgments.

Acknowledgments
This work has been supported under the GALE program of the Defense Advanced Research Projects
Agency, Contract No. HR0011-06-C-0022. We
would like to thank Philip Resnik, Grazia RussoLassner, and Jimmy Lin for several fruitful discussions on the TER error metric and its variants, aswell as Nathanial Waisbrot for discussions on thealgorithmic aspects of TER. We are also thankful
to Alon Lavie and his students for freely distributing the METEOR software, and to Audrey Le forproviding access to the system output and humanjudgments upon which these experiments were conducted. We would also like to thank the reviewers
for their valuable feedback.

References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL 2005 Workshop on Intrinsic andExtrinsic Evaulation Measures for MT and/or Summarization.
George Doddington. 2002. Automatic Evaluationof Machine Translation Quality Using N-gram Co-

Occurence Statistics. In Human Language Technology: Notebook Proceedings, pages 128–132.
Robert Frederking and Sergei Nirenburg. 1994. Three
Heads are Better than One. In Proceedings of the
Fourth Conference on Applied Natural Language Processing, ANLP-94.
Margaret King. 1996. Evaluating Natural Language
Processing Systems. Communication of the ACM,
29(1):73–79, January.
Kevin Knight and Ishwar Chander. 1994. Automated
Postediting of Documents. In Proceedings of National
Conference on Artificial Intelligence (AAAI).
Inderjeet Mani, Gary Klein, David House, and Lynette
Hirschman. 2002. SUMMAC: A Text Summarization
Evaluation. Natural Language Engineering, 8(1):43–
68.
S. Niessen, F.J. Och, G. Leusch, and H. Ney. 2000. An
evaluation tool for machine translation: Fast evaluation for MT research. In Proceedings of the 2nd International Conference on Language Resources and
Evaluation (LREC-2000), pages 39–45.
Joseph Olive. 2005. Global Autonomous Language Exploitation (GALE). DARPA/IPTO Proposer Information Pamphlet.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Traslation. In Proceedings of the40th Annual Meeting of the Association for Computational Linguistics.
Grazia Russo-Lassner, Jimmy Lin, and Philip Resnik.
2005. A Paraphrase-Based Approach to MachineTranslation Evaluation. Technical Report LAMPTR-125/CS-TR-4754/UMIACS-TR-2005-57, University of Maryland, College Park.
Dana Shapira and James A. Storer. 2002. Edit Distancewith Move Operations. In Proceedings of the 13th Annual Symposium on Computational Pattern Matching,pages 85–98.
Somayajulu Sripada, Ehud Reiter, and Lezan Hawizy. 2004. Evaluating an NLG System Using PostEditing. Technical Report AUCS/TR0402, Department of Computing Science, University of Aberdeen.
Joseph P. Turian, Luke Shen, and I. Dan Melamed. 2003.
Evaluation of Machine Translation and its Evaluation.
In Proceedings of MT Summit IX.