Scientific Credibility of Machine Translation Research:
A Meta-Evaluation of 769 Papers
Benjamin Marie
Atsushi Fujita
Raphael Rubino
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{bmarie,atsushi.fujita,raphael.rubino}@nict.go.jp

Abstract
This paper presents the first large-scale metaevaluation of machine translation (MT). We
annotated MT evaluations conducted in 769
research papers published from 2010 to 2020.
Our study shows that practices for automatic
MT evaluation have dramatically changed during the past decade and follow concerning
trends. An increasing number of MT evaluations exclusively rely on differences between
BLEU scores to draw conclusions, without
performing any kind of statistical significancetesting nor human evaluation, while at least
108 metrics claiming to be better than BLEU
have been proposed. MT evaluations in recent
papers tend to copy and compare automatic
metric scores from previous work to claim thesuperiority of a method or an algorithm without confirming neither exactly the same training, validating, and testing data have been used
nor the metric scores are comparable. Furthermore, tools for reporting standardized metricscores are still far from being widely adopted
by the MT community. After showing how theaccumulation of these pitfalls leads to dubious
evaluation, we propose a guideline to encourage better automatic MT evaluation along witha simple meta-evaluation scoring method to assess its credibility.

1

Introduction

New research publications in machine translation(MT) regularly introduce new methods and algorithms to improve the translation quality of MT systems. In the literature, translation quality is usually
evaluated with automatic metrics such as BLEU
(Papineni et al., 2002) and, more rarely, by humans.
To assess whether an MT system performs better
than another MT system, their scores given by anautomatic metric are directly compared. While
such comparisons between MT systems are exhibited in the large majority of MT papers, there are

no well-defined guideline nor clear prerequisites
under which a comparison between MT systems
is considered valid. Consequently, we assume that
evaluation in MT is conducted with different degrees of thoroughness across papers and that evaluation practices have evolved over the years. What
could be considered, by the research community,
as a good evaluation methodology ten years ago
may not be considered good today, and vice versa.
This evolution has not been studied and whether
MT evaluation has become better, or worse, is debatable.
On the other hand, several requirements for MT
evaluation have been well-identified. For instance,
the limitations of BLEU are well-known (CallisonBurch et al., 2006; Reiter, 2018; Mathur et al.,2020) and the necessity to report automatic metricscores through standardized tools, such as SacreBLEU, has been recognized (Post, 2018). Moreover, a trustworthy evaluation may adopt statisticalsignificance testing (Koehn, 2004) and strong baselines (Denkowski and Neubig, 2017). However, towhat extent these requirements have been met in
MT publications is unclear.
In this paper, we propose the first large-scale
meta-evaluation of MT in which we manually annotated 769 research papers published from 2010
to 2020. Our study shows that evaluation in MT has
dramatically changed since 2010. An increasing
number of publications exclusively rely on BLEU
scores to draw their conclusions. The large majority of publications do not perform statistical significance testing, especially since 2016. Moreover,an increasing number of papers copy and compare
BLEU scores published by previous work whiletools to report standardized metric scores are still
far from being extensively adopted by the MT community. We also show that compared systems areoften trained, validated, or even evaluated, on datathat are not exactly the same. After demonstratinghow the accumulation of these pitfalls leads to dubious evaluation, we propose a general guideline for
automatic evaluation in MT and a simple scoring
method to meta-evaluate an MT paper. We believethat the adoption of these tools by authors or reviewers have the potential to reverse the concerning
trends observed in this meta-evaluation.

A4. Whether it makes comparisons with automatic
metric scores directly copied from previous
work to support its conclusion: yes or no.
Most papers copying scores (mostly BLEU)
clearly mention it. If there is no evidence thatthe scores have been copied, we annotatedthese papers with “no” for this question.

2

A5. Whether SacreBLEU has been used: yes or
no. If there is no mention or reference to“SacreBLEU,” we assume that it has not beenused. Note that “yes” does not mean that the
paper used SacreBLEU for all the MT systems
evaluated.

A Survey on MT Evaluation

We manually annotated the MT evaluation in research papers published from 2010 to 2020 at*ACL conferences.1 To identify MT papers, wesearched the ACL Anthology website2 for the terms“MT” or “translation” in their titles3 and analyzed
among them the 769 papers that make comparisonsof translation quality between at least two MT systems. For each year between 2010 and 2020, we
respectively annotated the following numbers of
papers: 53, 40, 59, 80, 67, 45, 51, 62, 94, 115, and103.
We annotated each paper as follows:
A1. All the automatic metrics used to evaluate thetranslation quality of MT systems. We did notlist variants of the same metric: e.g., chrF3
and chrF++ are labeled chrF (Popović, 2015).
Moreover, we did not consider metrics which
only target specific aspects of the translation
quality, such as pronoun translation and rareword translation.
A2. Whether a human evaluation of the translation
quality has been conducted: yes or no. If thehuman evaluation only targets specific typesof errors and did not evaluate the translation
quality of the entire text, we answered “no.”4
A3. Whether any kind of statistical significancetesting of the difference between automatic
metric scores has been performed: yes or no.
Potentially, some papers did perform significance testing without mentioning it, but due
to the lack of evidences such papers have beenannotated with “no” for this question.
1
We considered only *ACL main conferences, namely
ACL, NAACL, EACL, EMNLP, CoNLL, and AACL, as they
are the primary venues for publishing MT papers.
2
www.aclweb.org/anthology/
3
There are potentially MT papers falling outside thesesearch criteria but we considered the 769 papers we obtained
to be representative enough for the purpose of this study.
4Note that we only check here whether the automatic evaluation is supported by a human evaluation. Previous work
already studied pitfalls in human evaluation (Läubli et al.,2020).

A6. If previous work has not been reproduced but
copied, whether it has been confirmed that all
the compared MT systems used exactly thesame pre-processed training, validating, andtesting data: yes or no.
Except for A6, the annotation was straightforward
since most papers present a dedicated section forexperimental settings with most of the information we searched for. Answering A6 required tocheck the data exploited in the previous work used
for comparison. Note that answering “yes” to thequestions from A2 to A6 may only be true for atleast one of the comparisons between MT systems,
while we did not evaluate how well it applies. For
instance, answering “yes” to A5 only means thatat least one of the systems has been evaluated with
SacreBLEU but not that the SacreBLEU signature
has been reported nor that SacreBLEU scores havebeen correctly compared with other BLEU scoresalso computed with SacreBLEU.
Our annotations are available as a supplemental
material of this paper. To keep track of the evolution of MT evaluation, we will periodically update
the annotations and will make it available online.5

3

Pitfalls and Concerning Trends

This section discusses the four main pitfalls identified in our meta-evaluation of MT: the exclusive useof BLEU, the absence of statistical significance testing, the comparison of incomparable results fromprevious work, and the reliance on comparison between MT systems that do not exploit exactly thesame data. We report on how often they affected
MT papers and recent trends. Based on previous
5
The up-to-date version can be found here: github.
com/benjamin-marie/meta_evaluation_mt.

100

BLEU
TER
METEOR
RIBES
NIST
chrF
Other
Human

90
80
% publications

70
60
50
40
30
20
10
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

Figure 1: Percentage of papers using each evaluation metric per year. Metrics displayed are used in more than
five papers. “Other” denotes all other automatic metrics. “Human” denotes that a human evaluation has been
conducted.

work and supporting experiments, we show howeach of these problems and their accumulation leadto scientifically dubious MT evaluation.
3.1

The 99% BLEU

Automatic metrics for evaluating translation quality have numerous advantages over a human evaluation. They are very fast and virtually free torun provided that a reference translation is alreadyavailable. Their scores are also reproducible. As
such, automatic metrics remained at the center ofMT evaluation for the past two decades. New metrics that better correlate with human judgments are
regularly introduced. We propose in this section
to analyze the use of automatic metrics in MT research, relying on our annotations for A1 and A2.
This is probably the most expected finding in ourstudy: the overwhelming majority of MT publications uses BLEU. Precisely, 98.8% of the annotatedpapers report on BLEU scores. As shown in Figure 1, the ratio of papers using BLEU remained
stable over the years. On the other hand, BLEU
scores used to be more often supported by scoresfrom other metrics, such as TER (Snover et al.,2006) and METEOR (Banerjee and Lavie, 2005),
than they are now. The large majority of papers,
74.3%, only used BLEU scores to evaluate MT systems, i.e., without the support of any other metrics
nor human evaluation. It increases to 82.1% if we
consider only the years 2019 and 2020.
This tendency looks surprising considering thatno less than 108 new metrics6 have been proposedin the last decade. They have been shown to better correlate with human judgments than BLEU.
Some are even easier to use and more reproducible
6
We did not count variants of the same metric and excluded
metrics only proposed for an evaluation at segment level.

by being tokenization agnostic, such as chrF. We
counted 29 metrics proposed at *ACL conferences
since 2010 while the remaining metrics were proposed at the WMT Metrics Shared Tasks. 89%of these 108 new metrics have never been used inan *ACL publication on MT (except in the papers
proposing the metrics). Among these metrics, only
RIBES (Isozaki et al., 2010) and chrF have beenused in more than two MT research paper.
When properly used, BLEU is a valid metric for evaluating translation quality of MT systems (Callison-Burch et al., 2006; Reiter, 2018).
Nonetheless, we argue that better metrics proposedby the research community should be used to improve MT evaluation. To illustrate how wrong anevaluation can become by only relying on one metric, we computed with BLEU and chrF scores7 of
WMT20 submissions to the news translation sharedtask8 (Barrault et al., 2020) using SacreBLEU andshow rankings given by both metrics in Table 1.
Results show that BLEU and chrF produce two different rankings. For instance, for the Ja→En task,
NiuTrans system is the best according to BLEU by
being 1.1 points better than the Tohoku-AIP-NTT
system ranked second. In most MT papers, sucha difference in BLEU points would be considered
as a significant evidence of the superiority of an
MT system and as an improvement in translation
quality. Relying only on these BLEU scores without any statistical significance testing nor humanevaluation would thus lead to the conclusion that
NiuTrans system is the best. However, according
to another metric that better correlates with human7

SacreBLEU (short) signatures: chrF2+l.{ja-en,zhen}+n.6+s.false+t.wmt20+v.1.5.0 and BLEU+c.mixed+l.{jaen,zh-en}+#.1+s.exp+t.wmt20+tok.13a+v.1.5.0
8data.statmt.org/wmt20/
translation-task/

Rank
1
2
3
4
5

BLEU
26.6♠
25.5
24.8
22.8
22.2

Japanse-to-English (Ja→En)
SystemchrF
NiuTrans
Tohoku-AIP-NTT
OPPO
NICT Kyoto
eTranslation

0.536
0.535
0.523
0.507
0.504

SystemBLEU

Tohoku-AIP-NTT
NiuTrans
OPPO
Online-A
Online-B

36.9
36.8
36.6
36.6
35.9

Chinese-to-English (Zh→En)
SystemchrF
WeChat AI
Tencent Translation
DiDi NLP
Volctrans
THUNLP

0.653
0.648
0.645
0.644
0.643

SystemVolctrans
Tencent Translation
DiDi NLP
DeepMind
THUNLP

Table 1: Rankings of WMT20 top 5 submissions for the News Translation Shared Tasks according to BLEU and
chrF scores. Superscripts indicate systems that are significantly worse ( ) and better (♠ ) according to each metric(p-value < 0.05) than Tohoku-AIP-NTT and Volctrans systems for Ja→En and Zh→En, respectively.

3.2

The Disappearing Statistical Significance
Testing

Statistical significance testing is a standard methodology designed to ensure that experimental results
are not coincidental. In MT, statistical significancetesting has been used on automatic metric scoresand more particularly to assess whether a particular difference of metric scores between two MT
9For both Ja→En and Zh→En tasks, systems ranked firstby chrF were also ranked first by the human evaluation.
10
Examples of such requests or related comments byreviewers can be found in the ACL 2017 review corpus(github.com/allenai/PeerRead), e.g., in the review
ID 369 we read: “I am also rather suspicious of the fact thatthe authors present only METEOR results and no BLEU.”

70
60% publications

judgment, i.e., chrF, this does not hold: TohokuAIP-NTT system is better. Similar observations
are made for the Zh→En task.9 These observations have often been made by the MT community,
for instance at WMT shared tasks, but nonethelessrarely seen in research papers.
We assume that MT researchers largely ignore
new metrics in their research papers for the sakeof some comparability with previous work or simply because differences between BLEU scores may
seem more meaningful or easier to interpret thandifferences between scores of a rarely used metric. Most papers even qualify differences between
BLEU scores as “small,” “large,” or “significant”(not necessarily statistically), implying that there
is a scientific consensus on the meaning of differences between BLEU scores. As we show in
the following sections, all these considerations areillusory. Moreover, BLEU may also be directly requested by reviewers, or even worse, other metrics
may be requested to be dropped.10 We believe thatthe exclusive reliance on BLEU can be ended andthe use of better metrics should be encouraged, in
addition to or in lieu of BLEU, by the adoptionof a guideline for automatic MT evaluation (seeSection 4).

50
40
30
20
10
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Figure 2: Percentage of papers testing statistical significance of differences between metric scores.

systems is not coincidental. Two methods are prevalent in MT: the paired bootstrap test (Koehn, 2004)
and the approximate randomization test (Riezler
and Maxwell, 2005), for instance respectively implemented in Moses11 and MultEval.12
Dror et al. (2018) report that while the natural
language processing (NLP) community assigns agreat value to experimental results, statistical significance testing is rarely used. We verified if thisapplies to MT evaluations based on our annotations
for A3. Figure 2 shows the percentage of papers
that performed statistical significance testing. We
found out that the observations by Dror et al. (2018)
apply to MT since never more than 65.0% of the
publications in a year (2011) performed statistical significance testing. Furthermore, our metaevaluation shows a sharp decrease of its use since2016. Most papers did not check whether theirresults are not coincidental but drew conclusions
from them.
MT papers mainly relied on the amplitude of thedifferences between metric scores to state whether
they are significant or not. This was also observedby Dror et al. (2018) for NLP in general.
For illustration, we also performed statisticalsignificance testing13 with BLEU and chrF scores
11

github.com/moses-smt/mosesdecoder
github.com/jhclark/multeval
13
For all the statistical significance testing performed in thispaper, we used the paired bootstrap test with 1,000 samples
and 1,000 iterations.
12

Ja→EnBLEU

chrF

System

Zh→En
BLEU

chrF

Tohoku-AIP-NTT

25.5

0.536

Volctrans

36.6

0.653

Custom 1
Custom 2

25.5
18.7

0.536
0.503

Custom 1
Custom 2

36.6
32.2

0.653
0.638

Table 2: BLEU and chrF scores of the customized
Tohoku-AIP-NTT and Volctrans outputs from which
only one sentence has been modified. The first row
shows the results of the original WMT20 submissions.
Custom 1 replaced the last sentence with an empty line,
while Custom 2 replaced the last sentence with a sequence repeating 10k times the same token. None ofthese systems are significantly different according tostatistical significance testing on these scores.

on the WMT20 submissions in Table 1. For Ja→En,
NiuTrans system is significantly better in BLEU
than Tohoku-AIP-NTT system. In contrast, they
are not significantly different in chrF. Using only
BLEU, we would conclude that NiuTrans system
is significantly the best. This is not confirmed by
chrF hence we need to report on more than one
metric score to conduct a credible evaluation, evenwhen performing statistical significance testing.
Furthermore, to show that the significance of adifference between metric scores is independent
from its amplitude, we performed additional experiments by modifying only one sentence, replacing
it with an empty line or by the repetition of thesame token many times,14 from Tohoku-AIP-NTT
and Volctrans systems’ outputs. Results in BLEU
and chrF are reported in Table 2. We observe thata difference in only one sentence can lead to a difference in BLEU of 6.8 points (Ja→En, Custom2).15 Nonetheless, our statistical significance tests
did not find any system significantly better than theothers.
While the importance of statistical significancetesting is regularly debated by the scientific community (Wasserstein et al., 2019), it remains oneof the most cost-effective tools to check how trustworthy a particular difference between two metricscores is.16
14
This could be considered as a simulation of potential defects from an MT framework or model, e.g., when translating
extremely long sequences.
15
For “Custom 2,” BLEU greatly penalized the increase of
the number of tokens in the output. This is indicated by thelength ratio reported by SacreBLEU but rarely shown in MT
papers.
16
Wasserstein et al. (2019) give several recommendations
for a better use of statistical significance testing.

% publications

System40
30

Copied scores
SacreBLEU
Copied w/o SacreBLEU

20
1002010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Figure 3: Percentage of papers copying scores fromprevious work (“Copied scores”), using SacreBLEU(“SacreBLEU”), and copying scores without using
SacreBLEU (“Copied w/o SacreBLEU”).

3.3

The Copied Results

An MT paper may compare the automatic metricscores of proposed MT systems with the scores
reported in previous work. This practice has theadvantage to save the time and cost of reproducing
competing methods. Based on our annotations for
A4, we counted how often papers copied the scoresfrom previous work to compare them with theirown scores. As pointed out by Figure 3, copying scores (mostly BLEU) from previous work
was rarely done before 2015. In 2019 and 2020,
nearly 40% of the papers reported on comparisonswith scores from other papers. While many papers copied and compared metric scores acrosspapers, it is often unclear whether they are actually comparable. As demonstrated by Post (2018),
BLEU, as for most metrics, is not a single metric.
It requires several parameters and is dependent on
the pre-processing of the MT output and referencetranslation used for scoring. In fact, Post (2018)
pointed out that most papers do not provide enoughinformation to enable the comparability of theirscores with other work. Post (2018) proposed atool, SacreBLEU, to standardize metrics17 in order to guarantee this comparability, provided that
all the scores compared are computed with SacreBLEU.18 This is the only tool of this kind used by
the papers we annotated. However, based on our annotations for A5, Figure 3 shows that SacreBLEU
is still far from widely adopted by the MT community, even though it is gradually getting more
popular since its emergence in 2018. Moreover,papers that copy BLEU scores do not always use
SacreBLEU, even in 2020.
To illustrate how deceiving a comparison of
copied scores can be, we report on BLEU and chrF
scores using different processing,19 commonly
17

Currently BLEU, chrF, and TER.
SacreBLEU also generates a “signature” to further ensure
this comparability: two scores computed through SacreBLEU
with an identical signature are comparable.
19
For all our processing, we used Moses (code version mmt18

Processing

Tohoku-AIP-NTT (Ja→En)
BLEU
chrF

Volctrans (Zh→En)
BLEU
chrF

original
fully lowercased
norm. punct.

25.5
26.9
25.5

0.536
0.549
0.537

36.6
38.2
37.8

0.653
0.664
0.657

tokenized+ norm. punct.
+ aggressive

26.7
26.8
27.8

0.541
0.541
0.541

37.1
38.5
39.5

0.653
0.659
0.659

Table 3: BLEU and chrF scores computed by SacreBLEU after applying different processing on some
WMT20 MT system outputs (from Tohoku-AIP-NTT
and Volctrans) and on the reference translations. Noneof these rows are comparable.

adopted by MT researchers, applied to some MT
system outputs and reference translations of the
WMT20 news translation shared tasks. Our results
are presented in Table 3. The first row presents original SacreBLEU scores, i.e., detokenized. Second
and third rows respectively show the impact of lowercasing and punctuation normalization on metricscores. Scores are increased. Last three rows show
the results on tokenized MT outputs. Applying
both punctuation normalization and aggressive tokenization with Moses scripts leads to BLEU scores
several points higher than the original SacreBLEU
scores. Obviously, none of the scores in different
rows are comparable. Nonetheless, MT papers still
often report on tokenized BLEU scores comparedwith tokenized, or even detokenized, BLEU scoresfrom other papers without exactly knowing how tokenization has been performed. Tokenized BLEU
scores reported in MT papers are often computed
using the multi-bleu script of Moses even though
it displays the following warning:20 “The scores
depend on your tokenizer, which is unlikely to be
reproducible from your paper or consistent acrossresearch groups.”
Even though the work of Post (2018) is a wellacclaimed initiative towards better MT evaluation,
we believe that it can only be a patch for questionable evaluation practices. A comparison witha copied score is de facto associated with the absence of statistical significance testing since theMT output used to compute the copied score is
not available. We also observed several misusesof SacreBLEU, such as the comparison of scoresobtained by SacreBLEU against scores obtained by
mvp-v0.12.1-2851-gc054501) scripts.
20This warning has been added on 20 Oct. 2017. Insightful
discussions on this commit can be found there:
github.com/moses-smt/
mosesdecoder/commit/
545eee7e75487aeaf45a8b077c57e189e50b2c2e.

other tools. SacreBLEU signatures are also often
not reported despite being required to ensure thecomparability between SacreBLEU scores.
Ultimately, comparisons with copied scores must
be avoided. As we will show in the next section,
copying scores also calls for more pitfalls.
3.4

The Data Approximation

In MT, datasets are mostly monolingual or paralleltexts used in three different steps of an experiment:
training a translation model, tuning/validating themodel, and evaluating it. Henceforth, we denotethese datasets as training, validating, and testing
data, respective to these three steps. How thesedatasets are pre-processed strongly influences translation quality. MT papers regularly propose new
methods or algorithms that aim at better exploiting training and/or validating data. Following thescientific method, we can then define these new
methods/algorithms and datasets as independent
variables of an MT experiment while the translation quality, approximated by metric scores, would
be our dependent variable that we want to measure. Testing the impact of a new algorithm on
our dependent variable requires to keep all otherindependent variables, such as datasets, unchanged.
In other words, changing datasets (even slightly)
and methods/algorithms in the same experimentcannot answer whether the change in metric scoresis due to the datasets, methods/algorithms, or the
combination of both.
Relying on our annotation for A6, we examined
how often MT papers compared MT systems forwhich the datasets and/or their pre-processing21 described in the papers are not exactly identical. Notethat we only performed this comparison for papers that copied and compared metric scores fromprevious work. Here, we also excluded comparisons between systems performed to specificallyevaluate the impact of new datasets, pre-processing
methods, and human intervention or feedback (e.g.,
post-editing and interactive MT). If we had any
doubt whether a paper belongs or not to this category, we excluded it. Consequently, our estimation
can be considered as the lower bound.
To illustrate the impact of modifications of thesedatasets on metric scores, we conducted experiments using the training, validating, and testingdata of the WMT20 news translation tasks. We
21

For pre-processing, we checked, for instance, tokenization (framework and parameters), casing, subword segmentations (method and vocabulary size), data filtering, etc.

--type transformer --mini-batch-fit --valid-freq
5000 --save-freq 5000 --workspace 10000--disp-freq 500 --beam-size 12 --normalize
1 --valid-mini-batch 16 --overwrite--early-stopping 5 --cost-type ce-mean-words--valid-metrics bleu --keep-best --enc-depth
6 --dec-depth 6 --transformer-dropout
0.1 --learn-rate 0.0003 --lr-warmup 16000--lr-decay-inv-sqrt 16000 --lr-report--label-smoothing 0.1 --devices 0 1 2 3 4 5 6
7 --optimizer-params 0.9 0.98 1e-09 --clip-norm
5 --sync-sgd --exponential-smoothing --seed 1234

% publications

Table 4: Hyper-parameters of Marian used for training
our NMT systems.
40
30
20
10
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Figure 4: Percentage of papers that compared MT systems using data that are not identical.

trained neural MT (NMT) systems with Marian22(Junczys-Dowmunt et al., 2018), using the hyperparameters in Table 4, on all the provided parallel
data (“all” configurations) and removed sentencepairs based on their length (“Max Len.”). This
simple filtering step is usually applied for a more
efficient training or due to some limits of the framework, method, or algorithm used. Yet, it is socommon as a pre-processing step that it is rarely
described in papers. As shown in Table 5, we observed that BLEU scores vary by several points
depending on the maximum length used for filtering. Another common pre-processing step isthe truecasing of the datasets. While it is rather
commonly performed by participants in the WMTtranslation shared tasks, how casing is handled israrely mentioned in research papers. In our experiments, applying this step changed BLEU scores bymore than 0.5 points. Further experiments applying
language identification filtering or removing one
corpus from the training data also lead to variations
in metric scores. The best configurations according
to metric scores do not use truecasing and has amaximum sentence length set at 120 (second row).
A comparison of this configuration with the thirdrow, which uses truecasing and a different maximum sentence length, cannot lead to the conclusionthat truecasing decreases translation quality, since
we changed two variables at the same time.
While these observations may be expected oreven obvious, Figure 4 shows that we found in
22

Version: v1.7.6 1d4ba73 2019-05-11 17:16:31 +0100

Data

Max
Len.

tc

En→De
BLEU
chrF

Ja→En
BLEU
chrF

all

120

X

30.9

0.599

20.4

0.478

♠

♠

♠

all
all
all
all

120
100
80
60

X
X
X

31.5
30.8
29.8
26.6

0.604
0.597
0.584
0.549

21.1
20.5
20.0
18.5

0.481
0.476
0.471
0.453

lid filtered

120

X

30.3

0.596

20.7

0.480

0.600



0.468

- 1 corpus

120

X

30.7

19.6

Table 5: BLEU and chrF scores of systems using differently pre-processed parallel data for training. “MaxLen.” denotes that sentence pairs with sentence longer
than the specified number, in terms of subword tokens,are removed. “tc” denotes whether truecasing is doneor not. If not, original case of the data is kept for all
datasets. “lid filtered” denotes that the training data arefiltered with language identification tools. Last row denotes that we remove one corpus from the training data:“Rapid” for En→De and “OpenSubtitles” for Ja→En.
and ♠ respectively denote systems that are significantly worse and better (p-value < 0.05), according to
the metric, than the system in the first row.

our meta-evaluation an increasing amount of MT
papers (38.5% for the 2019–2020 period) drawing conclusions of the superiority of a particularmethod or algorithm while also using different data.
While their conclusions may be valid, the evaluation conducted in these papers is scientificallyflawed and cannot support the conclusions. We
assume that this is mainly due to a rather common lack of detailed experimental settings. Consequently, it makes a specific experiment often
impossible to be reproduced identically. In mostcases, ensuring the comparability with the published scores of an MT system is only possible byreplicating the MT system by ourselves. There
have been initiatives towards the release of preprocessed datasets for MT, for instance by the
WMT conference that released pre-processed data
for WMT19.23 Nonetheless, we only noticed a verysmall number of papers exploiting pre-processedtraining/validating/testing data publicly released byprevious work.24 We believe that the current trend
should be reversed. Reviewers should also request
more rigor to the authors by checking the configurations of the compared MT systems to make surethat their comparison can, indeed, answer whether
the proposed method/algorithm improves MT inde23

This effort has not been conducted for WMT20.
For instance, Ma et al. (2020) and Kang et al. (2020) usedexactly the same pre-processed data for research on documentlevel NMT released by Maruf et al. (2019).
24

100
90
80
% publications

70

BLEU only
w/o sigtest
w copied scores
w/o SacreBLEU
w/ diff. data

a better evaluation but not a flawless evaluation.
4.2

The Guideline

This guideline and the scoring method that follows
are proposed for MT papers that rely on automatic
metric scores for evaluating translation quality.

60
50
40
30
20
10
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Figure 5: Percentage of papers affected by the accumulation of pitfalls. Each bar considers only the papers
counted by the previous bar, e.g., the last bar considers
only papers that compared MT systems exploiting different datasets while exclusively using BLEU (“BLEU
only”), without performing statistical significance testing (“w/o sigtest”), to measure differences with BLEU
scores copied from other papers (“w/ copied scores”)
while not using SacreBLEU (“w/o SacreBLEU”).

pendently of the data and their pre-processing.

4

A Guideline for MT Meta-Evaluation

4.1

Motivation

The MT community is well-aware of all the pitfalls
described in Section 3. They have all been described by previous work. Nonetheless, our metaevaluation shows that most MT publications areaffected by at least one of these pitfalls. More puzzling are the trends we observed. Figure 5 showsthat an increasing number of publications accumulate questionable evaluation practices. In the
period 2019–2020, 17.4% (38 papers) of the annotated papers exclusively relied for their evaluation
on differences between BLEU scores of MT systems, of which at least some have been copiedfrom different papers, without using SacreBLEU
nor statistical significance testing, while exploiting
different datasets.
While these pitfalls are known and relatively
easy to avoid, they are increasingly ignored andaccumulated. We believe that a clear, simple, andwell-promoted guideline must be defined for automatic MT evaluation. Such a guideline would
be useful only if it is adopted by authors and its
application is checked by reviewers. For the latter,
we also propose a simple scoring method for themeta-evaluation of MT.
Note that the proposed guideline and scoring
method only cover the aspects discussed in this paper. Thus, their strict adherence can only guarantee

1. An MT evaluation may not exclusively relyon BLEU. Other automatic metrics that better
correlate with human judgments, or a humanevaluation, may be used in addition or in lieu
of BLEU.
2. Statistical significance testing may be performed on automatic metric scores to ensurethat the difference between two scores, whatever its amplitude, is not coincidental.
3. Automatic metric scores copied from previous work may not be compared. If inevitable,
copied scores may only be compared withscores computed in exactly the same way,
through tools guaranteeing this comparability,
while providing all the necessary information
to reproduce them.
4. Comparisons between MT systems throughtheir metric scores may be performed todemonstrate the superiority of a method oran algorithm only if the systems have been
trained, validated, and tested with exactly thesame pre-processed data, unless the proposed
method or algorithm is indeed dependent on aparticular dataset or pre-processing.
The purpose of the following scoring method
is to assess the trustworthiness of an automatic
evaluation performed in an MT paper. Ultimately,
it can be used for authors’ self-assessment or by
MT program committees to identify trustworthy
papers.
Each “yes” answer to the following questions
brings 1 point to the paper for a maximum of 4
points.
1. Is a metric that better correlates with humanjudgment than BLEU used or is a human evaluation performed?
2. Is statistical significance testing performed?
3. Are the automatic metric scores computed forthe paper and not copied from other work?
If copied, are all the copied and comparedscores computed through tools that guarantee
their comparability (e.g., SacreBLEU)?

0 point
1 point
2 points
3 points
4 points

90% publications

80

2.5
2.4
2.3

70

2.2

60

2.1

50

2.0

40

1.9

30

1.8

20

1.7

10

1.6

0

Points100

1.52010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

researchers to perform a similar meta-evaluation in
their respective area of expertise. As we showed, it
can unveil pitfalls and concerning trends that canbe reversed before becoming prevalent.

Acknowledgments
We would like to thank the reviewers for their insightful comments and suggestions. This work waspartly supported by JSPS KAKENHI grant numbers 20K19879 and 19H05660.

Figure 6: Average meta-evaluation score per year.

4. If comparisons between MT systems are performed to demonstrate the superiority of amethod or an algorithm that is independent
from the datasets exploited and their preprocessing, are all the compared MT systems
exploiting exactly the same pre-processeddata for training, validating, and testing? (ifnot applicable, give 1 point by default)
We scored all the annotated papers, and reporton the average score and score distribution for eachyear in Figure 6. Based on this meta-evaluation,
MT evaluation worsens.

5

Conclusion

Our meta-evaluation identified pitfalls in the MT
evaluation in most of the annotated papers. The
accumulation of these pitfalls and the concerning
trends we observed lead us to propose a guideline
for automatic MT evaluation. We hope this guideline, or a similar one, will be adopted by the MT
community to enhance the scientific credibility of
MT research.
This work also has its limitations since it does
not cover all the pitfalls of MT evaluation. For
instance, we noticed that MT papers regularly relyon the same language pairs to claim general improvements of MT. They also almost exclusively
focus on translation from or into English. Another,more positive observation, is that MT papers tend
to use stronger baseline systems, following someof the recommendations by Denkowski and Neubig (2017), than at the beginning of the last decade
when baseline systems were mostly vanilla MTsystems. For future work, we plan to extend ourmeta-evaluation of MT to publications at conferences in other research domains, such as Machine
Learning and Artificial Intelligence.
As a final note, we would like to encourage NLP

References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, USA. Association for Computational Linguistics.
Loı̈c Barrault, Magdalena Biesialska, Ondřej Bojar,
Marta R. Costa-jussà, Christian Federmann, Yvette
Graham, Roman Grundkiewicz, Barry Haddow,
Matthias Huck, Eric Joanis, Tom Kocmi, Philipp
Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof
Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 Conference on Machine Translation (WMT20). In Proceedings of the Fifth Conference on Machine Translation, pages 1–55, Online. Association for Computational Linguistics.
Chris Callison-Burch, Miles Osborne, and Philipp
Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In Proceedings of the
11th Conference of the European Chapter of the
Association for Computational Linguistics, Trento,
Italy. Association for Computational Linguistics.
Michael Denkowski and Graham Neubig. 2017.
Stronger baselines for trustable results in neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 18–27,
Vancouver, Canada. Association for Computational
Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito
Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language
pairs. In Proceedings of the 2010 Conference on

Empirical Methods in Natural Language Processing,pages 944–952, Cambridge, USA. Association for
Computational Linguistics.
Marcin Junczys-Dowmunt, Roman Grundkiewicz,
Tomasz Dwojak, Hieu Hoang, Kenneth Heafield,
Tom Neckermann, Frank Seide, Ulrich Germann,
Alham Fikri Aji, Nikolay Bogoychev, André F. T.
Martins, and Alexandra Birch. 2018. Marian: Fast
neural machine translation in C++. In Proceedingsof ACL 2018, System Demonstrations, pages 116–
121, Melbourne, Australia. Association for Computational Linguistics.
Xiaomian Kang, Yang Zhao, Jiajun Zhang, andChengqing Zong. 2020. Dynamic context selection
for document-level neural machine translation via reinforcement learning. In Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2242–2254, Online. Association for Computational Linguistics.
Philipp Koehn. 2004. Statistical significance tests
for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–
395, Barcelona, Spain. Association for Computational Linguistics.
Samuel Läubli, Sheila Castilho, Graham Neubig, Rico
Sennrich, Qinlan Shen, and Antonio Toral. 2020.
A set of recommendations for assessing human–machine parity in language translation. Journal of
Artificial Intelligence Research, 67:653–672.
Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020.
A simple and effective unified encoder for documentlevel machine translation. In Proceedings of the58th Annual Meeting of the Association for Computational Linguistics, pages 3505–3511, Online. Association for Computational Linguistics.
Sameen Maruf, André F. T. Martins, and Gholamreza
Haffari. 2019. Selective attention for context-awareneural machine translation. In Proceedings of the2019 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long andShort Papers), pages 3092–3102, Minneapolis, USA.
Association for Computational Linguistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn.
2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation
metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
pages 4984–4997, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia,
USA. Association for Computational Linguistics.

Maja Popović. 2015. chrF: character n-gram F-score
for automatic MT evaluation. In Proceedings of theTenth Workshop on Statistical Machine Translation,
pages 392–395, Lisbon, Portugal. Association for
Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU
scores. In Proceedings of the Third Conference on
Machine Translation: Research Papers, pages 186–
191, Brussels, Belgium. Association for Computational Linguistics.
Ehud Reiter. 2018. A structured review of the validity
of BLEU. Computational Linguistics, 44(3):393–
401.
Stefan Riezler and John T. Maxwell. 2005. On some
pitfalls in automatic evaluation and significance testing for MT. In Proceedings of the ACL Workshop
on Intrinsic and Extrinsic Evaluation Measures for
Machine Translation and/or Summarization, pages
57–64, Ann Arbor, Michigan. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study oftranslation edit rate with targeted human annotation.
In Proceedings of the 7th Biennial Conference of the
Association for Machine Translation in the Americas, pages 223–231, Cambridge, USA. Association
for Machine Translation of the Americas.
Ronald L. Wasserstein, Allen L. Schirm, and Nicole A.
Lazar. 2019. Moving to a world beyond “p < 0.05”.
The American Statistician, 73(sup1):1–19.