An Awkward Disparity
between BLEU / RIBES and
Human Judgment in MT
Liling Tan, Jon Dehdari and Josef van Genebith
Saarland University, Germany@alvations

Introduction• There’s always a bone to pick on MT
evaluation metrics (Babych and Hartley, 2004; CallisonBurch et al. 2006; Smith et al. 2014; Graham et al. 2015)
Hypothesis 1:
Appeared calm when he was taken to the American plane , which will to
Miami , Florida .

Almost SameBLEU?!

Hypothesis 2:
which will he was , when taken Appeared calm to the American plane to
Miami , Florida .

Reference:Orejuela appeared calm as he was led to the American plane which will takehim to Miami , Florida .
2Introduction• “Conventional” wisdom:– lower BLEU not necessarily worse translation(Callison-Burch et al. 2006)– higher BLEU = better translation(Callison-Burch et al. 2006; Nakazawa et al., 2014;
Cettolo et al., 2014; Bojar et al., 2015)3

Introduction

Callison-Burch et al. (2006) meta-evaluation on 2005 NIST MT Eval
4

Introduction• “Conventional” wisdom:– lower BLEU not necessarily worse translation(Callison-Burch et al. 2006)– higher BLEU = better translation(Callison-Burch et al. 2006; Nakazawa et al. 2014;
Cettolo et al. 2014; Bojar et al. 2015)

But is higher BLEU = better translation true?
5BLEU

Penalize if the length of thehypothesis is too longCount the proportion of n-grams thatappears in hypothesis and reference
6

BLEU (in practice)

Penalize if the length of thehypothesis is too longCount the proportion of n-grams thatappears in hypothesis and reference
7

BLEU
Hypothesis Baseline
P1 : 90.0
P1 : 84.2
P2 : 78.9
P2 : 66.7
P3 : 66.7
P3 : 47.1
P4 : 52.9
P4 : 25.0
BP: 0.905
BP: 0.854
BLEU: 64.03 BLEU: 43.29
HUMAN: -5 HUMAN: 08

RIBES
Hypothesis Baseline
RIBES: 94.04 RIBES: 86.33
BLEU: 53.3 BLEU: 58.8
HUMAN: -5 HUMAN: 09

System Level HUMAN

Hyp < Base=0<5= -1 HUMAN
10

System Level HUMAN

Hyp > Base=3>2= +1 HUMAN

11

System Level HUMAN

Hyp == Base
= +0 HUMAN

12

Segment Level HUMAN#Hyp - #Base= 3 -2= +1 HUMAN

13

Segment Level HUMAN#Hyp - #Base= 2 -2=014Segment Level HUMAN#Hyp - #Base=0-5= -5 HUMAN
15

Experiment Setup(Our WAT Submission)

16

Results(Our WAT Submission)+15 BLEU -> -17.75 HUMAN !!!
17

Results(Our WAT Submission)

higher BLEU = better translation is
not always true.
18

Segment level Meta-Evaluation(+ve HUMAN)

19

Segment level Meta-Evaluation(+ve HUMAN)

An interactive graph can be found here: https://plot.ly/171/~alvations/(Hint: click on the bubbles here on the interactive graph

20

Segment level Meta-Evaluation(+ve HUMAN)

Higher BLEU =Better translation(with 1-5 HUMAN)

21

Segment level Meta-Evaluation(+ve HUMAN)

Mostly, very goodtranslation (4-5 HUMAN)
don’t go beyond +30
BLEU from baseline

22Segment level Meta-Evaluation(+ve HUMAN)
Occasionally, lower BLEU is
better translation but still in
the range of 1-3 HUMAN score

23

Segment level Meta-Evaluation(+ve HUMAN)
There are some cases
where >+30 BLEU is as
good as the baseline

24Segment level Meta-Evaluation(+ve HUMAN)
Sometimes, there aretranslations with >+50 BLEU
with low HUMAN scores.

25

Segment level Meta-Evaluation(-ve HUMAN)

An interactive graph can be found here: https://plot.ly/173/~alvations/(Hint: click on the bubbles here on the interactive graph26

Segment level Meta-Evaluation(-ve HUMAN)

Generally, -BLEU or –RIBES
from baseline means
worse translations

An interactive graph can be found here: https://plot.ly/171/~alvations/(Hint: click on the bubbles here on the interactive graph27

Segment level Meta-Evaluation(-ve HUMAN)

Note that the grey bubbles are the same as theprevious graph
It’s more prominent here since there are many
more instances of +BLEU with 0 HUMAN scorethan negative HUMAN score

28

Segment level Meta-Evaluation(-ve HUMAN)
There are +0 BLEU but around+10 RIBES that achieved -5
HUMAN score

29

Segment level Meta-Evaluation(-ve HUMAN)

Then, there’s a whole lot of+BLEU that achieves – HUMAN
scores, i.e. worse than baseline30

Segment level Meta-Evaluation• With regards to positive HUMAN scores, it fits
the “conventional wisdom” that– lower BLEU/RIBES = worse translation– Higher BLEU/RIBES = better translation• When it comes to negative HUMAN scores, it
is inconsistent with the “conventionalwisdom”
31Conclusion• Higher BLEU and RIBES doesn’t necessary meanbetter translations– At segment level, >+30 BLEU might not be reliable• Possible reasons for BLEU/RIBES to not correlatewith human judgments includes:– Minor lexical differences -> huge difference in n-gram precision– Minor MT evaluation metric differences not reflecting majortranslation inadequacy
32

References•Bogdan Babych and Anthony Hartley. 2004. Ex- tending the BLEU MT evaluation method with frequency weightings. In ACL.

•Onderej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, VarvaraLogacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the2015 workshop on statistical machine translation. In WMT.

•Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of Bleu in machine translation research.
In EACL.

•Mauro Cettolo, Jan Niehues, Sebastian StÃijker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLTevaluation campaign, IWSLT 2014. In IWSLT.

•Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation
metrics. In ACL.

•Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation
quality for distant language pairs. In EMNLP.

•Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro Sumita. 2015. Overview of the2nd workshop on Asian translation. In WAT.

•Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machinetranslation. In ACL.

•

Liling Tan and Francis Bond. 2014. Manipulating in- put data in machine translation. In WAT.

•

Liling Tan, Josef van Genabith, and Francis Bond. 2015. Passive and pervasive use of bilingual dictionary in statistical machine
33
translation. In HyTra.

Experiment Setup(Our WAT Submission)

35

Results(Our WAT Submission)+15 BLEU -> -17.75 HUMAN !!!
36

Models’ Log-Linear Weights(Our Baseline Replica)# core weights[weight]
LexicalReordering0= 0.0316949 0.0566969 0.0546839 0.0814468
0.0359473 0.0426681
Distortion0= 0.0445616
LM0= 0.274422
WordPenalty0= -0.132106
PhrasePenalty0= 0.0733761
TranslationModel0= 0.110846 0.030776 -0.013284 0.0174904
UnknownWordPenalty0= 1.
37Models’ Log-Linear Weights(Our MERT Run 2)# core weights[weight]
LexicalReordering0= 0.0156288 -0.0580331 0.0126421 0.0664739
0.137966 0.0303402
Distortion0= 0.048086
LM0= 0.301798
WordPenalty0= -0.029068
PhrasePenalty0= 0.0512106
TranslationModel0= 0.173756 0.0386685 -0.0237588 0.0125696
UnknownWordPenalty0= 1

Despite the model differences, the resultsshows that higher BLEU = better translation
is not always true.

38