An Awkward Disparity between BLEU / RIBES Scoresand Human Judgements in Machine Translation
1

1

Liling Tan, Jon Dehdari and Josef van Genabith
Universität des Saarlandes
1
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)
liling.tan@uni-saarland.de, {first.last_name}@dfki.de

Introduction

RIBES (Isozaki et al. 2014)•MT metrics criticized for various reasons• Kendall Tau prior on unigram• Adequacy not measured(Babych and Hartley, 2004; Smith et al. 2014; Graham et al. 2015)• Overcomes reordering• Correlates with BLEU (naturally)•Low BLEU != Bad MT (Callison-Burch et al. 2006)•Higher BLEU -> Better MT (c.f. WMT, WAT, IWSLT, OpenMT)BLEU: (Papineni et al. 2002)• Precision based• Weak recall penalty• Disregards orderMeta-Evaluation

Bubble Graph of Diff. Hypothesis - Baseline BLEU against RIBES for Positive HUMAN Scores

System Setup + Results• Crude ngram weights• Over-sensitive tominor difference

Bubble Graph of Diff. Hypothesis - Baseline BLEU against RIBES for Negative HUMAN Scores

Conclusion•Organizers:
RIBES = 94.13 ; BLEU = 69.22 ; HUMAN = 0.0•Ours :RIBES = 95.15 ; BLEU = 85.23 ; HUMAN = -17.75 !!!
Acknowledgements: The research leading to these results has received funding from the People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/2007-2013/ under REA grant agreement n ◦ 317471.

Note: This is our Unicode2String submission for KO->JA patent subtask in WAT 2015; the otherresults of other subtasks are presented in Tan and Bond (2014) and Tan et al. (2015)• Higher BLEU/RIBES correlates with +ve
HUMAN, not -ve HUMAN• Minor lexical diff. cause huge diff. in
BLEU ; RIBES mostly measures fluency• Minor metric score diff. not reflecting
major translation inadequacy• Higher BLEU != Better MTReferences• Bogdan Babych and Anthony Hartley. 2004. Ex- tending the BLEU MT evaluation method with frequency weightings. In ACL.
• OndËĞrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp,
Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In WMT.
• Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of Bleu in machine translation research. In EACL.
• Mauro Cettolo, Jan Niehues, Sebastian StÃĳker, Luisa Bentivogli, and Marcello Federico. 2014. Reporton the 11th IWSLT evaluation campaign, IWSLT 2014. In IWSLT.
• Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In ACL.
• Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic
evaluation of translation quality for distant language pairs. In EMNLP.
• Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro Sumita.
2015. Overview of the 2nd workshop on Asian translation. In WAT.
• Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic
evaluation of machine translation. In ACL.
• Liling Tan and Francis Bond. 2014. Manipulating in- put data in machine translation. In WAT.
• Liling Tan, Josef van Genabith, and Francis Bond. 2015. Passive and pervasive use of bilingual dictio- naryin statistical machine translation. In HyTra.