Is This Translation Error Critical?
Classification-Based Human andAutomatic MT Evaluation Focusing on
Critical Errors
Katsuhito Sudoh*, Kosuke Takahashi, Satoshi Nakamura
Nara Institute of Science and Technology (NAIST)*PRESTO, Japan Science and Technology Agency

This work is supported by JST PRESTO (Grant number JPMJPR1856).

2

Quick Overview
端⼦は互いに接触しないように配置されている

Terminals are placed not to bein contact with each other.
Critical error in meaning

Contradiction

Machinetranslation

Terminals each other do notcontact to be placed.
Critical error in comprehension

Incomprehensible

Revisitng classification-based MT evaluationin two dimensions: Adequacy & Fluency

Background• Regression-based MT evaluation• Founded on Human Direct Assessment(Graham+ 2016)• Predict human DA scores usingreference and hypothesis translations• BERT regressor (Shimanaka+ 2019),
BERTScore (Zhang+ 2020), BLEURT(Sellam+ 2020), …

• Can they identify critial errors?

34(Artificial) Examples
ExamplesThe Pleiades is situated 445 light-years
from Earth. [same as ref.]
The Pleiades is not situated 445 lightyears from Earth.
The Pleiades is situated 445 light-years
from Mars.
Is Earth from Pleiades the light-years
situated cluster 445.
Turn off the light for saving the Earth.

BLEU

BERT
Score BLEURT

1.00

1.00

0.94

0.70

0.89

0.03

0.78

0.91

0.64

0.07

0.49

-0.66

0.09

0.04

-1.55

Research objective• Classification-based MT evaluationthat can identify such critical errors• Two-dimensional• Fluency (including comprehension)• Adequacy• Sentence-based• cf. segment-level annotations by Popovic(CoNLL and COLING 2020)• Both human and automatic evaluation

5

Human evaluation• Dataset: WMT Metrics Task (2015-17)• 9,280 MT results in English• A linguistic data development
company hired three annotators:• Native speakers of English• Work experience in translation intoEnglish• No specific training conducted6

Human evaluation (cont’d)• Evaluation in a monolingual way• The annotators can see only MT results
along with corresponding references• Independent among the annotators• The evaluation corpus is available
under CC BY-NC-SA 4.0• https://github.com/ksudoh/wmt15-17humaneval7

8Evaluation criteria
Fluency

AdequacyIncomprehensible (F)The sentence is not comprehensible.

Poor (D)Some contents are not easy tounderstand by typographical /grammatical errors and problematic
expressions.

Fair (B)All the contents are easy tounderstand in spite of sometypographical / grammatical errors.

Good (A)All the contents are easy tounderstand and free fromgrammatical errors, but some
expressions are not very fluent.

Excellent (S)All the contents are easy tounderstand, and all the expressions
are flawless.

Incomprehensible (F)The contents cannot be understood due tofluency and comprehension issues, so thehypothesis is not eligible for the adequacy
evaluation.

Unrelated(O)The hypothesis delivers information that is not
related to the reference

Contradiction (C)The hypothesis delivers information that
contradicts the reference

Serious (F)The hypothesis delivers information that may
cause serious misunderstanding due to somecontent errors but does not contradict thereference

Fair (B)All the contents are easy to understand in
spite of some typographical / grammaticalerrors.

Good (A)All the contents are easy to understand andfree from grammatical errors, but some
expressions are not very fluent.

Excellent (S)All the contents are easy to understand, and all
the expressions are flawless.

9Analysis: Agreement• Agreement was not high (~0.3)• Similar to previous studies with older
WMT datasets (Callison-Burch+ 2007)• Annotator B was strict in Fluency• Annotator C was strict in Adequacy
Fluency
Adequacy

Kappa
Concordance
Kappa
Concordance

A-B
.286
.451
.395
.546

A-C
.377
.511
.268
.587

B-C
.249
.401
.277
.575

10

Analysis: Human DA scores

Human DA score• Fluency almost works as a Likert scale

1
0.8
0.6
0.4
0.2
0-0.2
-0.4
-0.6
-0.8 Incomprehensible-1

Excellent

Fluency

GoodFair
PoorA

BC

11Analysis: Human DA scores• Interesting finding in Adequacy

Human DA score• Unrelated hypotheses are scored worst• Contradiction are scored better thanother error categories (due to thesimilarity with the reference contents?)
1
0.8
0.6
0.4
0.2
0-0.2
-0.4
-0.6
-0.8
-1

Excellent

Adequacy

GoodFairContradiction
Incomprehensible

Serious
UnrelatedABC

Automatic evaluation• A RoBERTa-based classifier model• Data split• Training: 4,824 from 2015-16• Development: 536 from 2015-16• Test: 3,920 from 2017 (560 each for{cs,de,fi,lv,ru,tr,zh}-en)• Label agreement among annotators• Majority• Pessimistic heuristics (details in paper)

12

Results by confusion matrix• Fluency accuracy: 57.8%• Serious confusion between adjacentcategories>> Prediction>> Correct labels

Fluency

Inc.

Poor

Fair Good Exc.

Incomprehensible

20645

2281

Poor

45266250

434

Fair

15134782358

52Good211187560

139

Excellent

0235306453

13

Results by confusion matrix

14• Adequacy accuracy: 60.0%• Confusion between Serious and Fair
Adequacy

Inc.

Unr.

Con.

Ser.

Fair Good Exc.

Incomprehensible

224

0083

38

41

Unrelated

01013

500

Contradiction

008

913100Serious

3708385

242

450

Fair

29013237

878

274

10Good40920302

77

59

Excellent

000169784

Summary of the results• Fluency• # of serious classification errors withdistant categories was small• Adequacy• Less frequent categories (Unrelated andContradiction) were difficult to predict• Prediction of Excellent seemed good;their actual judgements were mostly
Excellent or Good (93.5%)

15

Conclusions• Classification-based human and automatic MT evaluation• Fluency & Adequacy, motivated• Human evaluation can be improvedfor better agreement• More careful evaluation instruction?

• Automatic evaluation should beimproved for the practical use

16

Future work
• Further development of humanevaluation corpora, not limited toWMT Metrics Task• Data augmentation to tackle the label
imbalance• Shared task data does not fully coveractual MT problems…

• MT training/fine-tuning based onthese evaluation criteria

17