Ja to En/Zh/Ko Datasets
for Transla3on Quality Es3ma3onand Automa3c Post-Edi3ng
Atsushi Fujita Eiichiro Sumita
ASTREC, NICT, Japanﬁrstname.lastname@nict.go.jp

NICT QE/APE Dataset (google it!)
Mul3lingual parallel corpus
Japanese (Ja) à English (En), Chinese (Zh), Korean (Ko)l Spoken language: travel (8,783 segs), hospital (1,676 segs)
l 5-tuples for each target language
lhyp: generated by phrase-based SMT systems
n ref, pe, and grade: manually creatednSource segment in Ja (src) 片道だけで買えますか。
Human transla3on (ref) May I get it for one way?
MT output (hyp) Can I buy just one way?
Manually post-edited hyp (pe) Can I just buy a one way 3cket?
Quality grade of hyp (grade) B{S, A, B, C, D}
2BackgroundHuman-machine collabora3on for transla3on
A spectrum of transla3on methods [Hutchins+, 1992]
Humaninvolvement(a)
Humantransla3on(b)
Machine-aided
Human transla3on

Automa3on(c)
Human-aided
MT(d)Fully automa3cMTQues3ons(a) How oaen only human can translate?
l (b)(c) How to collaborate?
l (d) What is a realis3c goal?
ln“Fully automa3c MT is not achievable” [Bar-Hillel, 51]
The word “transla3on” consists of 11 leeers.
4

Prevalent approaches: Post-Edi3ng
Post-edi3ng MT outputs (PEMT)lStandard in industries in Europe [ISO18587, 17]

Automa3c PE (APE)lTrainable with (src, hyp, pe) triplets [Allen+, 00]
src

hyppe
APE
modelsrc

hyphyp’pe

5

Prevalent approaches: Quality Es3ma3on
Quality Es3ma3on (QE) [Blatz+, 04][Specia+, 10]
For the users (!= conﬁdence score of MT systems)l Word-level QE
lne.g., visualiza3on of unreliable parts [Bach+, 11]
Source‫?ن‬E‫و‬F#‫?آ?ة وا‬6"#‫*)('& وا‬#‫داب ا‬-. /0 1234 -567 80 ‫و‬-"9‫& و‬:‫< ز‬9 =>?"7 @(*A> B4‫أ‬

MT output

you totally different from zaid amr , and not to deprive yourself in a basement of imitation
and assimilation .

We predict

you totally different from zaid amr , andand visualize

a basement of imitation and assimilation .

Humanyou are quite different from zaid and amr , so do not cram yourself in the tunnel ofsimulation , imitation and assimilation .

correctionl(a)Sentence-level QE
Sourcenot to deprive yourself in

01 7AC+*D E6F GHI )*IJF ‫**@ات‬LC ‫ل‬+M‫ون ?د‬$%&'( )*(+,-‫ول ا‬$-‫ ا‬01 2*‫رآ‬+56-‫ ا‬78%( ‫ ان‬+9:‫@ ا?>&=<ع ا‬AB‫وا‬
0M+,6-‫*@ ا‬L&-‫*@ات ا‬OPC QRM S*T> .

e.g., whether
hypcanthatbe[Soricut+,10]
the poll
also showedmost directlyof the participants deliveredin the developing countries
are readyto introduce qualitative changes in the pattern of their lives for the sake of reducing theeffects of climate change.
n e.g., rou3ng according to the expected PE labor [Specia, 11]nMT output

We predictsrc

As ishypPost-edit
the survey also showed that most of the participants in developing countries are readyHuman
to introduce changes to the quality of their lifestyle in orderto reduce the transla3on
effects ofthe poll also showed that most of the participants in the developing countries are readyqualitative

to introducechanges in the pattern of their lives for the sake ofand visualize
reducing the effects of climate change.
Human
correction

climate change .

6Limita3ons of exis3ng datasets
Only for European languages
lThose with 10k+ segments# of segments

Prior art

Transla3on
direc3on

DomainMTsystemtrain[Potet+, 12]

Fr à En

news

SMT

10,881[Bojar+, 15]

En à Es

news

SMT

11,271 1,000 1,817[Bojar+, 16]

En à DeIT

SMT

12,000 1,000 2,000[Bojar+, 17]

En à DeIT

SMT

23,000 1,000 2,000De à En

pharmaceu3cal

SMT

25,000 1,000 2,000

dev

test

0 1,500

Only PE-based QE
Word-level QE: OK/BAD tags generated from (hyp, pe) pairsl Sentence-level QE: HTER computed from (hyp, pe) pairslOnly wrieen texts

Can we rely on HTER to decide
which hyp to be delivered?

7

Our 6-step procedure for corpus construc3on
Cover Asian languagesl Cover spoken languagel Conduct also manual gradinglSteps 1 & 2: Crea3on of parallel corpus

1srcref2

9Step 1: Collec3ng Japanese ueerances
Spoken language in two domainsltravel: transcribed ueerancesnlLog data of our speech transla3on app, VoiceTrahospital: wrieen ueerancesn

Manually created role-play dialogs

De-duplica3on & manual cleaning
Non-understandable onesl Ungramma3cal onesl Inappropriate ones wrt. social standards
l10Step 2: Manual transla3onNa3ve speakers of each target language
Instruc3ons
Translate each segmentl Assuming reasonable context
lnl

But without adding too much informa3on

Only one ref, even if the src has more than one interpreta3one.g., ﬁlling zero-anaphor (he/she)n e.g., specifying sg./pl. (book/books)
nsrc 片道 だけ で 買え ます か 。
(one way) (only) (ACC)(buy) (modesty) (INT).

ref May I get it for one way?

11

Steps 3 & 4: Crea3on of QE/APE dataset

src

3

refhyppe4

12

Steps 3 & 4: MT and post-edi3ng
Step 3: src à hyp using phrase-based SMT systems
Transla3onMT
direc3on system

Model# of sentences
for training

Ja à En

SMT

2013/Oct.

736k

Ja à Zh

SMT

2016/Sep.

1.44M

Ja à Ko

SMT

2016/Sep.

1.40M

Step 4: src, hyp, and ref à pel

Correc3ng hyp with minimum number of edits [Snover+, 06]
Dele3on of a word (Del)n Inser3on of a word (Ins)n Subs3tu3on of a word (Sub)n Shia of a word or a phrase (Shia)nl

A lenient criterion: TER(hyp à pe)
nProhibited: copying ref to pe

TER(hyp à ref)13

Detail of compu3ng TER
tercom-0.7.25 w/ OSS tokenizers
En: Moses’s tokenizerl Zh: Characterl Ko: MeCab + mecab-ko-dic
lAsymmetric
For Step 4: TER(hyp à pe), TER(hyp à ref)l For sentence-level QE: TER(pe à hyp) (=HTER)
lsrc あなたは22時までにチェックインする必要があります。
hyp You shouldcheck in by twenty two o'clock .

pe You have to check in by 22:00=Sub Del== =Sub.
InsIns=14

Steps 5 & 6: Addi3onal human evalua3onsrc

6

refgradehyp5

pe15

Step 5: Manual grading
Evaluate the quality of hyp wrt. its src
lOp3onally referring to ref

5-ary classiﬁca3on [Goto+, 13] rather than scoring
S: Perfect (na3ve)l A: Goodl B: Fairl C: Acceptablel D: IncorrectlCorrect (goal of PE)

Some errors (need PE)

16

Deﬁni3on of acceptable classes
S: Perfect (“AA” in [Goto+, 13])lInforma3on of the source text has been completely translated.
There are no gramma3cal errors in the target text. Lexical
choice and phrasing are natural even from a na3ve speaker
point-of-view.

A: Goodl

Informa3on of the source text has been completely translated.
There are no gramma3cal errors in the target text, but lexical
choice and phrasing are slightly unnatural.

17

Deﬁni3on of unacceptable classes
B: Fairl

There are some minor errors in the target text of less important
textual informa3on, but the meaning of the source text can beeasily understood.

C: Acceptablel

Important parts of the source text are omieed or incorrectlytranslated, but the meaning of the source text can be easilyunderstood.

D: Incorrect (“F” in [Goto+, 13])l

The meaning of the source text is incomprehensible from target
text.

18

Step 6: Consistency check
Resolve the discrepancies
If (grade is “S” or “A”) && (pe != hyp)à then, retry both grading and post-edi3ng
l If (grade is “B” or “C” or “D”) && (pe == hyp)à then, retry both grading and post-edi3nglMinimality of post-edits is assessed again (if necessary)l

If TER(hyp à pe) > TER(hyp à ref)à then, retry post-edi3ng

19Analyses of the created datasets

Distribution of segments [%]Results of manual gradingSAB

Ja->Zhtravel

Ja->Kotravel

Ja->EnhospitalCD

Ja->Zhhospital

Ja->Kohospital

10080
60
40
200
Ja->EntravelDirection and domain hyp were ofvery bad quality
21

Proximity between transla3ons
Observa3onsl(a) hyp à ref: automa3c MT evalua3onnl(b) hyp à pe: amounts of post-editsnl

Correlates with the propor3on of “S” and “A”
Signiﬁcantly beeer than (a), except for JaàEn hospital(c) pe à ref: unrelated
nFallen between (a) and (b), except for JaàEn taskse.g., TER scores in the hospital domain
hyp(a) 75.35

hyp(b) 66.03
peref(c) 24.69

JaàEn
Too bad to perform PE(a) 48.54

hyp(b) 8.67
peref(c) 43.78
JaàZh(a) 32.44(b) 4.12
peref(c) 30.00
JaàKo22

Distribu3on of sentence-wise HTER score
Overall: some level of correla3onBut, lower HTER scores do not mean higher quality
3Ja->En, travel
2Ja->Zh, travel
2.5

2.5

21.5HTERHTERHTER

21.5

110.50DCB
Human judgment for MT output

1.5

1.5

0.5
0Ja->Ko, hospital
0.8

10DCB
Human judgment for MT output

10.5

DCB
Human judgment for MT output

0DCB
Human judgment for MT output

HTER

211Ja->Zh, hospital

2HTER

HTER

Ja->En, hospital

1.5

0.5

0.5
0Ja->Ko, travel

0.6
0.4
0.2

DC
B
Human judgment for MT output

0DCB
Human judgment for MT output

23

HTER vs. gradeJa->En, hospital
2HTER

1.5

Too much edits

10.50

Cri3cal errors werecorrected with small
numbers of edits

DC
B
Human judgment for MT output
24

JaàEn examples with HTER/grade mismatch
HTER=0.22 / “D” Drop of nega3onsrc 多額の現金は持ってこないでください。
hyp Please bring a lot of cash.
pe Please don’t bring a lot of cash.

HTER=0.13 / “D” Incorrect referencesrc 首が痛くありませんか。
hyp Do you have pain in my neck?
pe Do you have pain in your neck?

HTER=0.78 / “B” Too much edits
src 素晴らしい景色だね
hyp It’s beau3ful scenery.
pe The scenery’s beau3ful, isn’t it?

HTER and grade deﬁne diﬀerent
objec3ves for sentence-level QE
25

Benchmarking
How the data resemble WMT?
l Are exis3ng methods useful?
lBenchmarking experiments
Tasks
Word-level QEl Sentence-level QE: two variantsl Automa3c Post-edi3ng (APE)lExternal resourcelWe reluctantly used our in-house parallel corpus
Daily life conversa3ons
n Reasonably large
nPar33on # of sentences# of tokens

train

1.57M

20.1M-2.51M

dev

14k

179k-224k

27

Word-level QE: Methods
Feed-forward NN model [Kreutzer+, 15]lRelying on word alignment and context words in a window
src あなた は 22時 まで に チェックイン する 必要 が あり ます 。
hyp You should

check in by twenty two o'clock .

yi-1=You
yi=should
yi+1=checkxa(i)-1=する

P(OK)

xa(i)=必要xa(i)+1=がUse of large pseudo training datalgenerated from (hyp, ref) pairs [Liu+, 17]28

Word-level QE: Results
On F1 score for BAD tags (F1-BAD)
The less the BAD tags were, the lower F1-BAD was
Were pseudo training data useful?
✔ Ja à En and Ja à Zh: as in En à Es [Liu+, 17]l ✗ Ja à Ko: we have not yet found the reason
l0.6

All BAD
ManualPseudo
Manual+Pseudo

F1-BAD

0.4

0.2
BAD=29%

BAD=10%

BAD=7%

BAD=19%

Ja->EnJa->Zh

Ja->KoWMT15 En->Es0From [Liu+, 17]29

Sentence-level QE: Methods
Two variant tasks
Predic3on of HTER as regression [Bojar+, 14;15;16;17]l Binary classiﬁca3onl

No PE is needed: “S” and “A”
n PE is needed: “B”, “C”, “D”n

SVM with heuris3c featuresl

QuEst17: the 17 MT system-independent features [Specia+, 15]nl# of tokens, LM prob., Avg. ambiguity based on phrase table, etc.

SntEmb: distributed representa3on for each of src & hypnAvg. of 300-dim embeddings of cons3tuent words [Shah+, 16]30

Sentence-level QE: Results
SntEmb improved the performance of QuEst features
0.6

Predic3on of HTER
No naïve baselinen On Pearson’s cor. Coeﬀ.
n0.5
Pearson’s rl

QuEst17
QuEst17+SntEmb

0.4
0.3
0.2
0.1
0Ja->En

Binary classiﬁca3on
No WMT counterpart
n No naïve baselinen On F1-mult (F -BAD * F -OK)
11n Two expected usages
nTo ﬁlter out truly BAD hypl To ﬁlter truly OK hypl

Ja->Ko0.5

WMT16 En->De

QuEst17
QuEst17+SntEmb

0.4
F1-multl

Ja->Zh0.3
0.2
0.1
0Ja->EnJa->Zh

Ja->Ko31Conclusion
NICT QE/APE Dataset (google it!)Japanese (Ja) à English (En), Chinese (Zh), Korean (Ko)l Spoken language: travel (8,783 segs), hospital (1,676 segs)
l 5-tuples for each target language
lhyp: generated by phrase-based SMT systems
n ref, pe, and grade: manually creatednSource segment in Ja (src) 片道だけで買えますか。
Human transla3on (ref) May I get it for one way?
MT output (hyp) Can I buy just one way?
Manually post-edited hyp (pe) Can I just buy a one way 3cket?
Quality grade of hyp (grade) B{S, A, B, C, D}
32

Future work(ongoing) Crea3ng QE data for NMT outputs
lgrade only (by Mar. 2018?)
PE is more diﬃcult to performn PE costs signiﬁcantly moren

Evalua3on of recent NN-based methodse.g., Predictor-es3mator NN model for QE [Kim+, 17]l e.g., Mul3-source NN-based APE [Junczys-Dowmunt+, 16]lApplica3on to MT-related taskse.g., Enhancing our transla3on services
l e.g., Filtering automa3cally harvested parallel sentenceslA shared-task in the next WAT?
33