Understanding Pre-Editing for Black-Box Neural Machine Translation
Rei Miyata (Nagoya University; miyata@nuee.nagoya-u.ac.jp)
1. Objectives

4. Analyses

1. Investigate the capability of the pre-editing + black-box NMT
through a human-in-the-loop protocol for collecting pre-editing instances
2. Analyse the instances to understand the diversity and impact of pre-editinga. Characteristics ofpre-edited sentences

Org• Structural complexity ↑• Low-frequency words ↓2. Protocol for Collecting Pre-Edit Instancesb. Diversity of pre-editingoperations

Incrementally and minimally rewrite an ST on a trial-and-error basis with the aim ofobtaining better MT output [Miyata & Fujita 2017]

Tree representation ofversions of STs in a ‘unit’

Original

Complete = satisfactory output is obtained(information is fully conveyed without grammatical errors)
Give up = satisfactory output is unlikely to be obtained

Sentence length
Attachment distance
Dependency depth
Type/Token Ratio
Word freq. rank (Med.)
Subcategory#Information
addition

142

25.4
1.95
3.57
0.398
170

Ja-En
Google Textra
27.8
26.9
1.97
1.99
3.73
3.68
0.386
0.395143154

Example of ST pre-editing
12日は台湾の休日のため休場。

BestJa-Zh
Google Textra
28.6
27.1
1.99
1.99
3.73
3.77
0.387
0.392
143
155

Ja-KoGoogle Textra
27.8
26.9
2.00
1.98
3.78
3.76
0.384
0.392
143
169.5

MT output
The twelfth is a holiday in Taiwan.

→ 12日は台湾の休日のため株式市 The stock market was closed on the• 39 pre-editing operation
twelfth due to a holiday in Taiwan.
場は休場。
types under 6 categories
来院しなくても10日前後で登録 Withdraw from your registered credit• Structure / Phrase /のクレジットカードから引き落と card in about 10 days without visitingthe hospital.
Use ofしを行います。
Content word /103
clear relation
Functional word /来院しなくても10日前後で登録 Even if you do not visit the hospital,→ のクレジットカードから引き落と your credit card will be debited in
Orthography / Error
about 10 days.
しが行われます。
• Classification based on the
採尿と採便を出してください。
Please collect urine and feces.
Use of54
information strategies
narrower sense→ 採尿と採便を提出してください。 Please submit urine and stool samples.
• Explicitation (37%)
単位は億円。
Figures are in billions of yen.
Normalisation 30• Implicitation (10%)
The unit is 100 million yen.
→ 単位は億円です。
• Preservation (54%)

c. Impact of pre-editing
operations on NMTBestAtsushi Fujita (NICT)

Structure

PhraseContentword

Functional
word

Orthography

Phrase reordering

TER between
MT outputs

3. Collected Instances• 3 translation directions: Ja-En, Ja-Zh, Ja-Ko# instances
Lang.
Ja-En

• 2 MT system: Google, TexTra• 100 STs in Japanese from 4 domains

Ja-Zh(hospital conversation, municipal procedure,Japanese-origin news article, English-origin news article) Ja-Ko• One professional translator for eachtranslation direction# satisfactory versionsSystem

Total

Avg.

OrginalBestGoogle

1332

13.32

27/10098/100

Textra

1260

12.60

23/10096/100Google

1371

13.71

2/100

91/100

Textra

812

8.12

8/100

94/100Google

950

9.50

1/100

96/100

Textra

927

9.27

8/100

96/100

6652

11.09

69/600571/600

Total

Attainable accuracy of pre-editing + NMT• Structural edits→ Major impacts• Functional words→ Minor impacts

5. Findings• Enhancing the explicitness of an ST is more important
than making the ST shorter and simpler• Although NMT is generally unpredictable, there arerecognisable tendencies of impact of pre-editing on NMT

6. Future Application• Tools to support human pre-editors• Automatic pre-editors for black-box NMT• Robust NMT models using our collected data