Our poster: MT-LRE 1 (ID: 636) on Apr 21

EACL 2021
19–23 April, 2021

Understanding Pre-Editing for Black-Box
Neural Machine Translation

Rei Miyata (Nagoya University)
Atsushi Fujita (NICT)Overview• Pre-editing is a well-tried practice of using MT•

Controlled language rules for RBMT and SMT [O’Brien 2003; Nyberg+ 2003; Hartley+ 2012]•… but not fully studied particularly for NMT [Hiraoka & Yamada 2019; Mehta+ 2020]• This presentation explains
1. our methodology for collecting fine-grained pre-editing instances
2. our extensive analyses of the collected instances• We revealed•

potential of pre-editing and NMT approach•possible and effective pre-editing operation types•

general tendencies of impact of pre-editing on NMT output

The theme is material developmentwith 22.6%.

テーマは材料開発が二十二．六%でトップ。
Pre-edit

MTThe top theme was material development,
which accounted for 22.6%.

一位のテーマは材料開発で22.6%を占めた。

2

1. Protocol
Incrementally and minimally rewrite an ST on a trial-and-error basiswith the aim of obtaining better MT output [Miyata & Fujita 2017]

Complete = satisfactory output is obtained(information is fully conveyed without grammatical errors)
Give up= satisfactory output is unlikely to be obtained3

1. Protocol
Incrementally and minimally rewrite an ST on a trial-and-error basiswith the aim of obtaining better MT output [Miyata & Fujita 2017]
Org-ST

Best-ST

4

Implementation• Condition•

3 translation directions: Ja-En, Ja-Zh, Ja-Ko•

2 MT systems: Google, TexTra•

4 text domains: hospital conversation, municipal procedure,Japanese-origin news article, English-origin news article• Data•

100 Japanese source sentences(Org-ST, 25 sentences × 4 domains)•for all of the 6 settings (3 translation directions × 2 MT systems)• Editor•

One professional translator for each translation direction•

each worked for 100 Org-STs × 2 MT systems
5Collected pre-editing instances# instances# units

Lang. System

Total

Avg.

Google

1332

13.32

27/10098/100

Textra

1260

12.60

23/10096/100Google

1371

13.71

2/100

91/100

Textra

812

8.12

8/100

94/100Google

950

9.50

1/100

96/100

Textra

927

9.27

8/100

96/100

6652

11.09

69/600571/600

Ja-En

Ja-Zh

Ja-Ko

Total

Org=Satisfactory Best=Satisfactory✓ Potential of pre-editing + NMT
6

2. Analyses of the collected instances

a. Characteristics of pre-edited sentencesb. Diversity of pre-editing operations
c. Impact of pre-editing on NMT

7

a. Characteristics of pre-edited sentences
Org-ST

Best-ST• Data: 100 Org-ST vs. 100 Best-ST (for each of the six conditions)• Viewpoints•

Structural characteristics: length, attachment distance, dependency depth•

Lexical characteristics: Type/Token Ratio (TTR), frequency rank in Wikipedia

8

a. Characteristics of pre-edited sentences
BestOrg

Ja-En

Ja-Zh

Ja-Ko

GoogleTextraGoogleTextraGoogleTextraSentence length

25.4

27.8

26.9

28.6

27.1

27.8

26.9

Attachment distance

1.95

1.97

1.99

1.99

1.99

2.00

1.98

Dependency depth

3.57

3.73

3.68

3.73

3.77

3.78

3.76

Type/Token Ratio

0.398

0.386

0.395

0.387

0.392

0.384

0.392

Word freq. rank (Med.)

170

143

154

143

155

143

169.5

✓ The structural complexity generally increased✓ Low-frequency words tended to be avoided
9

a. Characteristics of pre-edited sentences
BestOrg

Ja-En

Ja-ZhJa-Ko

GoogleTextraGoogleTextraGoogleTextraSentence length

25.4

27.8

26.9

28.6

27.1

27.8

26.9

Attachment distance

1.95

1.97

1.99

1.99

1.99

2.00

1.98

Dependency depth

3.57

3.73

3.68

3.73

3.77

3.78

3.76

Type/Token Ratio

0.398

0.386

0.395

0.387

0.392

0.384

0.392

Word freq. rank (Med.)

170

143

154

143

155

143

169.5

✓ The structural complexity generally increased✓ Low-frequency words tended to be avoided
10

a. Characteristics of pre-edited sentences
BestOrg

Ja-En

Ja-Zh

Ja-Ko

GoogleTextraGoogle

TextraGoogleTextraSentence length

25.4

27.8

26.9

28.6

27.1

27.8

26.9

Attachment distance

1.95

1.97

1.99

1.99

1.99

2.00

1.98

Dependency depth

3.57

3.73

3.68

3.73

3.77

3.78

3.76

Type/Token Ratio

0.398

0.386

0.395

0.387

0.392

0.384

0.392

Word freq. rank (Med.)

170

143

154

143

155

143

169.5

✓ The structural complexity generally increased✓ Low-frequency words tended to be avoided
11

b. Pre-editing operation types
ID
S01
S02
S03
S04
S05
S06
S07
ID
P01
P02
P03
P04
P05
P06
P07
P08P09
P10P11P12P13#1427
19141245#71
283174036161328
17595

Structure
Sentence splitting
Structural change
Use/disuse of topicalisation
Insertion of subject/object
Use/disuse of clause-ending noun
Change of voice
Other structural changes
PhraseInsertion/deletion of punctuation
Use/disuse of chunking marker(s)Phrase reordering
Change of modification
Change of connective expression
Change of parallel expression
Change of apposition expression
Change of noun/verb phrase
Use/disuse of compound noun
Use/disuse of affix
Change of sahen noun expression
Change of formal noun expression
Other phrasal changes[cf. Miyata & Fujita 2017]
ID
C01
C02
C03
C04
C05
C06
C07
ID
F01
F02
F03
F04
F05
F06
ID
O01
O02
O03
O04
ID
E01
E02

#118
21
14
1130
1094
#47
31
12
4
1110
#
61
16
24
#
2216

Content word
Use of synonymous words
Use/disuse of abbreviation
Use/disuse of anaphoric expression
Use/disuse of emphatic expression
Category indication/suppression
Explanatory paraphrase
Change of content
Functional word
Change of particle
Change of compound particle
Change of aspect
Change of tense
Change of modality
Use/disuse of honorific expression
Orthography
Japanese orthographical change
Change of half-/full-width character
Insertion/deletion/change of symbol
Other orthographical changeError
Grammatical errors
Content errors✓ Sentence splitting was not frequently used12

b. Strategies for effective pre-editing• Informational strategies [e.g., Vinay & Darbelnet 1958; Chesterman 1997]•Explicitation, Implicitation, Preservation

Explicitation
Strategy
Information
addition

Use ofclear relation#Example of ST pre-editing
12日は台湾の休日のため休場。

142→ 12日は台湾の休日のため株式市場は休場。

The twelfth is a holiday in Taiwan.
The stock market was closed on the twelfth
due to a holiday in Taiwan.

来院しなくても10日前後で登録のクレジッ Withdraw from your registered credit cardin about 10 days without visiting the hospital.
トカードから引き落としを行います。
103→

来院しなくても10日前後で登録のクレジッ Even if you do not visit the hospital, yourcredit card will be debited in about 10 days.
トカードから引き落としが行われます。

採尿と採便を出してください。
Use of54
narrower sense→ 採尿と採便を提出してください。
Normalisation

MT output

30

Please collect urine and feces.
Please submit urine and stool samples.

単位は億円。

Figures are in billions of yen.

→ 単位は億円です。

The unit is 100 million yen.

✓ Explicitation is the key to the effective pre-editing13

c. Impact of editing operations on MT
Structure

Contentword

PhraseFunctionalword

Orthography✓ Structural edits in ST tend to cause major changes in MT✓ Rewriting functional words and orthography seldom impacts✓ Phrase reordering in ST does not affect MT output much

14

c. Impact of editing operations on MT
Structure

Contentword

PhraseFunctional
word

Orthography✓ Structural edits in ST tend to cause major changes in MT✓ Rewriting functional words and orthography seldom impacts✓ Phrase reordering in ST does not affect MT output much

15

c. Impact of editing operations on MT
Structure

Contentword

PhraseFunctional
word

Orthography✓ Structural edits in ST tend to cause major changes in MT✓ Rewriting functional words and orthography seldom impacts✓ Phrase reordering in ST does not affect MT output much

16

c. Impact of editing operations on MT
Structure

Content
word

PhraseFunctional
word

Orthography✓ Structural edits in ST tend to cause major changes in MT✓ Rewriting functional words and orthography seldom impacts✓ Phrase reordering in ST does not affect MT output much17

Summary
1. 6,652 manual pre-editing instances were collected•across multiple translation directions, MT systems, and domains•

demonstrating the potential of pre-editing + NMT

2. Detailed analyses revealed:•Enhancing the explicitness of an ST is more important thanmaking the ST shorter and simpler•

Although NMT is generally unpredictable, there are recognisable
tendencies of impact of pre-editing on NMT

Future research directions
1. Implementation of an automatic pre-editor for black-box NMT scenarios
2. Development of tools to support human pre-editors

Our poster: MT-LRE 1 (ID: 636) on Apr 2118