Dissecting Human Pre-Editing toward Better Use of Off-the-Shelf Machine Translation Systems
Rei Miyata (Nagoya University; miyata@nuee.nagoya-u.ac.jp)
1. Objective

Example of Best ST with satisfactory MT quality
STMT output
同国は、前年の過剰輸出と、今年の減産 Excess exports in the previous year, and reduced
Original によって、穀物不足に直面しており、大量 production this year, is facing a shortage of grain,a large amount of wheat imports plan.の小麦輸入の計画を表明している。
当年の減産と前年の過剰輸出による穀物の Based on the shortage of grain due to production
25
cuts in the current year and excessive exports
Best
不足をふまえ、この国は小麦を大量に輸
edits
last year, this country has announced plans to入する計画を表明している。
import a large amount of wheat.

1) Investigate the capability of the pre-editing strategy• Design a human-in-the-loop protocol to collect pre-edit instances• Collect pre-edit instances in Japanese-to-English translation tasks on 4 datasets
2) Provide an overview of possible edit operations• Create a typology of edit operations

2. Protocol for Collecting Pre-Edit Instances
Human editors incrementally edit source texts (STs) relying on their introspection,
so that improved MT quality is achieved. (Miyata et al. 2015)
New features:• Record ST after every minimal edit is performed
Tree representation of• Allow editors to resume editing from any given past versions of ST
versions of STs in a ‘unit’
Status“In progress”“Complete”“Give up”

Atsushi Fujita (NICT)

Source text (ST)MT output

Original 1
2

Versions of ST

The country, currently battling an acute grain shortage due to excessive exports last year,
Reference
faces a poor harvest this year and intends to import large quantities of wheat.

4. MTranslatability for Different Languages
Human evaluation of MT quality (Chinese & Korean)

ST: Original vs. Best• TL: Chinese & Korean• MT: TexTra• Criterion: 5-point scale•

7

Corresponding MT output

3Best version

Child node(s)Chinesehosp
muni
bccwj
reuters

Avg. score

Num. of units(Org vs. Best)

Org

Best＞

＝

＜2.73 2.93 **2.84 2.89
2.39 2.75 **2.61 2.77732
13227031424523374533

Best path of 10 randomly sampled units= 979 instances of primitive edit operations•

5

Original version

6hospmuni
bccwj
reuters

Org

Best＞＝＜3.32 3.56 **3.58 3.67
3.37 3.60 *3.31 3.36

1232
18245729474731393529

5. Typology of Edit Operations

4Parent nodeKorean

Avg. score

Num. of units(Org vs. Best)

8 BestFrequent types
C01: Alternative lexical choice
一度 → 一回 (once): Best Path

Assess the MT output for the present ST on a 5-point scale criterion (5: Perfect — 1: Incorrect/nonsense).
Step 1
Go to Step 4, if it has satisfactory quality (>=4: Good); otherwise, go to Step 2.
Step 2 Select one version of ST to be edited, and go to Step 3; if none is likely to achieve satisfactory quality, go to Step 4.
Step 3 Minimally edit the selected version of ST, while keeping the meaning of the ST. Go back to Step 1.
Step 4 Choose one version of ST that achieves the best MT quality (Best ST).

•53 types of edit operations in 7 major categories:(learn)
習得する→学ぶ
S (Structure), C (Content word), F (Functional word),
T (Terminology), O (Orthography), I (Information),
S05: Phrase reordering
E (Edit that causes/resolves error in ST)
S07: Insertion/deletion of punctuation
Domain specific types (e.g. news domain)

3. Pilot Run

Less frequent type (not covered by related work)

4 datasets: hospital conversation (hosp), municipal information (muni),
2 news articles (bccwj: Japanese-origin & reuters: English-origin)• MT system: TexTra (freely-available, state-of-the-art phrase-based SMT)
• A Japanese editor with ample experience in evaluating MT quality

S13: Head-switching of verb phrase•

Statistics of the collected dataDataset

Modehosp spoken
muni written
bccwj written
reuters written

Avg. num. of tokens
in original ST (s.d.)12.1 (4.5)
21.3 (12.0)
26.9 (16.0)
34.8 (12.6)

懸念を強め(strengthen anxiety)→ 強い懸念を抱き (have strong anxiety)More than 85% of the STs achieved satisfactory quality
Num. of versions of ST
Total
Avg. Med. Max

1199
2119
3823
5546

12.0
21.2
38.2
55.5

3142645

105
89
209
258

Num. of units
Original=Best Satisfactory quality

40
30
4

S15: Use/disuse of clause-ending noun
S20: Use/disuse of nominal/verbal suffix

9797
8693

6. Conclusion and Future Work1) Collected 12,287 pre-edit instances• High capability of the off-the-shelf MT system• Improved Chinese and Korean translatability2) Built a typology of a wide range of editoperations

前月比で → 前月に比べて(compared to the previous month)

Develop an automatic pre-editor• Controlled language formulation
by assessing the effectivenessof each type of edit operation• Statistical model based on ourcollection of pre-edit instances