Dissecting Human Pre-Editing toward Better Use of Off-the-Shelf
Machine Translation Systems
Rei Miyata†
Atsushi Fujita‡†
Nagoya University
Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
miyata@nuee.nagoya-u.ac.jp‡
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japanatsushi.fujita@nict.go.jp

Abstract
Machine translation (MT) systems are notable to always produce translations ofhuman-level quality. As a practical meansof such MT systems, we investigated thepotential of pre-editing strategy, by collecting actual pre-edit instances using ahuman-in-the-loop protocol. In our study,
targeting Japanese-to-English translation
on four different datasets and using an offthe-shelf MT system, we collected a total of 12,687 pre-edit instances for 400
source sentences and showed promising
results; more than 85% of source sentences turned out to be accurately translated by the MT system. We also foundthat the pre-edited Japanese source sentences were better translated into Chineseand Korean, confirming the usefulness of
pre-editing strategy in a multilingual setting. Through decomposing the collected
pre-edit instances, we built a typology ofprimitive edit operations comprising 53
types, which unveils the subjects for further research.

1

Introduction

Given the improved quality of machine translation(MT) and the increased demand for rapid delivery of translations, a number of off-the-shelf MT
systems have become available. However, noneof them can guarantee that their raw outputs arealways of sufficient quality. When we consider
embedding such MT systems in computer-aided
c 2017 Rei Miyata and Atsushi Fujita. Licensed under the⃝Creative Commons BY-ND 4.0 license. Some rights reserved.
https://creativecommons.org/licenses/by-nd/4.0/

translation (CAT) settings, it is indispensable to explore practical means to obtain high-quality translations without configuring the MT systems.
One option to make better use of such MT
systems is to edit source text (ST) so that it isamenable to the targeted MT system, i.e., preediting. As demonstrated in the literature, preediting ST leads to improved MT quality (Bernthand Gdaniec, 2001; Miyata et al., 2015) and reduced post-editing effort (Pym, 1988; O’Brien andRoturier, 2007; Aikawa et al., 2007). Controlling
ST is particularly effective in multilingual settings(Ó Broin, 2009).
Several studies have examined human-in-theloop protocols that include pre-editing ST in order to improve MT quality. Uchimoto et al. (2006)
have used back translation as a means to spot nonmachine-translatable spans in ST, which are subsequently served to humans to be edited. Resnik
et al. (2010) have taken advantage of monolingual
human knowledge of the target language to identify spans of ST that are likely to cause translation
errors. Mirkin et al. (2013) have devised an interactive tool for monolingual authors. It suggests appropriate alternatives along with confidence scoresfor MT outputs.
In this paper, we investigate the capability ofthe pre-editing strategy and provide an overview ofpossible edit operations used for pre-editing. First,
we empirically demonstrate the potential usefulness of the pre-editing strategy, i.e., how often
STs turn out to be accurately translated by a targeted MT system. To this end, we designed ahuman-in-the-loop protocol, in which human editors incrementally edit given STs (Section 2), andexperimented with Japanese-to-English translation
tasks on four different datasets (Section 3). Using the original and the best-edited STs, we also

Status“In progress”“Complete”“Give up”

Source text (ST)

Versions of ST

MT output

Corresponding MT output

Best version

Child node(s)Parent node

Original version

Figure 1: Our platform for collecting pre-edit instances: when an edited ST in the upper pane is submitted, it is registered with its MT output in a sequential order as shown in the bottom pane.
investigated the usefulness of the pre-edited STs
in translating Japanese STs into Chinese and Korean (Section 4). To give an overall picture ofpre-editing, we built a typology of edit operations
upon actual pre-edit instances, i.e., pairs of STs
before/after minimal pre-editing, collected throughthe above protocol followed by manual decomposition (Section 5). The typology can act as a guidepost to determine useful operations, such as thosehaving the largest impact on the MT quality andthose that are easy to automate.

2

Protocol for Collecting Pre-editing
Instances

As in Miyata et al. (2015), we ask human editors
to incrementally edit STs relying on their introspection, so that improved MT quality is achieved.
Miyata et al. (2015) collected only the final versions of edited STs and directly compared themwith the originals. In contrast, we aim to observe
the trials and errors of editors and to achieve translations of satisfactory quality as much as possible.
To that end, we developed a Web-based platform,shown in Figure 1, with the following two features.
• We record ST after every minimal edit is performed in order to capture the detailed process of pre-editing.
• We allow editors to resume editing from any
given past version of ST in order to facilitate
their trial and error.

Editors are asked to follow the iterative procedure given below for each original ST. We refer tothe set of collected versions of STs for the same
original ST as a unit.
Step 1. Assess the MT output for the present ST
according to the 5-point scale criterion in
Table 1. Go to Step 4, if it has satisfactory
quality;1 otherwise, go to Step 2.
Step 2. Select one version of ST to be edited, fromthe past versions of STs, referring to the corresponding MT outputs, and go to Step 3; ifnone is likely to achieve satisfactory quality
even if edited, go to Step 4.
Step 3. Minimally edit the selected version of ST,
while keeping the meaning of the ST, referring to the MT output for it, so that the MT
system would be able to generate a bettertranslation. When the edited ST is submitted,
its MT output is automatically generated andregistered together. Go back to Step 1.
Step 4. Choose one version of ST that achieves
the best MT quality among all the versions
registered in the unit (called the Best ST), andterminate the procedure for this ST.
To observe fine-grained edit operations, we instructed editors to make edits primitive as muchas possible in Step 3, showing some examples.
Table 2 shows an example; a phrase reordering forsentence (a) makes sentence (b), and a passiviza1“Perfect” or “Good” quality in our criterion in Table 1.

5. Perfect
4. Good
3. Fair
2. Acceptable
1. Incorrect/nonsense

Information of the original text has been completely translated. There are no grammatical errors
in the translation. Word choice and phrasing is natural even from a native speaker’s point-of-view.
Word choice and phrasing is slightly unnatural, but the information of the original text has been
completely translated and there are no grammatical errors in the translation.
There are some minor errors in the translation of less important information of the original text,
but the meaning of the original text can be easily understood.
Important parts of the original text are omitted or incorrectly translated, but the core meaning of
the original text can still be understood with some efforts.
The meaning of the original text is incomprehensible.

Table 1: Criterion for evaluating MT quality.
(a) 来院しなくても十日前後で 登録のクレジットカー
ドから引き落としを行います。

(b) 来院しなくても登録のクレジットカードから 十日前後で引き落としを 行 い ます。

...............

(c) 来院しなくても登録のクレジットカードから十日前後で引き落としが 行 わ れ ます。

....................

Table 2: Examples of primitive edits on a Japanese
sentence whose meaning is “You’ve registered
your credit card. We will charge on that card inaround 10 days regardless of your visit.”
5
2

3

4

6

1
7

8

Figure 2: Tree representation of versions of STs
shown in Figure 1.
tion of sentence (b) leads to sentence (c). In thiscase, the edit from sentence (a) to sentence (c) is
not considered as primitive.
Also in Step 3, we prohibited editors from registering an ST identical to any past versions of ST.
With this constraint, versions of ST in each unit
form a tree structure, as illustrated in Figure 2.
Each node comprises a version of ST accompaniedby the MT output for it; the number in the nodestands for the chronological order of the version
in each unit, with one node, No. 8 in this example, labeled the Best ST. Every node, except for theoriginal one (No. 1), is derived from a parent node.
It is guaranteed that the path between the Best ST
and the original one (henceforth, best path) in each
unit, e.g., gray nodes in Figure 2, contains edit operations effective in improving MT quality.

3

Pilot Run

Using our protocol presented in Section 2, we collected versions of STs and pre-edit instances inJapanese-to-English translation of four sets of STs

in three domains: hospital conversation2 (hosp),
living information provided by municipalities3(muni), and two types of news articles, Japaneseorigin ones from BCCWJ4 (bccwj) and Englishorigin ones from Reuters5 (reuters). While hosp
is spoken, the others are written; sentence length
is markedly diverse (see also Table 3). These domains are so different from each other that we expect that the applicability of our proposed protocol
can be evaluated from diverse points of view. For
each dataset, we randomly sampled 100 Japanese
sentences and used them as original STs.
As the off-the-shelf MT system, we used
TexTra,6 a freely-available, state-of-the-art phrasebased statistical MT system, through its REST
API. We assigned the pre-editing task to one native Japanese speaker who has a good commandof English and ample experience in evaluating thequality of various types of MT systems according
to the criterion in Table 1, while she has no priorknowledge of TexTra.
As a result, 13,087 versions of STs and thus
12,687 pre-edit instances were collected; see
Table 3 for statistics. As shown in the rightmost
column, more than 85% of the STs were ended
with MT outputs of satisfactory quality. This
demonstrates the high potential of the MT system
when proper human intervention is incorporated.
In general, the longer the original ST was, the more
edit operations were required to attain satisfactory
quality. Table 4 shows an example of the Best ST
of a unit in reuters, which was obtained after 25
consecutive edits in the best path and the MT output of which met satisfactory quality.
2

An in-house speech transcription corpus of conversationalutterances in a hospital.
3
Excerpts from websites of municipalities in Japan (Miyata etal., 2015).
4
http://pj.ninjal.ac.jp/corpus_center/bccwj/
5
http://www2.nict.go.jp/univ-com/multi_
trans/member/mutiyama/jea/reuters/
6
https://mt-auto-minhon-mlt.ucri.jgn-x.
jp/Dataset

Mode

hosp
muni
bccwj
reuters

spokenwrittenwrittenwrittenAvg. num. of tokens
in original ST (s.d.)12.1 (4.5)
21.3 (12.0)
26.9 (16.0)
34.8 (12.6)

Num. of pre-edit instances
Total Avg. Med. Max
1199 12.031052119 21.2
14893823 38.2
26
2095546 55.5
45
258

Num. of units
Original=Best Satisfactory quality
40
97
3
97
0
86
4
93

Table 3: Statistics of the collected data.

Original
Best
ReferenceST
MT output
同国は、前年の過剰輸出と、今年の減産 Excess exports in the previous year, and reduced production
によって、穀物不足に直面しており、大 this year, is facing a shortage of grain, a large amount of wheat
量の小麦輸入の計画を表明している。
imports plan.
当年の減産と前年の過剰輸出による穀物 Based on the shortage of grain due to production cuts in theの不足をふまえ、この国は小麦を大量に current year and excessive exports last year, this country has
輸入する計画を表明している。
announced plans to import a large amount of wheat.
The country, currently battling an acute grain shortage due to excessive exports last year, faces a poor
harvest this year and intends to import large quantities of wheat.

Table 4: An example of Best ST with satisfactory MT quality.

Original
Best
ReferenceST
MT output
ＷＳＣによると、４日には弱い複数の降雨の可能 WSC, although the possibility of weak more rainfall
性があるものの、５–６日には全般に乾燥した天候 within 4 days, the weather in general dry return to 5-6が戻る見通し。
days.
ＷＳＣによると、４日には弱い降雨の可能性が存 WSC said, while the possibility of a weak rain exists
在する一方で、５日から６日にかけては、乾燥し on June 4, from June 5 to 6, the dry weather comes
た天候が全般に戻ってくる見込み、とのこと。
back, in general.
WSC said the outlook was for a chance of a few light showers on 4th, and generally dry conditions on 5th
and 6th.

Table 5: An example of Best ST for which our protocol cannot achieve satisfactory MT quality.
It should also be noted that 27 out of 400 units
did not attain satisfactory quality in our human-inthe-loop protocol. Among these “Give up” cases,
we identified that mis-translation of proper nouns
and incorrect lexical choices were the most difficult types of MT errors to rectify. For example,
the Best ST in Table 5 contains expressions fordates, “4 日,” “5 日,” and “6 日,” proper translationsof which are “4th,” “5th,” and “6th,” respectively.
The MT system specified “June” improperly. Thiserror stems from the wrong phrase alignment in thestatistical model. These types of errors should be
addressed during training the models and/or postediting, rather than pre-editing. Our protocol enables us to identify MT errors that are difficult toamend only by the pre-editing strategy. This will
eventually help us streamline the overall translation workflow using off-the-shelf MT systems.

4

Machine Translatability into Different
Languages

We examined the effectiveness of the pre-editing
strategy in a multilingual translation setting, i.e.,
whether an ST, edited so that it is better translated
into one target language, can also be better translated into other languages. First, all the original

and the Best STs in the four datasets (800 sentences in total) were translated into Chinese andKorean using the corresponding models of TexTra.
Then, for each set of Chinese and Korean translations, one human evaluator was asked to assess theMT quality using the 5-point scale in Table 1.
As shown in Table 6, the MT quality for the BestSTs was, on average, higher than that for the original STs for all the datasets, indicating that edit
operations that improved English-translatability ofJapanese STs are portable to Chinese- and Koreantranslatability to a certain degree. For both languages, the MT quality for the Best STs in hosp
and bccwj well surpassed that for the originalSTs, while there were no significant improvements
in muni and reuters. Further scrutiny into thelanguage dependency of machine-translatability isimportant to justify the pre-editing approach toother target languages and domains.

5 Typology of Edit Operations
We analyzed the diversity of edit operations exhibited during our pre-editing exercise. As mentioned
in Section 2, it is likely that the best path contains
edit operations effective in improving MT quality.
We therefore focused on pre-edit instances in theChinese
hosp
muni
bccwj
reuters
Korean
hosp
muni
bccwj
reuters

Avg. score
Org
2.73
2.84
2.39
2.61

Best
2.93∗∗
2.89
2.75∗∗
2.77

Avg. score
Org
3.32
3.58
3.37
3.31

Best
3.56∗∗
3.67
3.60∗
3.36

Num. of units(Org vs. Best)
>
=
<
7
70 23
32 31 37
13 42 45
22 45 33
Num. of units(Org vs. Best)>=<
12 57 31
32 29 39
18 47 35
24 47 29

Table 6: Results of human evaluation of MT quality: “∗” and “∗∗” indicate significant differences
over “Org(inal ST)” tested by Wilcoxon signedrank test with p < 0.05 and p < 0.01, respectively.
Dataset
hosp
muni
bccwj
reutersNum. of instances in best path(a) raw (b) decomposed (b)/(a)
97
185
1.91
106
186
1.75
174
340
1.95
191
268
1.40

Table 7: The number of decomposed pre-editing
instances in the best path of 10 sampled units.
best path of 10 randomly sampled units for eachof the four datasets. First, we decomposed each of
the sampled 568 instances into a sequence of primitive edit operations, because our editor might notstrictly seek the primitiveness. Indeed, as shownin Table 7, this process increased the number of
instances by from 1.40 to 1.95 times, resulting a
total of 979 instances of primitive edit operations.
We then manually created a typology of edit operations, by categorizing each instance, regarding
surface-level differences of each pair of STs as
clues. Table 8 shows the resulting typology with
53 types of edit operations that cover all of theanalyzed pre-edit instances, with their frequency
in each dataset. We observed an extended varietyof edit operations in our collection, ranging fromones at surface-level, such as insertion/deletion ofpunctuation and word reordering, to various typesof syntactic alternation.
The most frequent type across the datasets was
C01 (Alternative lexical choice), including edit
operations such as replacing “一度” with “一回”
(both mean once), and “習得する” (acquire) with
“学ぶ” (learn). This type of edit operations would
be automated by constructing lexical resources tailored to particular MT systems. We also identified
several frequent types of edit operations that arelikely to be effective for improving MT quality.
For example, S05 (Phrase reordering) and S07(Insertion/deletion of punctuation) can help MT
systems parse the input sentences correctly, which
subsequently leads to better MT outputs.
Some types of edit operations were observed
only in specific domains. For example, we observed S15 (Use/disuse of clause-ending noun)and S20 (Use/disuse of nominal/verbal suffix)
only in the news domain (bccwj and reuters). Bothtypes reflect the fact that the elliptic expressions
often used in news articles could degrade the MT
quality. Our method is also useful to unveil thesekinds of domain-specific issues.
Last but not least, let us describe a less frequent
type of edit operations, i.e., S13 (Head-switchingof verb phrase):[before] 懸念を強め (strengthen anxiety)............
[after] ......
強 ......
い 懸念を抱き (have strong anxiety)This type of edit operation has not been covered byexisting controlled language rule sets in Japanese,such as (Ogura et al., 2010; Hartley et al., 2012;
Miyata et al., 2015), nor even by a comprehensive
typology of paraphrases.7 It is worth exploring towhat extent these types of edit operations are effective in improving MT quality.

6 Conclusion and Future WorkIn this paper, we have presented our human-in-theloop protocol for collecting pre-edit instances. Using this protocol, we collected 12,687 pre-edit instances for four different datasets, demonstratingthat most of the source sentences can be edited
into machine-translatable ones. Human evaluation revealed that, for some datasets, Englishtranslatable Japanese STs significantly improved
the quality of translations into Chinese and Korean. We also built a typology comprising a wide
range of edit operations, and found that alternating
lexical choice was the most frequent one taken by
our editor.
Based on this study, we plan to develop an automatic pre-editor. One approach to this is controlled language formulation by assessing the effectiveness of each type of edit operation (Bernthand Gdaniec, 2001; Miyata et al., 2015). Anotheris to build a statistical model. It is worth investigating data-driven methods based on our collection ofpre-edit instances, although this data do not guar7

http://paraphrasing.org/paraphrase.html

ID
S01
S02
S03
S04
S05
S06
S07
S08
S09
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
S22
S23
S24
S25
S26

Freqency
H M B R
Division/synthesis of sentence(s)4 1 7 2
Use of line break0 3 0 0Use of compound/complex sentence 0 0 0 1
Split of phrase
0 0 1 0Phrase reordering
24 6 22 13
Insertion/deletion of subject
0 2 2 2Insertion/deletion of punctuation
24 5 27 27
Change of scope of subject
0 0 1 1
Use of nominative case “ga” or topic 0 1 3 2marker “wa”
Change of marked element
0 2 11 0
Change of voice
3 1 13 3
Change of restrictive/continuous 2 0 12 13
modification
Head-switching of verb phrase
0 0 0 3
Indication of conditional clause2 7 2 0Use/disuse of clause-ending noun
0 1 3 5
Change of subject in noun phrase
0 0 1 0Use of noun phrase or verb phrase
3 4 9 0Use/disuse of compound verb
2 0 2 0Use/disuse of compound noun
2 7 5 8Use/disuse of nominal/verbal suffix 2 1 10 5
Change of connective expression
6 16 12 13
Change of parallel expression
2 3 1 0Change of apposition expression
0 0 0 5
Change of specification expression 0 0 0 3
Change of locative expression
0 0 0 2
Change of hearsay expression
0 0 0 4
Type

S27
S28
S29
S30
S31
C01
C02
C03
C04
C05
F01
F02
F03
F04
F05
F06
F07
T01
O01
O02
O03
O04
O05
I01
I02
E01
E02

Change of expression for indirectquestion
Change of sahen noun expression
Change of formal noun expression
Change of substantive verb expression
Change of ni-/to-naru expression
Alternative lexical choice
Lexical elaboration
Lexical simplification
Change of reference expression
Use of redundant expression
Use of honorific expression
Change of tense
Change of conjunctive word
Change of auxiliary verb
Insertion/deletion of particle
Use of particle
Use of compound particle
Change of named entity
Orthographical change
Change of sentence-ending expression
Insertion/deletion/change of symbol
Insertion of omitted element
Specification of chunk with brackets
Change of content
Change of nuanceGrammatical errors
Other errors

0 0

0

1

1 2 7 40 1 3 5
1 0 0 10 0 0 1129 36 69 33
5 3 2 10 5 0 00 0 0 10 1 0 1
19 11 14 4
0 3 1 24 4 0 1
1 0 0 04 9 24 9
4 3 3 100 1 1 50 0 3 6
1 7 7 40 1 2 00 6 0 00 0 3 20 5 3 118 20 27 16
0 7 17 6
3 1 4 6
19 0 6 6

Table 8: Our typology of edit operations (H: hosp, M: muni, B: bccwj, R: reuters): The first letter of ID indicates seven major categories: S (Structure), C (Content word), F (Functional word), T (Terminology),
O (Orthography), I (Information), and E (Edit that causes/resolves error in ST).
antee to improve MT quality as directly addressedby post-editing (Simard et al., 2007).

Acknowledgments
We are deeply grateful to the anonymous reviewers for their valuable comments on the earlier version of this paper. This work was partly supported
by JSPS KAKENHI Grant Numbers 25730139 and25240051. One of the corpora used in our study
was created under a MIC program “Promotion of
Global Communications Plan: Research, Development, and Social Demonstration of Multilingual
Speech Translation Technology.”

References
Aikawa, Takako, Lee Schwartz, Ronit King, Monica CorstonOliver, and Carmen Lozano. 2007. Impact of controlled
language on translation quality and post-editing in a statistical machine translation environment. In Proc. of MT
Summit, pages 1–7.
Bernth, Arendse and Claudia Gdaniec. 2001. MTranslatability. Machine Translation, 16(3):175–218.
Hartley, Anthony, Midori Tatsumi, Hitoshi Isahara, Kyo
Kageura, and Rei Miyata. 2012. Readability and translatability judgments for ‘Controlled Japanese’. In Proc. of
EAMT, pages 237–244.

Mirkin, Shachar, Sriram Venkatapathy, Marc Dymetman, andIoan Calapodescu. 2013. SORT: An interactive sourcerewriting tool for improved translation. In Proc. of ACL:
System Demonstrations, pages 85–90.
Miyata, Rei, Anthony Hartley, Cécile Paris, Midori Tatsumi,
and Kyo Kageura. 2015. Japanese controlled language
rules to improve machine translatability of municipal documents. In Proc. of MT Summit, pages 90–103.
O’Brien, Sharon and Johann Roturier. 2007. How portable
are controlled language rules? A comparison of two empirical MT studies. In Proc. of MT Summit, pages 345–
352.
Ó Broin, Ultan. 2009. Controlled authoring to improve localization. Multilingual, Oct./Nov.:12–14.
Ogura, Eri, Mayo Kudo, and Hideo Yanagi. 2010. Simplified
technical Japanese: Writing translation-ready Japanese
documents. IPSJ SIG Technical Reports, 2010-DD78(5):1–8. (in Japanese).
Pym, Peter. 1988. Pre-editing and the use of simplified writing for MT: An engineer’s experience of operating an MT
system. In Proc. of Translating and the Computer 10,
pages 80–96.
Resnik, Philip, Olivia Buzek, Chang Hu, Yakov Kronrod,
Alex Quinn, and Benjamin B. Bederson. 2010. Improving
translation via targeted paraphrasing. In Proc. of EMNLP,
pages 127–137.
Simard, Michel, Cyril Goutte, and Pierre Isabelle. 2007.
Statistical phrase-based post-editing. In Proc. of NAACLHLT, pages 508–515.
Uchimoto, Kiyotaka, Naoko Hayashida, Toru Ishida, andHitoshi Isahara. 2006. Automatic detection and semiautomatic revision of non-machine-translatable parts of asentence. In Proc. of LREC, pages 703–708.