Machine Translation Summit XV
http://www.amtaweb.org/mt-summit-xv

Proceedings of MT Summit XV

Vol. 1: MT Researchers’ Track

Yaser Al-Onaizan & Will Lewis, Editors

MT Summit XV
October 30 – November 3, 2015 -- Miami, FL, USA

Proceedings of

MT Summit XV,
Vol. 1: MT Researchers’ Track
Yaser Al-Onaizan (IBM) & Will Lewis (Microsoft), Eds.

Association for Machine Translation in the Americas
http://www.amtaweb.org©2015 The Authors. These articles are licensed under a Creative Commons 3.0 license, no derivative works, attribution, CC-BY-ND.

Research Program Co-chairs
Yaser Al-Onaizan
Will Lewis

IBM T.J. Watson Research Center
Microsoft Research

Research Program Committee
Yuki Arase
Wilker Aziz
Graeme Blackwood
Thorsten Brants
Marine Carpuat
Daniel Cer
Boxing Chen
Colin Cherry
Jonathan Clark
Marta R. Costa-jussà
Steve DeNeefe
Jacob Devlin
Markus Dreyer
Andreas Eisele
Ahmad Emami
Marcello Federico
Christian Federmann
Minwei Feng
Mikel Forcada
George Foster
Kevin Gimpel
Spence Green
Hany Hassan
Yifan HeFei Huang
Abe IttycheriahPhilipp Koehn
Roland Kuhn
Shankar Kumar
Phillippe Langlais
Gregor Leusch
Mu Li
Lemao Liu
Qun Liu
Yanjun Ma
Wolfgang Macherey
Klaus Macherey
Saab Mansour

Osaka University
University of Amsterdam
IBM Research
GoogleUniversity of Maryland
GoogleNRC
NRC
Microsoft ResearchInstituto Politécnico Nacional
SDL Language Weaver
Microsoft Research
SDL Language Weaver
DGT, European Commission
Bloomberg LabsFBK
Microsoft Research
IBM Watson Group
Universitat d'Alacant
GoogleToyota Technological Institute at Chicago
Lilt Inc
Microsoft Research
New York University
Facebook
IBM
Johns Hopkins University
National Research Council of Canada
GoogleUniversité de Montréal
eBay
Microsoft Research Asia
NICT
Dublin City University
Baidu
GoogleGoogleeBay

Daniel Marcu
Yuval Marton
Arne Mauser
Jonathan May
Arul Menezes
Haitao Mi
Michel Simard
Christoph Tillmann
Taro Watanabe
Bing Xiang
Muyun Yang
François Yvon
Rabih Zbib
Dongdong Zhang
Bing Zhao

ISI/USC
Microsoft
Google
USC Information Sciences InstituteMicrosoft Research
IBM Watson Research Center
NRC
IBM Research
GoogleIBM
Harbin Institute of Technology
LIMSI/CNRS
Raytheon BBN Technologies
Microsoft Research Asia
LinkedinContents
Page
1

Patent Claim Translation based on Sublanguage-specific Sentence Structure
Masaru Fuji, Atsushi Fujita, Masao Utiyama, Eiichiro Sumita and Yuji Matsumoto

17

Japanese News Simplification: Task Design, Data Set Construction, and Analysis of
Simplified Text
Isao Goto, Hideki Tanaka and Tadashi Kumano

32

Learning Bilingual Distributed Phrase Representations for Statistical MachineTranslation
Chaochao Wang, Deyi Xiong, Min Zhang and Chunyu Kit

44

Learning Bilingual Phrase Representations with Recurrent Neural Networks
Hideya Mino, Andrew Finch and Eiichiro Sumita

56

A Pilot Study Towards End-to-End MT Training
Feifei Zhai and Liang Huang

66

Automatic Detection of Antecedents of Japanese Zero Pronouns Using a JapaneseEnglish Bilingual Corpus
Dong Zhan and Hiromi Nakaiwa

80

METEOR for Multiple Target Languages using DBnary
Zied Elloumi, Hervé Blanchon, Gilles Sérasset and Laurent Besacier

90

Japanese Controlled Language Rules to Improve Machine Translatability of
Municipal Documents
Rei Miyata, Anthony Hartley, Cécile Paris, Midori Tatsumi and Kyo Kageura

104

Morphological Constraints for Phrase Pivot Statistical Machine Translation
Ahmed El Kholy and Nizar Habash

117

Using Joint Models for Domain Adaptation in Statistical Machine Translation
Nadir Durrani, Hassan Sajjad, Shafiq Joty, Ahmed Abdelali and Stephan Vogel

131

Machine translation evaluation made fuzzier: A study on post-editing productivity
and evaluation metrics in commercial settings
Carla Parra Escartín and Manuel Arcedillo

145

A Distributed Inflection Model for Translating into Morphologically Rich
Languages
Ke M. Tran, Arianna Bisazza and Christof Monz

160

Bandit Structured Prediction for Learning from User Feedback in Statistical
Machine Translation
Artem Sokolov, Stefan Riezler and Tanguy Urvoy

172

An Empirical Study on Segment Prioritization for Incrementally Retrained PostEditing SMT
Jinhua Du, Ankit Srivastava, Andy Way, Alfredo Maldonado-Guerra and David
Lewis

186

Effects of Word Alignment Visualization on Post-Editing Quality & Speed
Lane Schwartz, Isabel Lacruz and Tatyana Bystrova

200

A Systematic Evaluation of MBOT in Statistical Machine Translation
Nina Seemann, Fabienne Braune and Andreas Maletti

215

Register-Based Machine Translation Evaluation with Text Classification
Techniques
Mihaela Vela and Ekaterina Lapshinova-Koltunski

229

Error-Tolerant Speech-to-Speech Translation
Rohit Kumar, Sanjika Hewavitharana, Nina Zinovieva, Matthew Roy and Edward
Pattison-Gordon

240

Mixed-Domain vs. Multi-Domain Statistical Machine Translation
Matthias Huck, Alexandra Birch and Barry Haddow

256

Korean-to-Chinese Word Translation using Chinese Character Knowledge
Yuanmei Lu, Toshiaki Nakazawa and Sadao Kurohashi

270

Topic Adaptation for Machine Translation of E-commerce Content

Prashant Mathur, Marcello Federico, Selçuk Köprü, Shahram Khadivi and Hassan
Sawaf
283

Machine Translation with Source-Predicted Target Morphology
Joachim Daiber and Khalil Sima'an

297

Improved Beam Search with Constrained Softmax for NMT
Xiaoguang Hu, Wei Li, Xiang Lan, Hua Wu and Haifeng Wang

310

Bilingual Distributed Phrase Representations for Statistical Machine Translation
Peyman Passban, Chris Hokamp and Qun Liu

319

Improving the Search Methods for the Interactive Predictions in CAT Systems
Fatemeh Azadi and Shahram Khadivi

333

Improving Semantic SMT via Soft Semantic Role Label Constraints on ITG
Alignments
Meriem Beloucif, Markus Saers and Dekai Wu

346

Wiktionary-Based Word Embeddings
Gerard de MeloPatent Claim Translation based on
Sublanguage-specific Sentence Structure
Masaru Fuji
fuji.masaru@nict.go.jp
National Institute of Information and Communications Technology, Kyoto, Japan,
and Nara Institute of Science and Technology, Nara, Japan
Atsushi Fujita
atsushi.fujita@nict.go.jp
Masao Utiyama
mutiyama@nict.go.jp
Eiichiro Sumita
eiichiro.sumita@nict.go.jp
National Institute of Information and Communications Technology, Kyoto, Japan
Yuji Matsumoto
matsu@is.naist.jp
Nara Institute of Science and Technology, Nara, Japan
Abstract
Patent claim sentences, despite their legal importance in patent documents, still pose difficulties for state-of-the-art statistical machine translation (SMT) systems owing to their extreme
lengths and their special sentence structure. This paper describes a method for improving thetranslation quality of claim sentences, by taking into account the features specific to the claim
sublanguage. Our method overcomes the issue of special sentence structure, by transferring
the sublanguage-specific sentence structure (SSSS) from the source language to the target
language, using a set of synchronous context-free grammar rules. Our method also overcomesthe issue of extreme lengths by taking the sentence components to be the processing unit for
SMT. The results of an experiment demonstrate that our SSSS transfer method, used in conjunction with pre-ordering, significantly improves the translation quality in terms of BLEU
scores by five points, in both English-to-Japanese and Japanese-to-English directions. The
experiment also shows that the SSSS transfer method significantly improves structural appropriateness in the translated sentences in both translation directions, which is indicated bysubstantial gains over 30 points in RIBES scores.

1. Introduction
Advances in reordering techniques based on syntactic parsing (Isozaki et al., 2010b; de Gispert
et al., 2015), with growing volumes of parallel patent corpora available, have brought significant improvements in the performance of statistical machine translation (SMT) for translating
patent documents across distant language pairs (Goto et al., 2012; Goto et al., 2015). However,among various sentences within a patent document, patent claim sentences still pose difficulties
for SMT resulting in low translation quality, despite their utmost legal importance.
A patent claim sentence is written in a kind of sublanguage (Buchmann et al., 1984;
Luckhardt, 1991) in the sense that it has the following two characteristics: (i) comprising apatent claim by itself with an extreme length and (ii) having a typical sentence structure composed of a fixed set of components irrespective of language, such as those illustrated in Figures
1 and 2. The difficulties in patent claim translation lie in these two characteristics. Regarding
the first characteristic, the extreme lengths cause syntactic parsers to fail with consequent low

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 1

reordering accuracy. Regarding the second characteristic, the high regularity of the claim-specific sentence structure cannot be captured and transferred properly by the models trained only
on the other parts of patent documents, such as the abstract and background description.
This paper presents a method for improving the SMT translation quality of patent claims.
We have developed a system that is used as an add-on to state-of-the-art, off-the-shelf SMT
systems to deal with the sentence structure specific to the patent claim sublanguage. Our method
based on this sublanguage-specific sentence structure (henceforth, SSSS) has two major effects.
(1) Pre-ordering and SMT are applied for each sentence component, rather than for the entire
long sentence. This in effect shortens the input to pre-ordering and SMT, thus improves translation quality. (2) Claim sentences are translated according to the sentence structure, producing
structurally natural translation outputs. We manually extracted a set of language independentclaim components. Moreover, using these components, we constructed a set of synchronous
rules for English and Japanese to transfer the SSSS in the source language to the target language.
The results of an experiment demonstrate these two major effects of our SSSS transfer
method. Regarding the first effect, when used in conjunction with pre-ordering, our method
improves translation quality by five points in BLEU score (Papineni et al., 2002), in both English-to-Japanese and Japanese-to-English translations. Regarding the second effect, gains in
RIBES score (Isozaki et al., 2010a) of over 30 points are obtained, indicating that our SSSS
transfer is effective in transferring an input sentence structure to the output sentence.
Components
Preamble
Transitional phrase
Body

Element
Element
Element

Example strings
An apparatus,
comprising:a pencil;an eraser attached to the pencil; anda light attached to the pencil.

Figure 1. Example of an English patent claim (WIPO, 2014)
Components
Element
Body
Element
Element
Transitional phrase
Preamble

Example strings
鉛筆と；鉛筆に取り付けられた消しゴムと；鉛筆に取り付けられたライトとを備える装置

Figure 2. Japanese patent claim corresponding to Figure 1

2. Transferring Claim-Specific Sentence Structure
While patent claims share a common vocabulary and phrases with the rest of the patent document, they are written in a distinctive way that is different from the rest of the patent document,
comprising a sublanguage of its own. This writing style of patent claims developed through thehistory of filing patent applications, and is now described in the literature. According to the
WIPO Patent Drafting Manual (WIPO, 2014), the fundamental structure of an English claim isthat it is a single sentence consisting of three components:
S → PREA TRAN BODY

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 2

where S denotes the claim sentence, PREA the preamble, TRAN the transitional phrase and
BODY the body. The preamble is an introductory phrase that identifies the category of theinvention, the body is the main component of the claim that describes the elements or purposesof the invention, and the transitional phrase is the component that connects the preamble andthe body.
Figure 1 shows one of the typical structures of English claim sentences, in which the
body of the claim comprises claim elements. Each of the elements is a claim component comprising the invention. Figure 2 shows the structure of a Japanese claim sentence corresponding
to the English claim sentence shown in Figure 1. Note that the sets of components comprising
the claims in the two languages are identical, although the order of components is different in
the two languages.
Our manual analysis revealed that a claim consists of a fixed set of components and theset is common to the two languages. We also found that there are strict generation rules in eachlanguage. For example, the English patent claim sentence in Figure 1 is represented by the setof rules in Figure 3, where ELEM denotes the element component shown in Figure 1. The symbol “+” denotes a non-null list of the preceding components. The corresponding Japanese sentence is represented by another set of rules comprising the same components, as shown in Figure
4.
Having observed a strong regularity in the structure of patent claim sentences acrosslanguages, we represent the structural transfer in the form of synchronous context-free grammar(SCFG). For example, we derive the SCFG rules in Figure 5 by connecting the corresponding
rules in Figures 3 and 4, where the numeric indices indicate correspondences between nonterminals in both constituent trees. We handcrafted a set of SCFG rules for translating patentclaim sentences. The details of the process are presented in Section 3.1
S
TRAN
BODY

→
→
→

PREA TRAN BODY
“comprising:”
ELEM+

Figure 3. Example of generation rules for an English claim sentence
S
TRAN
BODY

→
→
→

BODY TRAN PREA
“備える”
ELEM+

Figure 4. Example of generation rules for a Japanese claim sentence
S
BODY
TRAN

→
→
→

〈PREA① TRAN② BODY③, BODY③ TRAN② PREA①〉〈ELEM+, ELEM+〉〈“comprising:”, “備える”〉

Figure 5. SCFG rules derived from English rules in Figure 3 and Japanese rules in
Figure 4

3. Pipeline for Patent Claim Translation
While patent claim sentences have a distinctive structure, their components, such as the elements and purposes of the claimed inventions, are described with the same vocabulary andphrases in the other parts of patent documents. We therefore implemented the SSSS transfer as
an add-on to off-the-shelf SMT systems. More specifically, given a patent claim sentence in theProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 3

source language, our method translates it through the following three-step pipeline (see also
Figure 6).
A button comprising: a plurality of first ribs integrally formed on the surface of theplate-like base portion, each rib radially extending from a center towards the circumference of the plate-like base portion; and an annular portion integrally formed on thesurface of the plate-like base portion, to which each center ends of the plurality of firstribs are coupled.
(a) Input English sentence[S [PREA A button] [TRAN comprising:] [BODY [ELEM a plurality of first ribs integrally
formed on the surface of the plate-like base portion, each rib radially extending from acenter towards the circumference of the plate-like base portion;] [ELEM and an annularportion integrally formed on the surface of the plate-like base portion, to which eachcenter ends of the plurality of first ribs are coupled.]]](b) Synchronously obtained English SSSS[S [BODY [ELEM a plurality of first ribs integrally formed on the surface of the plate-likebase portion, each rib radially extending from a center towards the circumference ofthe plate-like base portion;] [ELEM and an annular portion integrally formed on the surface of the plate-like base portion, to which each center ends of the plurality of firstribs are coupled]] [TRAN を備える] [PREA A button]](c) Synchronously generated Japanese SSSS[S [BODY [ELEM plate like base portion of circumference towards center from extending
plate like base portion of surface on formed integrally first ribs of plurality , each ribradially;] [ELEM and plate like base portion of surface, plurality of first ribs of each center ends coupled are which to on formed integrally annular portion]] [TRAN を備える][PREA A button]](d) Each SSSS component pre-ordered[S [BODY [ELEM 前記板状ベース部の前記表面で一体に形成され、各々が前記板状ベース部の中心から外周に向かって放射状に延在する複数の第１リブと、] [ELEM 前記板状ベース部の前記表面
で一体に形成され、 前記複数の第１リブ各々の中心端が連結された環状部と、]] [TRAN を備える] [PREA ボタン]](e) Each SSSS component translated by English-to-Japanese SMT

Figure 6. Overview of our translation pipeline

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 4

1.

Step 1. SSSS transfer (Figure 6: (a) → (b), (c)): The given sentence is analyzed
using a set of handcrafted SCFG rules. The goal of this step is not to obtain afine-grained parse tree of the input sentence, but to identify its sublanguagespecific structure, and transfer it to the target language. By the use of the set of
SCFG rules, the components in the given sentence are identified, and simultaneously the sentence structure in the target language is generated.

2.

Step 2. Pre-ordering (Figure 6: (c) → (d)): The words of each component arereordered so that the order becomes close to that in the target language. This
process is performed using a constituent parser. As a result of Step 1, shorter
word sequences are the input to this process, resulting in higher parsing andreordering accuracy.

3.

Step 3. Translation by SMT (Figure 6: (d) → (e)): Each component is translated by an SMT system, and the translated components joined up to form asentence, with words conjugated and conjunctions added as necessary. Again,
as a result of Step 1, shorter components are input that are easier to translate.

The rest of this section elaborates Steps 1 and 2 in turn.
3.1. SSSS Transfer
As described in Section 1, one of the major issues in patent claim translation is that, despite thehigh regularity, the claim-specific sentence structure cannot be captured and transferredproperly by models trained only on the other parts of patent documents.
This step is introduced to identify the structure of the given patent claim sentence and to
generate the structure in the target language simultaneously. This process is performed using a
set of handcrafted SCFG rules. We created the rules in the following manner. First, we manually
analyzed the English and Japanese claim sentences in our development set (described in Section
4.1) and found that each claim sentence is composed of a fixed set of components and that theset is common to the two languages. The set of components U we have identified is as follows:U ∈ {PREA, TRAN, BODY, ELEM, PURP},
where the first four are explained in the previous section, i.e., preamble, transitional phrase,body and element. PURP denotes the purpose component, which is similar to the element component in the sense that they comprise the body component.
We then constructed a set of generation rules for English and Japanese claims using U
as a set of non-terminal symbols, and obtained 8 and 16 generation rules respectively. We obtained a larger number of rules for Japanese, because the writing style of Japanese claim sentences is more flexible than that of English claim sentences. Finally, we handcrafted a total of
16 SCFG rules by combining the generation rules for the two languages that have the same setof symbols on both the left- and right-hand sides, respectively. Table 1 shows the entire SCFG
rule set for English-to-Japanese translation. Our SCFG rules for Japanese-to-English translation
are produced by reversing the above English-to-Japanese generation rules.
In the actual implementation of the SCFG rules, we designed each of the rules in the rule
set to be deterministic, by using regular expressions for obtaining a unique match for a terminal
symbol. For example, to analyze input sentences containing more than one occurrence of thestring “comprising:” we prepared a regular expression to match the first occurrence. This heuristic rule correctly matches the claim string in most cases.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 5

Table 1. SCFG rule set for English-to-Japanese translation
ID
R1
R2
R3
R4
R5

S
S
S
S
S

→
→
→
→
→

R6

S

→

R7
R8
R9
R10
R11
R12
R13
R14
R15
R16

BODY
BODY
TRAN
TRAN
TRAN
TRAN
TRAN
TRAN
TRAN
TRAN

→
→
→
→
→
→
→
→
→
→

SCFG rules〈PREA① TRAN② BODY③, PREA① BODY③ TRAN② PREA①〉〈PREA① TRAN② BODY③, BODY③ TRAN② PREA①〉〈PREA① TRAN② BODY③, PREA① BODY③ TRAN② PREA①〉〈PREA① TRAN② BODY③, BODY③ TRAN② PREA①〉〈PREA① TRAN② BODY③ TRAN② BODY③,
BODY③ TRAN② BODY③ TRAN② PREA①〉〈PREA① TRAN② BODY③ TRAN② BODY③,
PREA① BODY③ TRAN② BODY③ TRAN② PREA①〉〈ELEM+, ELEM+〉〈PURP+, PURP+〉〈“comprising:”, ”備えることを特徴とする”,〉〈“comprising:”, ”備える”〉〈“including:”, ”備えることを特徴とする”〉〈“including:”, ”備える”〉〈“having:”, ”備えることを特徴とする”〉〈“having:”, ”備える”〉〈“wherein:”, ”ことを特徴とする”〉〈“wherein:”, ”する”〉

3.2. Pre-ordering
Another major issue in patent claim translation is that the extreme lengths cause syntactic
parsers to fail with consequent low reordering accuracy. To evaluate the effect of introducing
our SSSS transfer on the translation quality, we also implemented a pre-ordering tool usingstate-of-the-art techniques (Isozaki et al., 2010b; Goto et al., 2012; Goto et al., 2015).
Our pre-ordering method is based on syntactic parsing. First, the input sentence is parsed
into a binary tree structure by using the Berkeley Parser (Petrov et al., 2006). For example,
when “He likes apples.” is inputted into our English-to-Japanese translation system, it is parsed
as shown in Figure 7. Second, the nodes in the parse tree are reordered using a classifier. For
example, according to the classifier's decision, the two children of the “VP” node, i.e., “VBZ”
and “NP”, are swapped, whereas the order of the two children of the “S” node, i.e., “NP” and“VP”, is retained. Once such a decision is made for every node with two children (henceforth,binary mode), the word order of the entire sentence becomes very similar to that in Japanese,i.e., “He (kare wa) apples (ringo ga) likes (suki da) . (.)”
The pre-ordering model is trained on a given parallel corpus through the following procedure (Section 4.5 of Goto et al., 2015):
1. Parse the source sentences of the parallel corpus. 1
2.

Perform word alignment on the parallel corpus.

3.

Reorder words in each source sentence by swapping some binary nodes so that
Kendall's τ over the aligned source and target sentences is maximized. As a

1 Note that we used the parse model trained from the source treebank, while Goto et al. (2015) used theparse model learned via cross-language syntactic projection.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 6

result, every binary node is classified as either SWAP, i.e., the two children of
the node are swapped, or STRAIGHT, i.e., they are not swapped.
4.

With the above data, a neural network classifier is trained for predicting
whether a given node is SWAP or STRAIGHT. 2

The constituent parser is also domain-adapted. The initial parsing model for English wastrained on the sentences in the Penn Treebank 3 as well as 3,000 patent sentences manuallyparsed by the authors. The initial model for Japanese was trained on the EDR Treebank 4 consisting of approximately 200,000 sentences. In contrast to what we did for English, we did notuse patent sentences in Japanese because no annotator was available.
We first parsed 200,000 patent sentences using the initial parsing model. We then built
a patent-adapted (not claim-adapted) parsing model by applying a self-learning procedure(Huang et al., 2009) to the above automatic parses.
(ROOT(S(NP (PRP He))
(VP (VBZ likes)(NP (NNS apples)))(. .)))

Figure 7. Parsing result of “He likes apples.”4. Experiments
We evaluated to what extent our SSSS transfer and pre-ordering improved the translation quality. As mentioned in Section 3, these methods are implemented as an add-on to off-the-shelf
SMT systems. In particular, we used phrase-based SMT (Koehn et al., 2003) as the base system.
We also regard it and its hierarchical version (Chiang, 2005) as baseline SMT systems.
4.1. Data
The training data for SMT consists of two subcorpora. The first is the Japanese-English Patent
Translation data comprising 3.2 million sentence pairs provided by the organizer of the Patent
Machine Translation Task (PatentMT) at the NTCIR-9 Workshop (Goto et al., 2011). We randomly selected 3.0 million sentence pairs. Henceforth, we call this Corpus A. SMT systems
trained on the corpus are reasonably good at lexical selection in translating claim sentences,because the vocabulary and phrases are commonly used in entire patent documents, and Corpus
A is of a substantial size to cover a large portion of them. However, the claim-specific sentence
structure would never be taken into account, as Corpus A does not contain any claim sentences.
To bring claim-specific characteristics into the SMT training, even for the baseline systems, we also used Corpus B comprising 1.0 million parallel sentences of patent claims. These
were automatically extracted from pairs of English and Japanese patent documents publishedbetween 1999 and 2012 using a sentence alignment method (Utiyama and Isahara, 2007). The
concatenation of Corpora A and B was used to train baseline SMT systems, as well as those forour extensions.
2 Note that Goto et al. (2015) learned the SWAP/STRAIGHT classification problem jointly with theparsing source sentences.
3 https://www.cis.upenn.edu/~treebank/
4 https://www2.nict.go.jp/out-promotion/techtransfer/EDR/index.html

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 7

Development and test data were constructed separately from the training data in the following manner. First, we randomly extracted English patent documents from patents filed in
the USA in 2014 and extracted up to the first five claims from each patent document. Then, we
randomly selected 2,000 sentences from the results and asked professional translators specializing in patent translation to translate them into Japanese, without informing them that theirtranslations would be used for tuning and testing SMT systems. Finally, the resulting set of
2,000 sentence pairs was randomly divided into development and test data respectively consisting of 1,000 English-Japanese claim sentence pairs.
4.2. Systems
In this experiment, we regard the implementation of phrase-based SMT in the Moses toolkit(Koehn et al., 2007) with distortion limit of six as the baseline. We examined each of our SSSS
transfer, and pre-ordering modules and their combination over the baseline. For reference, weinvestigated the performance of phrase-based SMT with a larger distortion limit 20, as well as
hierarchical phrase-based SMT.
Throughout the experiments, we used KenLM (Heafield et al., 2013) for training language models and SyMGIZA++ (Junczys-Dowmunt and Szał, 2010) for word alignment. We
used the grow-diag-final method for obtaining phrase pairs. Weights of the models were tuned
with n-best batch MIRA (Cherry and Foster, 2012) regarding BLEU (Papineni et al., 2002) as
the objective. For each system, we performed weight tuning three times and selected for the test
the setting that achieved the best BLEU on the development data.
4.3. Evaluation Metrics
Each system is evaluated using two metrics: BLEU (Papineni et al., 2002) and RIBES (Isozaki
et al., 2010a). Although our primary concern in this experiment is the effect of long distance
relationship, in general, n-gram based metrics such as BLEU alone do not fully illustrate it.
RIBES is therefore used alongside BLEU.
RIBES is an automatic evaluation method based on rank correlation coefficients; RIBES
compares the word order in the SMT translation output with those in the reference. Hence itreadily depicts the effects of drastic rearrangement in sentence components that often occurs
between distant languages. In fact, RIBES has shown high correlation with human evaluationin both English-to-Japanese and Japanese-to-English translation tasks including those in the
PatentMT at the NTCIR-9 Workshop (Goto et al., 2011).

Table 2. BLEU and RIBES scores for all systems
Settings
ID
P1
P1’
H1
P2
P3
P4

SSSS
transfer✔
✔

Pre-ordering

✔
✔

SMT
PB d=6PB d=20HPB
PB d=6PB d=6PB d=6

Results
English-to-JapaneseJapanese-to-English
BLEU
RIBES
BLEU
RIBES
23.9
43.9
21.4
40.2
23.4 (-0.5)
49.1 (+5.2)
22.4 (+1.0)
46.3 (+6.1)
24.3 (+0.4)
53.4 (+9.5)
23.2 (+1.8)
49.6 (+9.4)
24.7 (+0.8)
67.9 (+24.0) 20.8 ( -0.6)
63.8 (+23.6)
23.7 (-0.2)
55.1 (+11.2) 22.3 (+0.9)
74.4 (+34.2)
28.8 (+4.9)
74.9 (+31.0) 27.5 (+6.1)
74.7 (+34.5)

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 8

4.4. Results
Table 2 summarizes the BLEU and RIBES scores for all systems, where the numbers in thebrackets show the improvement over P1, the vanilla PBSMT system. The letter “d” in the SMT
column denotes the distortion limit of the SMT decoder. In both English-to-Japanese and Japanese-to-English directions, the combination of SSSS transfer and pre-ordering, i.e., P4, substantially improved the translation quality in terms of BLEU and RIBES scores. While both
SSSS transfer alone (P2) and pre-ordering (P3) alone also led to drastic increases of RIBES
scores, they achieved only marginal improvement of BLEU scores. Thus the substantial BLEU
improvement derived by their combination suggests that SSSS transfer also contributes to improving the performance of pre-ordering.
4.5. Analysis
Experimental results confirm that translation quality can be improved significantly by using our
SSSS transfer, irrespective of the existence of the pre-ordering process and translation directions. In this section, we first explain how our initial issues, i.e., extreme lengths and sublanguage-specific structures in claim sentences, are resolved by SSSS transfer and pre-ordering.
Subsequently, we provide and an in-depth analysis of the additional benefit of our SSSS transfer,i.e., making SMT inputs short. Finally, we discuss the different trends of the observed gains in
the two translation directions.
Complementary contribution of SSSS transfer and pre-ordering: Figure 8 illustrates a typical sequence of example translations generated by the four configurations, P1 to P4, in ourJapanese-to-English experiment. Throughout the figure, a labelled bracketing scheme is used
to illustrate claim components. The contributions of SSSS transfer and pre-ordering are summarized as follows.
(1)
Contribution of SSSS transfer: The order of components is not changed fromthe input Japanese sentence in P1. However, in P2, with the introduction of SSSS
transfer, the components are well arranged in the order of English. The entire
translation can be better understood by properly generating the transitional
phrase “comprising”. Regarding the translation quality of each component, P1
and P2 do not seem significantly different. In contrast, we obtain a better translation for the second element in P4 than in P3. This is an evidence that SSSS
transfer improves pre-ordering effectively.
(2)

Contribution of pre-ordering: As already demonstrated in the previous work,
pre-ordering techniques are effective in generating translations with a reasonable
word order in the target language. In fact, the words in P3 are better arranged
than in P1: the word order is closer to that of the English reference. However,from the viewpoint of sentence structure, the components are not arranged well,and somehow the preamble is generated twice. Conversely, explicitly teaching
the sentence-level structure through SSSS transfer, i.e., as in P4, suppresses suchan undesirable error. Furthermore, dividing the input into shorter components,
results in the words in each component being properly reordered.

In summary, SSSS transfer and pre-ordering complement each other in generating translations
that are natural both structurally and component-wise.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 9

Effects of shortening SMT inputs: As seen above, pre-ordering works better on components
obtained through SSSS transfer rather than on the entire input sentence. To estimate the shortening effect of SSSS transfer, we compared the distributions of lengths of the processing unitof the succeeding steps, i.e., the entire sentence for P1 and automatically identified claim components in P2. Figure 9 shows the cumulative ratio of original sentences and identified claim
components in English and Japanese, respectively. For example, a point (20,70) on the graphindicates that the sentences having lengths between 1 and 20 comprise 70% of all the sentences.
As clearly illustrated, SSSS transfer considerably shortened the input to pre-ordering and SMT.
This not only makes SMT easier, but also improves the accuracy of syntactic parsing and preordering. Table 3 shows the sentence-wise accuracy of the English parser invoked by our preordering module, calculated on the basis of 100 sentences sampled randomly from the test set.
The parse tree of each sentence is manually checked for correctness. The shorter sentences show
higher accuracy. This means that SSSS transfer does contribute to improving pre-ordering accuracy in addition to transferring sublanguage-specific sentence structure.
Input[S [ELEM ベース管と、] [ELEM 前記ベース管を囲む少なくとも部分的に拡張されたスク
リーンジャケットと] [TRAN を含む] [PREA 井戸スクリーンシステムであって、] [PURP 前
記スクリーンジャケットが、前記ベース管上 に位置決めされる前に拡張され、前記スクリーンジャケットのフィルタ層が、前記スクリーンジャケットの拡張により、前記スクリーンジャケットの外側シュラウドに接触する。]]

Reference[S [PREA A well screen system,] [TRAN comprising:] [ELEM a base pipe;] [ELEM and an
at least partially expanded screen jacket surrounding the base pipe,] [PURP the screen
jacket being expanded prior to being positioned on the base pipe, wherein a filter
layer of the screen jacket contacts an outer shroud of the screen jacket due to expansion of the screen jacket.]][S [ELEM The base tube and] [ELEM the base tube is extended to at least partially surround a screen jacket] [PREA and a screen system well,] [PURP the screen jacket is positioned on the base tube before the expansion of the cooling jacket of the screen,
the screen filter layer by an extension of the jacket of the cooling jacket of thescreen outer shroud contact well screen.]][S [PREA Well screen system] [TRAN comprising:] [ELEM base tube;] [ELEM and the basetube is extended to at least partially surround a screen jacket,] [PURP wherein: thescreen jacket is positioned on the base tube before extended to; and the screen jacket
filter layer , said screen jacket by the expansion of the screen jacket outer shroud
contact.]][S [PREA Well screen system] [ELEM base tube,] [ELEM well screen system] [PREA including a screen jacket extended at least partially surrounding the base tube,] [PURP
and the screen jacket, expansion before the positioning to the base tube, the screen
jacket filter layer, the expansion of the screen jacket contacts the outer shroud of thescreen jacket.]][S [PREA Well screen system] [TRAN comprising:] [ELEM base tube;] [ELEM and at least
partially extended screen jacket surrounding the base tube,] [PURP wherein: thescreen jacket, expansion before the positioning to the base tube; and the screen
jacket filter layer contacts the outer shroud of the screen jacket by the expansion of
the screen jacket.]

P1
BaselineP2
w/ SSSStransferP3
w/ pre-ordering

P4
Pipeline

Figure 8. Example Japanese-to-English translation: The bracket information in the input, reference, P1 and P3 are not determined automatically. We indicated them for explanation purpose only.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 10

(a) English input(b) Japanese input

Figure 9. Cumulative ratio of inputs to SMT with respect to the number of words,
with and without SSSS transfer

Table 3. Parsing accuracy of English parser used for English-to-Japanese pre-ordering
Number ofwords insentence
1-20
21-40
41-60
61-80
81-100101-120120-140

Number ofsampled sentences
10
35
18
5
9
5
2

Number of correctly parsed
sentences
10
32
11
2
1
0
0

Sentence-wise
accuracy
100%
91%
65%
40%
11%
0%
0%

Different trends for translation directions: In terms of RIBES score, pre-ordering improved
Japanese-to-English translation substantially, while showing less improvement in the Englishto-Japanese setting. We speculate that the difference lies in the difficulty of pre-ordering, andmore specifically, in the difficulty of parsing sentences in the source language. As Japanese isa strictly head-final language, parsing sentences is easier than in English. Consequently, preordering alone achieved almost the entire gain in the Japanese-to-English setting. Conversely,
English sentences are much more difficult to parse than Japanese. As a result, the pre-ordering
module can sometimes fail to bring the English word order close to that in Japanese. Nevertheless, as a result of SSSS transfer, which divides an input English sentence into shorter pieces,pre-ordering became more accurate, and the RIBES score was further improved.

5. Related WorkThe quality of machine translation across distant languages has been improved as a result of therecent introduction of syntactic information into SMT (Collins et al., 2005; Quirk et al., 2005;
Katz-Brown and Collins, 2008; Sudo et al., 2013; Hoshino et al., 2013; Cai et al., 2014; Goto

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 11

et al., 2015). One of the promising avenues for further improvement appears to be the incorporation of sublanguage-specific information (Buchmann et al., 1984; Luckhardt, 1991). This isparticularly important for translating formalized documents that tend to form sublanguage-specific document structures and sentence structures. In dealing with structures across close language pairs, an early study of sublanguage introduced the notion of flat trees which represents
both source and target sentences using minimal depth structures for facilitating the transfer between the source and target structures (Buchmann et al., 1984). Much of the recent work relating
to document and sentence structures between close languages focuses on structures centered on
discourse connectives (Miltsakaki et al., 2005; Pitler and Nenkova, 2009; Meyer et al., 2011;
Hajlaoui and Popescu-Belis, 2012; Meyer et al., 2012) and on resolving the ambiguity of discourse connectives connecting structural components.
Conversely, when dealing with structures across distant language pairs, a more comprehensive approach is more appropriate. A wide range of research has been conducted in thisdirection. A study by Marcu et al. (2000) proposed a method for improving Japanese-to-English
translation by transforming the source structure generated by a rhetorical structure theory (RST)parser, to the corresponding target structure. Some work in this direction has been conducted intranslations across distant languages, in which the source text is parsed using an RST parser,
and translation rules are automatically extracted from the source and target pair (Kurohashi andNagao, 1994; Wu and Fung, 2009; Joty et al., 2013; Tu et al., 2013). There are also approachesof simplifying long sentences by capturing the overall structure of a sentence, or a group ofsentences. The skeleton-based approach (Mellebeek et al., 2006; Xiao, 2014) attempts to extract
the key elements/structure (or skeleton) from the input sentence using a syntactic parser. The
divide-and-translate approach (Shinhori et al., 2003; Sudo et al., 2010; Hung et al., 2012) also
makes use of syntactically motivated features, such as phrases and clauses, for extracting subcomponents to be translated by SMT. There are also studies on pattern translation (Xia et al.,
2004; Murakami et al., 2009; Murakami et al., 2013) and sentence segmentation (Xiong et al.,2009; Jin and Liu, 2010) for dealing with long input sentences with complex structures.
Our approach is similar to the above models in the sense that it incorporates structural
information into SMT, but differs in that it uses sublanguage-specific sentence structures, rather
than syntactically motivated structures. This results in significant improvement in translation
quality for the claim sublanguage using only a handful of rules.

6. Conclusion
In this paper, we described a method for transferring sublanguage-specific sentence structure
for English-to-Japanese and Japanese-to-English patent claim translations. The experimental
results show that our proposed method, a combination of SSSS transfer and pre-ordering basedon syntactic parsing, achieved five point gains in BLEU scores, in both English-to-Japaneseand Japanese-to-English directions. In addition, a substantial gain of more than 30 points in
RIBES scores was observed in both SMT settings, indicating a significant contribution of SSSS
transfer. We achieved these results with only a handful of SCFG rules.
Our proposed method successfully improved the translation of patent claims with quality
comparable to that of the other parts of patent documents. In our future work, we will concentrate on the translation of independent claims which are the longest and most complex of claim
sentences.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 12

References
Buchman, Beat, Susan Warwick and Patrick Shann. (1984). Design of a Machine Translation System fora Sublanguage. In Proceedings of the 9th International Conference on Computational Linguistics,
pages 334-337.
Cai, Jingsheng, Masao Utiyama, Eiichiro Sumita and Yujie Zhang. (2014). Dependency-based Pre-ordering for Chinese-English Machine Translation. In Proceedings of the 52nd Annual Meeting of
the Association for Computational Linguistics, pages 155-160.
Cherry, Colin and George Foster. (2012). Batch Tuning Strategies for Statistical Machine Translation. In
Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 427-436.
Chiang, David. (2005). A Hierarchical Phrase-Based Model for Statistical Machine Translation. In Proceedings of Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 263-270.
Collins, Michael, Philipp Koehn and Ivona Kucerova. (2005). Clause Restructuring for Statistical Machine Translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 531–540.
de Gispert, Adrià , Gonzalo Iglesias and Bill Byrne. (2015). Fast and Accurate Preordering for SMT using Neural Networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies, pages 1012-1017.
Goto, Isao, Bin Lu, Ka Po Chow, Eiichiro Sumita and Benjamin K. Tsou. (2011). Overview of the Patent Machine Translation Task at the NTCIR-9 Workshop. In Proceedings of the 9th NII Test Collection for Information Resources (NTCIR) Conference, pages 559-578.
Goto, Isao, Masao Utiyama and Eiichiro Sumita. (2012). Post-ordering by Parsing for Japanese-English
Statistical Machine Translation. In Proceedings of the 50th Annual Meeting of the Association for
Computational Linguistics, pages 311-316.
Goto, Isao, Masao Utiyama, Eiichiro Sumita and Sadao Kurohashi. (2015). Preordering using a TargetLanguage Parser via Cross-Language Syntactic Projection for Statistical Machine Translation. In
ACM Transactions on Asian and Low-Resource Language Information Processing, Vol. 14, No. 3,
Article 13, pages 1-23.
Hajlaoui, Najeh and Andrei Popescu-Belis. (2012). Translating English Discourse Connectives into Arabic: A Corpus-based Analysis and an Evaluation Metric. In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages, pages 1-8.
Heafield, Kenneth, Ivan Pouzyrevsky, Jonathan H. Clark and Philipp Koehn. (2013). Scalable Modified
Kneser-Ney Language Model Estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 13-21.
Hoshino, Sho, Yusuke Miyao, Katsuhito Sudoh and Masaaki Nagata. (2013). Two-Stage Pre-ordering
for Japanese-to-English Statistical Machine Translation. In Proceedings of the International Joint
Conference on Natural Language Processing, pages 1062–1066.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 13

Huang, Liang, Hao Zhang, Daniel Gildea and Kevin Knight. (2009). Binarization of Synchronous Context-Free Grammars. In Journal of Computational Linguistics, Volume 35 Issue 4, pages 559-595.
Hung, Bui Thanh, Nguyen Le Minh and Akira Shimazu. (2012). Divide and Translate Legal Text Sentence by Using its Logical Structure. In Proceedings of 7th International Conference on
Knowledge, Information and Creativity Support Systems, pages 18-23.
Isozaki, Hideki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh and Hajime Tsukada. (2010a). Automatic
Evaluation of Translation Quality for Distant Language Pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944–952.
Isozaki, Hideki, Katsuhito Sudoh, Hajime Tsukada and Kevin Duh. (2010b). Head Finalization: A Simple Reordering Rule for SOV Languages. In Proceedings of the Joint 5th Workshop on Statistical
Machine Translation and MetricsMATR, pages 244–251.
Joty, Shafiq, Giuseppe Carenini, Raymond Ng and Yashar Mehdad. (2013). Combining Intra- andMulti-sentential Rhetorical Parsing for Document-level Discourse Analysis. In Proceedings of the
51st Annual Meeting of the Association for Computational Linguistics, pages 486–496.
Jin, Yaohong and Zhiying Liu. (2010). Improving Chinese-English Patent Machine Translation Using
Sentence Segmentation. In Proceedings 2010 International Conference on Natural Language Processing and Knowledge Engineering, pages 1-6.
Junczys-Dowmunt, Marcin and Arkadiusz Szał. (2010). SyMGiza++: A Tool for Parallel Computationof Symmetrized Word Alignment Models. In Proceedings of the International Multiconference on
Computer Science and Information Technology, pages 397–401.
Katz-Brown, Jason and Michael Collins. (2008). Syntactic Reordering in Preprocessing for JapaneseEnglish Translation: MIT System Description for NTCIR-7 Patent Translation Task. In Proceedings of the NTCIR-7 Workshop Meeting, pages 409-414.
Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin and Evan Herbst. (2007). Moses: Open Source Toolkit for Statistical MachineTranslation. In Proceedings of the Association for Computational Linguistics Demo and Poster
Sessions, pages 177–180.
Koehn, Philipp, Franz Josef Och and Daniel Marcu. (2003). Statistical Phrase-Based Translation. In
Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 48–54.
Kurohashi, Sadao and Makoto Nagao. (1994). Automatic Detection of Discourse Structure by Checking
Surface Information in Sentences. In Proceedings of the 15th International Conference on Computational Linguistics, pages 1123-1127.
Luckhardt, Heinz-Dirk. (1991). Sublanguages in Machine Translation. In Proceedings of the Fifth Conference of the European Chapter of the Association for Computational Linguistics, pages 306-308.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 14

Marcu, Daniel, Lynn Carlson and Maki Watanabe. (2000). The Automatic Translation of Discourse
Structures. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pages 9-17.
Mellebeek, Bart, Karolina Owczarzak, Declan Groves,Josef Van Genabith and Andy Way. (2006). A
Syntactic Skeleton for Statistical Machine Translation. In Proceedings of the 11th Conference of
the European Association for Machine Translation, pages 195-202.
Meyer, Thomas, Andrei Popescu-Belis, Najeh Hajlaoui and Andrea Gesmundo. (2012). Machine Translation of Labeled Discourse Connectives. In Proceedings of the Tenth Biennial Conference of the
Association for Machine Translation in the Americas.
Meyer, Thomas, Andrei Popescu-Belis, Sandrine Zufferey and Bruno Cartoni. (2011). Multilingual Annotation and Disambiguation of Discourse Connectives for Machine Translation. In Proceedingsof 12th SIGdial Meeting on Discourse and Dialogue, pages 194–203.
Miltsakaki, Eleni, Nikhil Dinesh, Rashmi Prasad, Aravind Joshi and Bonnie Webber. (2005). Experiments on Sense Annotations and Sense Disambiguation of Discourse Connectives. In Proceedingsof the 4th Workshop on Treebanks and Linguistic Theories.
Murakami, Jin’ichi, Isamu Fujiwara and Masato Tokuhisa. (2013). Pattern-Based Statistical MachineTranslation for NTCIR-10 PatentMT. In Proceedings of the 10th NTCIR Conference, pages 350355.
Murakami, Jin’ichi, Masato Tokuhisa and Satoru Ikehara. (2009). Statistical Machine Translation adding Pattern-based Machine Translation in Chinese-English Translation. In Proceedings of 6th International Workshop on Spoken Language Translation, pages 107-112.
Papineni, Kishore, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Pitler, Emily and Ani Nenkova. (2009). Using Syntax to Disambiguate Explicit Discourse Connectives
in Text. In Proceedings of the Joint conference of the 47th Annual Meeting of the Association for
Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Short Papers, pages 13–16.
Petrov, Slav, Leon Barrett, Romain Thibaux and Dan Klein. (2006). Learning Accurate, Compact, andInterpretable Tree Annotation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics,
pages 433-440.
Quirk, Chris, Arul Menezes and Colin Cherry. (2005). Dependency Treelet Translation: SyntacticallyInformed Phrasal SMT. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 271–279.
Shinhori, Akihiro, Manabu Okumura, Yuzo Marukawa and Makoto Iwayama. (2003). Patent Claim Processing for Readability - Structure Analysis and Term Explanation -. In Proceedings of the Workshop on Patent Corpus Processing, Association for Computational Linguistics, pages 56-65.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 15

Sudoh, Katsuhito, Kevin Duh, Hajime Tsukada, Tsutomu Hirao and Masaaki Nagata. (2010). Divide
and Translate: Improving Long Distance Reordering in Statistical Machine Translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages
418-427.
Sudoh, Katsuhito, Jun Suzuki, Hajime Tsukada, Masaaki Nagata, Sho Hoshino and Yusuke Miyao.
(2013). NTT-NII Statistical Machine Translation for NTCIR-10 PatentMT. In Proceedings of 9th
NTCIR Conference 2013, pages 294-300.
Tu, Mei, Yu Zhou and Chengqing Zong. (2013). A Novel Translation Framework Based on Rhetorical
Structure Theory. In Proceedings of the 51st Annual Meeting of the Association for Computational
Linguistics, pages 370–374.
Utiyama, Masao and Hitoshi Isahara. (2007). A Japanese-English Patent Parallel Corpus. In Proceedingsof the Eleventh Machine Translation Summit, pages 475-482.
The World Intellectual Property Organization (WIPO). (2014). WIPO Patent Drafting Manual. In IP Assets Management Series.
Wu, Dekai and Pascale Fung. (2009). Semantic Roles for SMT: A Hybrid Two-Pass Model. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American
Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers,
pages 13–16.
Xia, Fei and Michael McCord. (2004). Improving A Statistical MT System with Automatically Learned
Rewrite Patterns. In Proceedings of the 20th International Conference on Computational Linguistics, pages 508–514.
Xiao, Tong, Jingbo Zhu and Chunliang Zhang. (2014). A Hybrid Approach to Skeleton-based Translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics,
pages 563–568.
Xiong, Hao, Wenwen Xu, Haitao Mi, Yang Liu and Qun Liu. (2009). Sub-Sentence Division for Treebased Machine Translation. In Proceedings of the Joint Conference of the 47th Annual Meeting of
the Association for Computational Linguistics and 4th International Joint Conference on Natural
Language Processing of the Asian Federation of Natural Language Processing Conference Short
Papers, pages 137–140.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 16

Japanese News Simplification: Task Design, Data
Set Construction, and Analysis of Simplified Text
Isao Goto
Hideki Tanaka
Tadashi Kumano

goto.i-es@nhk.or.jp
tanaka.h-ja@nhk.or.jp
kumano.t-eq@nhk.or.jp

NHK Science & Technology Research Laboratories,
1-10-11 Kinuta, Setagaya-ku, Tokyo 157-8510, Japan

Abstract
In this paper we explore a Japanese news simplification task. We designed a Japanese news
simplification task, constructed the data set for the task, and analyzed the manual simplification process. We designed the task focusing on sentence-level simplification, which is part ofthe process of manual simplification of Japanese news for non-native speakers. We constructed
the data set consisting of Japanese news sentences and their corresponding simplified Japanese
news sentences, and verified the effectiveness of the data set for automatic simplification by
conducting preliminary experiments using phrase-based statistical machine translation. To reveal the processes behind manual simplification, such as simplification associated with word
order (syntactic structure), we analyzed manually simplified Japanese news sentences.

1 Introduction
Simplified texts increase readability and understandability for non-native speakers and children.
Simplified news texts are especially useful for daily living because news delivers informationneeded for life. There are a number of simplified texts available in English, such as Learning
English provided by Voice of America and the BBC, which are multimedia sources of news
and information geared toward learners, Simple English Wikipedia, and simplified technicalEnglish based on the ASD-STE100 standard1 . There is much research on automatic English
simplification (Chandrasekar et al., 1996; Carroll et al., 1998; Petersen and Ostendorf, 2007;
Zhu et al., 2010; Coster and Kauchak, 2011; Woodsend and Lapata, 2011; Wubben et al., 2012;
Kauchak, 2013; Narayan and Gardent, 2014). To realize automatic simplification using statistical machine translation (SMT), a parallel corpus consisting of sentences and their corresponding
simplified sentences is needed. As such, an English parallel corpus was constructed using English Wikipedia and Simple English Wikipedia (Coster and Kauchak, 2011). In contrast, thereis less research on the simplification of languages other than English: Portuguese (Aluı́sio et al.,2008), Spanish (Bott and Saggion, 2011; Bott et al., 2012), Italian (Dell’Orletta et al., 2011),
French (Seretan, 2012), German (Klaper et al., 2013), and Japanese (Inui et al., 2003; Moku andYamamoto, 2012; Tanaka et al., 2013). In these studies, work focusing on Japanese simplification can be summarized as follows. Inui et al. (2003) collected paraphrase readability rankingsby consulting with teachers at schools for the deaf, and trained a paraphrase ranking model fromthe collected rankings. Moku and Yamamoto (2012) performed the automatic simplification of
official documents in Japanese. They concluded through their pre-experiment, however, that
1 http://www.asd-ste100.org/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 17

their method was ineffective. Tanaka et al. (2013) reported on the Internet news service NEWSWEB EASY, which provides manually simplified Japanese news produced through the rewriting of Japanese news. Because the simplification is conducted manually by humans, automation
becomes the key to solving the issue of efficiency. To our knowledge, there is currently no workon the automatic simplification of Japanese news texts using SMT.
In this paper, we explore a Japanese news simplification task. We design the task, construct
the data set for it, then reveal the manual simplification process by analyzing the simplified
text. In the process of manually simplifying Japanese news for the NEWS WEB EASY service(Tanaka et al., 2013), designed for foreigners living in Japan, there are two types of operations: article-level shortening for conciseness (especially when articles are long), and sentencelevel simplification of expressions. News reporters mainly carry out article-level shortening for
conciseness, while Japanese instructors mainly carry out sentence-level simplification of expressions (Section 2). We focus on the process of sentence-level simplification of expressions,which is thought as closer to being serviceable for practical use than automatic article-level
shorting, and define this sentence-level process here as the simplification task (Section 3). We
construct a data set of parallel sentences consisting of sentences from Japanese news prior tobeing initially simplified by the Japanese instructors, and the resulting simplified sentences(Section 4). Then, we conduct preliminary experiments on automatic simplification using thedata set and phrase-based SMT to verify the effectiveness of the data set (Section 5). Additionally, we produce manually annotated word alignments between parallel sentences and reveal
the processes of manual simplification, such as rewriting with a different word order (i.e., syntactic structure) by analyzing the annotated word alignments (Section 6). Our contributions are
summarized as follows:• We proposed a Japanese news simplification task, constructed a data set for the task, andverified the effectiveness of the data set through experiments on automatic simplification
using phrase-based SMT.
• We revealed the processes of manual simplification, such as rewrites associated with word
order (syntactic structure), which are needed to design effective automatic simplification.

2 Simplification in Easy Japanese News Service
In this section we describe the Japanese news simplification processes in the simplified Japanese(easy Japanese) news service called NEWS WEB EASY (Tanaka et al., 2013); our research target for this study. The service is offered by NHK. NEWS WEB EASY is an online service thatprovides easy Japanese news for foreigners living in Japan who have reached pre-intermediatelevel Japanese. Easy Japanese news is mainly produced through two operations: (1) shortening
original news articles for conciseness, especially when the articles are long, and (2) simplifying expressions. Long Japanese news articles can be a burden for non-native speakers to read.
Japanese news articles can be long or short, but there are few long articles that feature easy
to understand Japanese. When longer news articles are shortened, the resulting articles are
sometimes one-half or one-third the original length. The degree of shortening depends on theoriginal length. News reporters mainly shorten with the goal of conciseness, while designatedJapanese instructors2 well-versed in easy Japanese news mainly simplify expressions found in
the text. For each original article to be simplified, a news reporter and a Japanese instructor taketurns rewriting the article. This rewriting process by the reporter and the Japanese instructor isrepeated two or more times, as needed.3
2 These instructors have experience teaching Japanese to non-native speakers.
3 The percentage of original articles first rewritten by Japanese instructors before being rewritten by news reporters
changed over time, as follows. In April 2012, when the NEWS WEB EASY test service started, the rate was 58%.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 18

Typical rewrite flow
Original
news

Objective of automatic simplification(First rewrite by Japanese instructor)

News reportor(Shorten)Japanese instructor(Simplify)

Source sentencesNews reportor(Modify)

Japanese instructor(Modify)

Target sentences

Figure 1: Typical rewrite flow and objective of automatic simplification

3 Sentence-Level Japanese News Simplification Task
Shortening news articles can be technically simulated to some extent using an automatic summarization technique. However, there are problematic issues related to the use of automatic
summarization for actual proper news services. Namely, to shorten long articles into lengthsof one-half or one-third of their original lengths, not only are repetitive sections removed, butsome of the actual news content is also eliminated. Therefore, there is a risk that some of thenews contents that should be delivered to the reader may be deleted. This is a significant riskin terms of accurately conveying news. Additionally, when news reporters shorten an article,
the article is reorganized as needed, and this process requires sophisticated editing. For thesereasons, it is considered difficult to obtain practical-level quality for news services through automatic summarization. In contrast, the operation Japanese instructors perform does not require
shortening and centers mainly on sentence-level simplification of expressions.
Therefore, we divided these two operations, that is, article-level shortening of length andsentence-level simplification of expressions into two separate tasks. In this paper, we define
the sentence-level simplification of expressions, which is performed by Japanese instructors, as
the simplification task. Automating this operation is thought to be more practical for news services, because sentence-level rewrites do not require removing certain content and a statisticalmachine translation technique can simulate this operation to some extent. In this task, the input
sentences for automatic simplification systems are the sentences prior to being simplified by
the Japanese instructors, while the simplified reference sentences are the sentences that are theresults of the initial simplification process performed by the Japanese instructors.

4 Data Set for Japanese News Simplification
We produced a data set for the Japanese news simplification task by constructing parallel sentences. These comprised sentences taken from Japanese news articles and their corresponding
simplified articles. In this section, we describe the construction method and the constructed data
set.
4.1

Construction Method

Here we explain how parallel sentences taken from Japanese news articles and their corresponding simplified Japanese sentences were constructed. In this paper, the sentences prior to being
initially simplified by the Japanese instructors are called source sentences, and their corresponding output simplified sentences are called target sentences. Additionally, a source sentence andits corresponding target sentence(s) are called a parallel sentence pair; similarly, an article prior
to being initially simplified by a Japanese instructor and its corresponding output simplified article are called a parallel article pair.
However, by May 2013, when the official NEWS WEB EASY service started, the rate had dropped to 6%. In September
2014, the rate was down to 1%. Because both Japanese instructors and news reporters began rewriting at the same time
when the service was just beginning, approximately half of the news articles were first rewritten by Japanese instructors.
As time went on, however, most of news articles came to be first rewritten by news reporters because it was discoveredthat it was more efficient to simplify expressions after articles had already been shortened.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 19

Table 1: Noise in parallel sentences consisting of one source sentence and one or more target
sentence(s)Without noise
With noise

Number of sentence
pairs
2,012
873

Rate
0.697
0.303

Table 2: Number of sentences in each manually annotated sentence alignment
Number of sentences(source–target)
1–1
1–2
1–3
1–0
0–1
2–2
2–1
2–3
1–4
OthersFrequency

Rate

2,201
575
90
71
67
52
46
17
16
20

0.698
0.182
0.029
0.023
0.021
0.016
0.015
0.005
0.005
0.006

Manual Extraction of Parallel Sentence Pairs for Test Data
We constructed test sentences and their simplified reference sentences by manual extraction toensure reliability. We manually aligned source sentences with their corresponding target sentences in parallel article pairs. Parallel sentence pairs can be extracted using these sentencealignments. The objective of the simplified reference sentences is to evaluate the quality of
automatically simplified sentences, as each test sentence is automatically simplified independently. Thus, we selected test sentences and their simplified reference sentences from parallel
sentence pairs under the following conditions:• One source sentence corresponds to one or more target sentences.
• We eliminate parallel sentence pairs for which major content in the source sentence is notincluded in its corresponding target sentence(s), or for which content that is not included
in a source sentence is included in its corresponding target sentence(s). There are thefollowing exceptions. When major news content in a source sentence is included in its corresponding target sentence(s), the omission of related detailed information is allowed. In
the target sentences, additional information not dependent on context and which is always
true is allowed.
The first condition is called the alignment condition, and the second condition is called the noise
condition. Here we show an example of additional information that does not depend on context
and is always true: “Aomori Prefecture” in a source sentence and “Aomori Prefecture in the
Tohoku region” in its target sentence.
We manually aligned sentence pairs for 490 parallel article pairs. Among these article
pairs, test sentences and their simplified reference sentences could be extracted from 485 article pairs under the abovementioned conditions. The rates for without noise (satisfying thenoise condition) and with noise (not satisfying the noise condition) for the parallel sentencesthat satisfied the alignment condition are shown in Table 1. The rate for without noise wasapproximately 0.7.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 20

ID

Sentences before rewrite

0 サッカーくじ 初めてワールドカップで売ることにした
日本のサッカーくじは平成１３年（２００１年）に始ま1りました。
売ったお金の一部でスポーツ選手を育てたり、スポーツ
2 施設を作るために、日本スポーツ振興センターが始めました。
Ｊリーグなどの試合が行われるとき、どのチームが勝つ
3 かを予想する「ｔｏｔｏ」や、「ＢＩＧ」など７つのくじを売ります。
しかし、今までワールドカップでくじを売ったことはあ4りませんでした。
日本スポーツ振興センターは、来月始まるワールドカッ
5 プブラジル大会で初めてサッカーくじを売ることになりました。
それによると、日本代表も出る１次リーグの試合で、
6「ｔｏｔｏ」や「ＢＩＧ」を３回に分けて売ります。
そして、決勝トーナメントでも、ゴールの数を予想する
7
くじを３回売ります。
8 このサッカーくじは５月３１日から売り始めます。

Sentencealignments

Null

ID

Sentences after rewrite

0 日本で初めて ワールドカップのサッカーくじ
日本のサッカーくじは平成１３年（２００１年）、日本
1
スポーツ振興センターが始めました。
くじを売ったお金で選手を育てたり、競技場を作ったり
2
しています。
Ｊリーグなどの試合で、「ｔｏｔｏ」や「ＢＩＧ」など
3
７つのくじを売っています。
4 「ｔｏｔｏ」は、どのチームが勝つか予想するくじです。
5 「ＢＩＧ」はコンピューターが勝手に予想するくじです。
日本スポーツ振興センターは、来月ブラジルで始まる
6 サッカーのワールドカップのくじを日本で初めて売ることにしました。
日本も出る１次リーグの試合の「ｔｏｔｏ」や「ＢＩ7Ｇ」などを３回売ります。
また、決勝トーナメントのゴールの数を予想するくじも
8
３回売ります。
日本では、今までワールドカップのサッカーくじを売っ9たことがありませんでした。
ワールドカップのサッカーくじは５月３１日から売り始10めます。

Figure 2: Example of sentence alignments and a parallel article pair before and after being
simplified by a Japanese instructor (sentence ID of 0 indicates title)
Requirements of Automatic Sentence Alignment
We use an automatic sentence alignment method for constructing training data because it reduces cost and allows us to automatically use news data produced daily. We therefore examined
the requirements of automatic sentence alignment.
As explained in Section 2, news reporters mainly shorten news articles for conciseness,while Japanese instructors mainly simplify the expressions used within the text. However, theseroles are not clearly divided. News reporters sometimes simplify parts of expressions, while
Japanese instructors not only replace difficult words with simple words and divide sentences
into simpler syntactic structures, but they also sometimes omit detailed content, add supplemental explanations, or change the sentence order in an article. Figure 2 shows an example ofsentence alignments and a parallel article pair, which consist of articles before and after being
simplified by a Japanese instructor.4 In the example, the source sentence ID (which specifies asentence) of 3 is split into the two target sentence IDs of 3 and 4. The target sentence ID of 5
is an added sentence. The order of the source sentence ID of 4 was changed in the respective
target sentence.
We checked the number of sentences in each manually annotated sentence alignment. The
results are shown in Table 2. The rate of one source sentence corresponding to one and moretarget sentence(s) is high (over 0.91). However, there are some cases of sentence omission,sentence addition, or the inclusion of two source sentences. The average of Kendall’s τ , which
here represents the correlation between the order of sentences in a parallel article pair, was
0.987.5 Because the value is close to 1, we can confirm that the sentence orders were the same
in most cases.
From these results, we could conclude that the requirements of sentence alignment between
the source and target sentences are as follows. In addition to one sentence to one sentence (oneto-one) alignments, alignments including two or more sentences can also be created. Sentenceomission and sentence addition can be accounted for, and sentence order can also be considered.
4 In the example in Figure 2, many expressions in the sentences on the left were already simplified by a news reporter.
Reporters sometimes simplify expressions like these.
5 This value was calculated as follows. We removed target sentences that were not aligned with source sentences.
We then projected source sentence IDs to their aligned target sentences. Here, we define the projected order of the target
sentences as the order of sentences based on the projected IDs. When two target sentences had the same projected ID,
the projected order follows the original target sentence order. We calculated Kendall’s τ between the projected orderand the order of the target sentence order.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 21

Table 3: Number of sentences in each automatic sentence alignment
Number of sentences(source–target)
1–1
1–2
1–3
2–1
2–2
1–4
0–1
1–0
3–1

Frequency

Rate

7,893
2,336
383
111
44
39
32
19
5

0.727
0.215
0.035
0.010
0.004
0.004
0.003
0.002
0.000

Table 4: Precision and recall of automatic sentence alignment
Number of sentences(source–target)
Whole
One–One or more

Precision

Recall

0.872
0.881

0.885
0.942

Automatic Sentence Alignment for Training Data
Source sentences and target sentences include many identical words, and the sentence order isalmost identical. Therefore, we decided to use an alignment method using identical words anddynamic programming, which can efficiently estimate sentence alignments with considerationof sentence order so that sentence alignments do not include crossing alignments. We used
Champollion (Ma, 2006), which is an implementation of such a method, for automatic sentencealignment. This method can treat alignments including two or more source or target sentences,sentence omission, and sentence addition.6 The number of sentences in each automatic sentencealignment for 1,559 parallel article pairs for training data are shown in Table 3.
Quality Evaluation of Automatic Sentence Alignments
We evaluated the quality of automatic sentence alignments using the 490 parallel article pairsthat were manually annotated with sentence alignments. Precision and recall for all sentencealignments and for sentence alignments consisting of one source sentence and one or moretarget sentences are shown in Table 4. Here, the unit of sentence alignment is defined as one
parallel sentence pair (e.g., one-to-one, one-to-two, two-to-one, or one-to-null).
Automatic simplification based on monolingual translation has the following characteristics. When output sentences include errors, the quality of the output sentences decreases
compared with the input sentences. However, when input sentences are output without modifications, the quality of the output sentences does not decrease compared with the input sentences.
Therefore, it is important for simplification to remove as much noise as possible from the training data.
4.2

Data Set for Evaluating Automatic Simplification

Here we describe the specifications of the constructed data set for evaluating automatic simplification. We constructed the data set using news archives from April 2012 to September 2014.
We used news articles with edit histories showing simplification by Japanese instructors. In
6 Target sentences whose sentence order was changed from the source sentence order cannot be aligned perfectly.
However, if such target sentences are not aligned with any source sentences, then the extraction of parallel sentences of
low quality can be avoided.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 22

Table 5: Specifications of the data set
Term (Article number)

Training
Development
Test2012/04/02 (1) – 2014/02/26 (3)2014/02/26 (4) – 2014/04/24 (2)2014/04/24 (3) – 2014/09/30 (5)Number ofarticle pairs
1,559
170
485

Number ofsentencepairs
10,651723
2,012

Extraction
MethodAutomatic
ManualManualkeeping with practical use, we divided the data as follows: Data in the latest term were used as
the test data, data in the term immediately prior to the test data term were used as the development data, and data in the term immediately prior to the development data term were used as thetraining data. The specifics of the data are shown in Table 5. The test data consist of 2,012 manually extracted source sentences with their corresponding simplified reference sentences. The
development data consist of 723 parallel sentence pairs manually extracted in the same way
as the test data. The training data consist of 10,651 automatically extracted parallel sentencepairs. The training data also include 1,559 news articles with full versions of edit histories inthe training term.

5 Experiments on Sentence-Level Automatic Simplification
To confirm the effectiveness of the constructed data set, we conducted preliminary experimentson sentence-level automatic simplification using the data set. We conducted automatic Japanese
news simplification as a monolingual translation task using phrase-based SMT, as Coster andKauchak (2011) did.
5.1

Setup

We used MeCab7 for Japanese segmentation. Continuous Arabic numerals were merged to
one word. We used the Moses implementation (Koehn et al., 2007) as the phrase-based SMT
system. The translation model was trained using sentences that are 80 words or less. GIZA++
and grow-diag-final-and heuristics were used to obtain word alignments. To assist the wordalignments for low frequency words, we added pairs of the same word to the training data whenword alignments were estimated. This is because in the monolingual parallel sentences there
are many words that should be aligned to the same words. We used a 5-gram language modelthat was trained using the target side of the training data. The SMT weighting parameters weretuned via MERT (Och, 2003) using the development data. We used distortion limits of 0 or 6
(default value), which limit the number of words for word reordering to a maximum number.
We used the MSD bidirectional lexicalized reordering models of Moses (Koehn et al., 2005).
We compare the trained SMT system (MOSES) to the baseline that does not simplify input
sentences and outputs the input sentences without any change (BASELINE). We evaluate using
the test data of the data set.
5.2

Results and Discussion

We evaluated the simplification quality based on the automatic evaluation scores from the
BLEU-4 (Papineni et al., 2002) and RIBES v1.02.4 (Isozaki et al., 2010), which are commonlyused for evaluating translation quality. Percentage scores were used for these scores. RIBES
is an automatic evaluation measure based on word-order correlation coefficients between reference sentences and output sentences. Evaluation results are shown in Table 6．Bold numbers
indicate that values are significantly higher than the result of BASELINE in each evaluation measure. To assess this, we used the bootstrap resampling test at a significance level of α = 0.01
7 http://taku910.github.io/mecab/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 23

Table 6: Results of sentence-level automatic simplification
SystemBASELINE
MOSES (distortion limit 0)
MOSES (distortion limit 6)

BLEU
41.62
46.06
46.02

RIBES
84.04
85.41
85.39

(Koehn, 2004).
MOSES (distortion limit 0), which learned simplification from the training data, obtained
a BLEU score that is 4.4 points higher than that of BASELINE. This confirms the effectivenessof the data set for automatic simplification. MOSES (distortion limit 0) also obtained a RIBES
score that is 1.3 points higher than that of BASELINE. In an experiment on English simplification using Wikipedia (Coster and Kauchak, 2011), the improvement of the BLEU score was 0.5
points. The improvement of the BLEU score using our data set is higher than theirs, indicatingthat our data set had a larger effect than their English Wikipedia data set. One of the reasons forthis larger effect is thought to be that more words in simplified Japanese news were rewritten
than those in the Simple English Wikipedia data set because the BLEU score (41.62) of Japanese
news BASELINE is lower than the BLEU score (59.37) of English Wikipedia BASELINE. The
training data size is smaller than for parallel corpora often used in SMT experiments on translation between languages. When the training data are small, certain methods can improve thetranslation quality (Xiang et al., 2010; Irvine, 2013; Irvine and Callison-Burch, 2014). Suchmethods will be useful for our Japanese news simplification task.
When MOSES (distortion limit 6), which allows phrase reordering, is compared with
MOSES (distortion limit 0), which does not allow phrase reordering, the automatic scores did
not improve. However, this result does not ensure that phrase reordering is unnecessary because
there is room for improvement of the RIBES scores. We cannot know what types of word reordering are needed based solely on these results. Therefore, in the next section we investigate
what types of rewrites are needed for simplification.

6

Analysis of Manually Simplified News Sentences

To reveal what types of rewrites are needed for automatic Japanese news simplification, we
analyzed what types of rewrites, such as those associated with word order (i.e., syntactic structure), were conducted by Japanese instructors in the manual simplification processes. For thisanalysis, we produced manually annotated word alignments for 50 parallel article pairs. The
50 article pairs include 309 source sentences and 530 target sentences. As explained in Section
4.1, news reporters sometimes simplify expressions, despite it not being their main role. Thus,
when analyzing the simplification of expressions by comparing sentences before and after simplification by Japanese instructors, if we use sentences that were rewritten by news reporters as
pre-simplification sentences, then parts of the simplification process may go undetected. Thisis because it is possible that a certain degree of expression simplification has already been conducted before being simplified by the Japanese instructors. Therefore, to exhaustively detect
the simplification of expressions, we only used simplified news articles that Japanese instructors had rewritten prior to being rewritten by news reporters.8 Here, articles prior to being
rewritten by Japanese instructors are called ORG, and articles that have been rewritten by such
instructors are called EASY.
8 As explained in footnote 3, recent articles were first rewritten mostly by news reporters and Japanese instructors

first rewrote around half of the articles produced in the early phase. We wanted to conduct this analysis independentlyof the construction of the data set because we can then start analyzing when small data become available. For thesereasons, we selected the 50 article pairs from the data produced in the early phase; that is, the 50 pairs are not a subsetof the manually sentence-aligned article pairs in the data set described in Section 4.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 24

Table 7: Rewrite categories associated with word order
Category
Changing adnominal clauses to sentences
Reordering case elements
Reversed relation of modification
Complement for sentence splitting
Changing case of case elements
Indicating relation of continuous nouns
Changing compound nouns
Changing part of speech
Adnominal clause modifying formal nouns
Verbalizing nouns
Changing quantity expressions
Extraction of difficult expressions
Moving from EOS to BOS
Changing clause to noun modifier
OthersFrequency
39
21
13
13
10
8
7
7
5
4
3
3
3
2
66

Reordering distance
type
globalgloballocal
globalglobal
local
local
local
globalglobalglobalglobalgloballocal
global/local

We then analyzed the simplifications associated with word order along with those not associated with word order.
6.1 Simplification Associated with Word OrderWe categorized the simplification with respect to its association with word order for the sentences in the 50 article pairs. We disregarded expressions that were too largely summarized or
too largely reorganized. Rewrite categories associated with word order are shown in Table 7.
For the item of reordering distance type, “local” represents word reordering in a phrase pair or
the reordering of contiguous phrases, and “global” represents longer word reordering comparedwith local, or the duplication of words. Explanations and examples of each category are shownin Appendix A. These results indicate that the most frequent rewrite type with word reordering is the extraction of adnominal clauses to become independent sentences. When adnominal
clauses are extracted to become independent sentences, the syntactic structures of the resulting
sentences become simpler. Thus, the effect of the extraction on readability is thought to belarge. The second most frequent type, reordering case elements, does not result in a reductionin the complexity of syntactic structure. Thus, the effect of extraction of adnominal clauses onreadability is thought to be larger than that of reordering case elements. Rewrites with local
reordering also do not reduce the complexity of syntactic structures in many cases. From theseanalyses, the extraction of adnominal clauses to become independent sentences was found to be
the most frequent and effective type of simplification with word reordering.
Although it is possible for phrase-based SMT to perform rewrites with local word reordering, converting adnominal clauses into independent sentences is difficult for phrase-based SMT
because it requires long-distance word reordering and the duplication of the words modified by
the adnominal clauses. Therefore, we believe that the following two-step conversion is suitable for use in automatic simplification. First, conversion with long-distance word reordering
or word duplication, such as extracting adnominal clauses to become independent sentences,
is conducted using syntactic structures and conversion rules. Second, sentences are simplified
using phrase-based SMT.9
9 The idea of splitting the process into two steps is the same as that of pre-ordering in SMT (Xia and McCord, 2004).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 25

Table 8: Rate of tokens (words) in EASY that were unchanged, changed, or added
Rate
0.56
0.37
0.07

Unchanged (identical) tokens
Changed (different) tokens
Added tokens

Table 9: Changing rates for passive voice expressions and causative expressions
Passive voice to active voice
Causative to non-causative

Change rate (Frequency)
0.92 (139/151)
1.0 (11/11)

Table 10: Number of sentences produced from one sentence in simplification
Number of sentences(source–target)
1–1
1–2
1–3
1–4 or more

6.2

Rate
0.40
0.46
0.11
0.02

Simplification Not Associated with Word Order

We also analyzed the word rewrite rates, the rewrite rates of passive voice and causative expressions, and sentence splitting.
We checked the rates of tokens (words) in EASY that were unchanged, changed, or added
by Japanese instructors.10 The results are shown in Table 8. The results indicate that 44% of
the tokens in EASY were either changed or added. We also checked the rate of omitted tokens
in ORG. It was 0.08.11,12
We checked the rate of passive voice expressions that were changed into active voice expressions, as well as the rate of causative expressions that were changed into non-causative
expressions.13,14 The results are shown in Table 9. It was found that most of the passive voice
expressions were changed into active voice expressions, and that most of the causative expressions were changed into non-causative expressions.
The number of sentences produced from a single sentence during simplification are shownin Table 10. Approximately half of the sentences were split into two separate sentences. The
most frequent cause of sentence splitting was the changing of continuous clauses into independent sentences without word reordering, and the second most frequent was the changing of theabove-mentioned adnominal clauses into independent sentences. The rate of sentence splitting
is larger than the rate shown in Table 2. The main reason for this result is thought to be the factthat the pre-simplification sentences were different. The current analysis used original sentences
as the pre-simplification sentences, whereas the analysis of Table 2 mainly used sentences that
10 We used MeCab with an IPA dictionary as the morphological analyzer.
11 Expressions that do not contain main content may be dropped. [E.g. 20] in Appendix A is such an example.
12 The lead sentence, which is the first sentence in a news article that describes a summary of the news, may be
dropped because its content overlaps with the text of the main body. To check the rate of dropped words under thecondition in which the reason for deletion was not overlapping, we checked the rate of omitted tokens using all sentences
except the lead sentences.
13 We only checked causative expressions in which the base forms were seru or saseru and the part of speech wasverb-postfix.
14 [E.g. 6] in Appendix A is an example of passive voice being changed into active voice. [E.g. 1] in Appendix A isan example of causative being changed into non-causative.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 26

had already been rewritten to some degree by news reporters as the pre-simplification sentences.

7 Conclusion
We designed a Japanese news sentence-level simplification task and constructed a Japanese
news simplification data set, which consisted of Japanese news sentences and their corresponding simplified sentences. This is, to the best of our knowledge, the first time a parallel corpus
consisting of sentences sourced from Japanese news and their simplified Japanese counterparts
has been constructed. We verified the effectiveness of the constructed data set through preliminary experiments on automatic simplification using phrase-based SMT, and we confirmed that
our result was 4.4 BLEU points higher than that of the baseline, which does not change theinput sentences. Additionally, we produced manually annotated word alignments between parallel sentences to analyze the human operation of simplifying expressions, and provided whattypes of rewrites, such as rewrites associated with word order, were conducted in the manual
simplification processes. The constructed data set and the knowledge obtained by our analysis
of manual simplification will be useful for future research on Japanese news simplification andwill serve as assistance in the production of simplified Japanese news.

References
Aluı́sio, S. M., Specia, L., Pardo, T. A., Maziero, E. G., and Fortes, R. P. (2008). Towards Brazilian
Portuguese automatic text simplification systems. In Proceedings of the Eighth ACM Symposium on
Document Engineering, DocEng ’08, pages 240–248.
Bott, S., Rello, L., Drndarevic, B., and Saggion, H. (2012). Can Spanish be simpler? LexSiS: Lexical
simplification for Spanish. In Proceedings of COLING 2012, pages 357–374.
Bott, S. and Saggion, H. (2011). An unsupervised alignment algorithm for text simplification corpus
construction. In Proceedings of the Workshop on Monolingual Text-To-Text Generation, pages 20–26.
Carroll, J., Minnen, G., Canning, Y., Devlin, S., and Tait, J. (1998). Practical simplification of english
newspaper text to assist aphasic readers. In Proceedings of AAAI-98 Workshop on Integrating Artificial
Intelligence and Assistive Technology.
Chandrasekar, R., Doran, C., and Srinivas, B. (1996). Motivations and methods for text simplification. In
Proceedings of the 16th International Conference on Computational Linguistics, pages 1041–1044.
Coster, W. and Kauchak, D. (2011). Simple English Wikipedia: A new text simplification task. In
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human
Language Technologies, pages 665–669.
Dell’Orletta, F., Montemagni, S., and Venturi, G. (2011). Read–it: Assessing readability of italian texts
with a view to text simplification. In Proceedings of the Second Workshop on Speech and Language
Processing for Assistive Technologies, pages 73–83.
Inui, K., Fujita, A., Takahashi, T., Iida, R., and Iwakura, T. (2003). Text simplification for reading assistance: A project note. In Proceedings of the Second International Workshop on Paraphrasing, pages
9–16.
Irvine, A. (2013). Statistical machine translation in low resource settings. In Proceedings of the 2013
NAACL HLT Student Research Workshop, pages 54–61.
Irvine, A. and Callison-Burch, C. (2014). Hallucinating phrase translations for low resource MT. In
Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 160–
170.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 27

Isozaki, H., Hirao, T., Duh, K., Sudoh, K., and Tsukada, H. (2010). Automatic evaluation of translation
quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in
Natural Language Processing, pages 944–952.
Kauchak, D. (2013). Improving text simplification language modeling using unsimplified text data. In
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1537–
1546.
Klaper, D., Ebling, S., and Volk, M. (2013). Building a German/simple German parallel corpus for automatic text simplification. In Proceedings of the Second Workshop on Predicting and Improving Text
Readability for Target Reader Populations, pages 11–19.
Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Lin, D. and Wu, D.,editors, Proceedings of EMNLP 2004, pages 388–395.
Koehn, P., Axelrod, A., Mayne, A. B., Callison-Burch, C., Osborne, M., and Talbot, D. (2005). Edinburgh system description for the 2005 IWSLT speech translation evaluation. In Proceedings of theInternational Workshop on Spoken Language Translation.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source
toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association
for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages
177–180.
Ma, X. (2006). Champollion: A robust parallel text sentence aligner. In Proceedings of the Fifth International Conference on Language Resources and Evaluation, pages 489–492.
Moku, M. and Yamamoto, H. (2012). Automatic easy Japanese translation for information accessibility offoreigners. In Proceedings of the Workshop on Speech and Language Processing Tools in Education,
pages 85–90.
Narayan, S. and Gardent, C. (2014). Hybrid simplification using deep semantics and machine translation.
In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages
435–445.
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the
41st Annual Meeting of the Association for Computational Linguistics, pages 160–167.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluationof machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational
Linguistics, pages 311–318.
Petersen, S. E. and Ostendorf, M. (2007). Text simplification for language learners: a corpus analysis. In
Proceedings of Workshop on Speech and Language Technology for Education.
Seretan, V. (2012). Acquisition of syntactic simplification rules for French. In Proceedings of the EighthInternational Conference on Language Resources and Evaluation (LREC-2012), pages 4019–4026.
Tanaka, H., Mino, H., Kumano, T., Ochi, S., and Shibata, M. (2013). News service in simplified Japaneseand its production support systems. In Proceedings of the IBC2013 Conference.
Woodsend, K. and Lapata, M. (2011). Learning to simplify sentences with quasi-synchronous grammar
and integer programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural
Language Processing, pages 409–420.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 28

Wubben, S., van den Bosch, A., and Krahmer, E. (2012). Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational
Linguistics, pages 1015–1024.
Xia, F. and McCord, M. (2004). Improving a statistical MT system with automatically learned rewrite
patterns. In Proceedings of Coling 2004, pages 508–514.
Xiang, B., Deng, Y., and Zhou, B. (2010). Diversify and combine: Improving word alignment for machinetranslation on low-resource languages. In Proceedings of the ACL 2010 Conference Short Papers, pages
22–26.
Zhu, Z., Bernhard, D., and Gurevych, I. (2010). A monolingual tree-based translation model for sentencesimplification. In Proceedings of the 23rd International Conference on Computational Linguistics,
pages 1353–1361.

Appendix A. Explanations and Examples of Each Category in Table 7
Here we explain the categories associated with word order in Table 7 and show their examples.
Red and blue are used for the parts reordered based on the factor of each category.
Changing adnominal clauses to sentences This refers to cases in which adnominal clauses
are extracted as independent sentences.
[E.g. 1] NASA は去年１１月に打ち上げた火星探査機「キュリオシティー」を日本時間の来月６日、午後２時
半過ぎに火星に着陸させます。
↓
NASA は去年１１月に「キュリオシティー」という火星探査機を打ち上げました。
「キュリオシティー」は、日本時間の来月６日、午後２時半すぎに、火星に着きます。
[E.g. 2] この遺伝子をマウスの脳の記憶などをつかさどる海馬という部分に大量に組み込みました。
↓
その遺伝子をマウスの脳の中の海馬という部分にたくさん入れました。
海馬は記憶などをコントロールする働きがあります。

Reordering case elements This refers to cases in which the order of the case elements ischanged.
[E.g. 3] 富士山が大規模に噴火した場合、山梨県は・・・・・→ 山梨県は、富士山が大規模に噴火した場合・Reversed relation of modification This refers to cases in which the word reordering is causedby changing words into other words with reversed relations of modification.
[E.g. 4] 半年余りで → 約半年でComplement for sentence splitting This refers to cases in which the case elements are replicated to complement split sentences when continuous clauses are changed into independent
sentences.
[E.g. 5] この有料サービスは、・・・の現在地を地図で把握できるというもので、１０日から運用が始まりました。
↓
この有料サービスは、・・・が今いる場所を地図で知ることができるというものです。
このサービスは１０日から始まりました。

Changing case of case elements This refers to cases in which the word reordering is causedby changing the case of the case elements.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 29

[E.g. 6] 笑った顔に見える埴輪が、パリで来月開かれる展覧会で展示されることになり、↓パリで来月開かれる展覧会に笑った顔に見える埴輪を出すことになりました。
[E.g. 7] 難しい役柄を表現力豊かに演じ → すばらしい表現力で難しい役を演じてIndicating relation of continuous nouns This refers to cases in which the word reordering is
caused by indicating the relations of continuous nouns.
[E.g. 8] 火星探査機「キュリオシティー」 → 「キュリオシティー」という火星探査機

Changing compound nouns This refers to cases in which the word reordering is caused bychanging compound nouns.
[E.g. 9] 家庭の電力消費 → 家庭で使う電力[E.g. 10] 新たな被害想定を → 被害について、新しい予想をChanging part of speech This refers to cases in which the word reordering is caused by changing the part of speech of words and their modifying points.
[E.g. 11] 具体的な場所を尋ね → 場所を細かく聞くと

Adnominal clause modifying formal nouns This refers to cases in which the verbs in adnominal clauses modifying formal nouns move backward.
[E.g. 12] 研究を行ったのは、・・・のグループです。
→ この研究は・・・のグループが行いました。

Verbalizing nouns This refers to cases in which the word reordering is caused by verbalizing
nouns (sahen nouns).
[E.g. 13] 着陸まであと２０日余り → 約２０日で着きます。

Changing quantity expressions This refers to cases in which the word reordering is causedby changing quantity expressions.
[E.g. 14] 避難が必要な人数の試算を進め → どのくらいの人が避難する必要があるか計算を進め

Extraction of difficult expressions This refers to cases in which difficult expressions are extracted and explained as independent sentences.
[E.g. 17] と述べ、異例の謝罪を行いましたが、→ と言って謝りました。このようなことは今までにありませんでした。

Moving from EOS to BOS This refers to cases in which parts of expressions at the end ofsentences move to the beginning of the sentence.
[E.g. 15] ・・・に提供することを明らかにし、 → 発表によると、・・・に渡したりしました。
[E.g. 16] ・・・を考えているという調査結果がまとまりました。
→ その結果、・・・を考えてみようとしていることが分かりました。

Changing clause to noun modifier This refers to cases in which the word reordering is causedby changing clauses to noun modifiers.
[E.g. 18] １１年８か月ぶりの円高水準を更新しました。
→ ２０００年１１月から今まででいちばんの円高です。

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 30

Others This refers to cases for which categorization is difficult because rewrites are complex
or they require background information, or when the frequency of the category is one.
[E.g. 19] IT 企業の間では、この分野を強化する動きが広がっています。
→ 地図に力を入れる IT 企業が増えています。

[E.g. 20] 電力会社が提供する需給状況のデータに基づいて、電力需要が少なく価格が安い時間帯に→ 電力が足りている時間は、電力会社のデータから分かります。

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 31

Learning Bilingual Distributed Phrase
Representations for Statistical Machine Translation
Chaochao Wang
Deyi Xiong ∗Min Zhangchaochaowang@foxmail.com
dyxiong@suda.edu.cn
minzhang@suda.edu.cn

Soochow University, Suzhou, China

Chunyu KIT

ctckit@cityu.edu.hk

Department of Linguistics and Translation, City University of Hong Kong

Abstract
Following the idea of using distributed semantic representations to facilitate the computationof semantic similarity between translation equivalents, we propose a novel framework to learnbilingual distributed phrase representations for machine translation. We first induce vector
representations for words in the source and target language respectively, in their own semantic
space. These word vectors are then used to create phrase representations via composition methods. In order to compute semantic similarity of phrase pairs in the same semantic space, we
project phrase representations from the source-side semantic space onto the target-side semantic space via a neural network that is able to conduct nonlinear transformation between the two
spaces. We integrate the learned bilingual distributed phrase representations into a hierarchical phrase-based translation system to validate the effectiveness of our proposed framework.
Experiment results show that our method is able to significantly improve translation quality
and outperform previous methods that only use word representations or linear semantic spacetransformation.

1

Introduction

Distributional semantic models provide a means to represent meanings of words under theassumption that words occurring in similar contexts tend to have the same meanings (Harris,1968). Various distributed representations have been successfully applied to various monolingual natural language processing tasks, such as word sense discrimination (Clark and Pulman,2007) and thesaurus compilation (Yang and Powers,2008). In this paper, we explore how
to learn semantic representations of bilingual phrases, rather than monolingual words, in thecontext of statistical machine translation (SMT) to facilitate the computation of semantic similarity between translation equivalents at the phrase level. We also study whether semantic
similarity scores calculated in terms of bilingual distributed phrase representations are complementary to phrase translation probabilities estimated by the conventional counting method in
SMT Koehn et al. (2003).
Very recently we have witnessed some studies on learning bilingual distributed representations for SMT. Mikolov et al. (2013b) train two neural network (NN) based models to learn
word embeddings in the source and the target language, respectively, and then map the embeddings from the source to the target language space using a transformation matrix that is learned∗Corresponding author

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 32

by minimizing the mapping cost on all word pairs. Zou et al. (2013) introduce bilingual wordembeddings into phrase-based machine translation: Word representations are first learned fromlanguage A via an NN-based model, word embeddings in the parallel language B are then initialized according to A’s embeddings and word alignments between A and B, and the final word
representations of B are obtained by a further training process that optimizes a combined objective on bilingual data. Gao et al. (2013) extend distributional representations from the wordlevel to the phrase level, adopting a fully connected neural network to transfer bag-of-words
vector representations of raw phrases (in the source or the target language) to distributional
representations in a language-independent low-dimensional semantic space and having the parameters of the neural network jointly learned with the feature weights of the log-linear modelof phrase-based SMT.
Partially inspired by these previous studies, we propose a new framework to learn bilingual
distributed phrase representations for SMT via semantic composition and bilingual projection.
Our current work differs from the previous ones distinctively in several aspects.
• We learn bilingual phrase representations, instead of representations at the word level
Mikolov et al. (2013b); Zou et al. (2013), so as to keep consistency with the SMT thatuses phrases rather than words as basic translation units.
• We learn phrase representations from distributed word representations via semantic composition, instead of from raw phrases Gao et al. (2013) in order to avoid the data sparsenessissue of directly learning phrase representations from data. Particularly, we empiricallycompare two different composition methods in our framework, namely, weighted vector
addition (Mitchell and Lapata,2008) and recursive autoencoder Socher et al. (2011).
• Rather than jointly learning phrase representations with feature weights of the log-linear
model of SMT Gao et al. (2013), we take a loose coupling strategy to simplify the learning
process. We adopt a neural network to project phrase representations from the source onto
the target language semantic space, in separation from the process of feature weight tuningin SMT.
• Rather than capturing only the linear transformation between the source and target language semantic space Mikolov et al. (2013b), our neural network for the bilingual projection can model both linear and nonlinear transformation between these two semantic
spaces. We hope that the nonlinear transformation can better model the bilingual projection.
We integrate the learned bilingual distributed phrase representations into a hierarchicalphrase-based SMT system (Chiang,2007) by calculating semantic similarity scores of bilingualphrases in terms of their representations. We also empirically compare combinations of learning methods for word representations, and phrase composition methods as well as bilingual
projection strategies. Experiments on Chinese-to-English translation show that our best results
outperform the baseline by 0.53 BLEU points, indicating the effectiveness of our approach.
The rest of the paper is organized as follows. Section 2 describes how we obtain word
representations for the source and target language and vector representations of phrases via two
different composition methods. Section 3 presents a nonlinear neural network to learn bilingualphrase representations by projecting source language phrase vectors onto the target language
space and Section 4 introduces the integration of bilingual representations into SMT. Section 5
elaborates our large-scale experiments on Chinese-to-English translation and analyzes experimental results. Section 6 discusses related work in relation to ours and Section 7 concludes thepaper with future directions of research.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 33

2

Distributed Phrase Representation Acquisition via Semantic Composition

In this section, we introduce semantics-based vector representations of words and vector representations of phrases via two different composition methods.
2.1

Word Representations

Vectors of words are basic elements in our bilingual phrase representation learning framework.
We employ two different models, namely a point-wise mutual information (PMI) based vector
space model (Pado and Lapata,2007) and a neural language model Mikolov et al. (2013a), to
derive word vectors.
PMI-based vector space word representations Vector space model provides an elegant
way to represent the meaning of a word: each element in its vector denotes a degree that measures how frequently it co-occurs in a predefined context window with every other word in the
vocabulary in question. A well-known measure for this is PMI, which estimates the strength of
the relationship between a context word c and a target word t as follows:pmi(c, t) = log

p(c, t)p(c)p(t)(1)

In order to get around certain unavoidable frequency bias, we use positive point-wise mutual
information (PPMI) (Turney and Pantel,2010) to calculate the elements in a word. It is definedas:
ppmi(c, t) =pmi(c, t)0ifpmi(c, t) > 0
otherwise(2)

Neural word representations Mikolov et al.(2013a) introduce an efficient neural language
model to learn high-quality word embeddings from extremely large amounts of raw texts. We
adopt their approach for learning word embeddings. After training the neural language model,
we can obtain a word embedding matrix M ∈ Rn×|V | , where each word in the vocabulary V
corresponds to a vector v ∈ Rn with n to denote vector size. Given this, the vector representation of the word assigned with index i in V can be retrieved simply by extracting the ith columnof M .
2.2

Composition Methods

Once having obtained vector representations for words, we can use them to construct those forphrases via various composition methods as phrases are composed of words. We explore twocomposition methods: one based on simple vector addition (Mitchell and Lapata,2008) and theother on a recursive autoencoder that takes the inner structure of a phrase into account Socher
et al. (2011).
Weighted vector addition Given a phrase p that consists of two words w1 and w2 , we−
→1 and −→2 by the following weighted vector addition:
obtain the vector →p from its word vectors −
w
w
→
−
−
→1 + β w−→2p = αw(3)

where α and β are weights denoting the relative importance of each word in the composition.
For a phrase with multiple words p = (w1 ,w2 ,...,wn ), we can use in a similar way to obtain thevector for p by summing over vectors of all its words,
→
−
p =

n
X

→i
λi −
w

(4)

i=1

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 34

p2 = f (W[c1; p1] + b)

p1 = f (W[c2;c3 ] + b)

c1

c3

c2

Figure 1: The architecture of a recursive autoencoder, where the nodes with black dots are input
word (or phrase) vectors and the nodes with circles are reconstructed vectors for computing
reconstruction errors.
W1

W2

Figure 2: The architecture of bilingual projection neural network that projects vector representations of source phrases to the semantic space of target language.
Although weighted vector addition is a simple way for composition, it has proven effective
in many tasks (Kartsaklis,2014). In our task, however, it cannot model word positions in aphrase. Therefore we only use it as a baseline to compare against a more advanced method:
recursive autoencoder.
Recursive autoencoder (RAE) RAE is a neural network that can learn representations forlarge linguistic expressions such as phrases and sentences in a bottom-up fashion along a tree
structure. Normally, word vectors learned via a distributional method can be input as leaf nodesof RAE. Figure 1 presents an illustration to visualize the architecture of RAE. Given a binarybranch p → c1 c2 where a child node is either a leaf or nonterminal node, the representation of
p can be calculated as:→−−−
p = f (W [→c1 ; →
c2 ] + b)(5)−−
where [→
c1 ; →
c2 ] denotes the combination of the two child vectors, W and b are model parameters,
and f is an element-wise activation function such as sigmoid. This will be used to further compute representations for larger structures. In order to judge how appropriately a parent vector
computed this way can represent its children, we can reconstruct the children in a reconstruction
layer as:−−−[→c1 0 ; →c2 0 ] = W 0 →p + b0(6)

For each nonterminal node, we compute the Euclidean distance between its original child vectors and the reconstructed vectors as the reconstruction error of the node according to the following equation:
1 →
2
−
−
−
−
−
Erec ([→
c1 ; →c2 ]) =[−
c1 ; →
c2 ] − [→c1 0 ; →c2 0 ](7)
2
The parameters of an RAE can be learned by minimizing the reconstruction error over the entire
tree.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 35

In this paper, we adopt a greedy unsupervised RAE that is proposed in Socher et al. (2011)as an extension to the standard RAE described above. The unsupervised RAE can learn not
only the representation of a phrase or sentence but also their tree structures in a greedy manner.

3

Learning Bilingual Phrase Representations

We use the methods introduced above to obtain phrase representations for the source and target
language respectively, and adopt a nonlinear bilingual projection neural network to project thephrase representations in the source language onto the semantic vector space of the target language so as to calculate similarity scores of bilingual phrases in the same semantic space. The
general architecture for this work is presented in Figure 2.
The adopted neural network for projection is a fully connected neural network with only
one hidden layer. The projection can be formulated in the following equation:→−−p = sigmoid(W2 (sigmoid(W1 →x + b1 )) + b2 )(8)

where W1 is the projection matrix from the input layer to the hidden layer, W2 is the projection
matrix from the hidden layer to the output layer, b1 and b2 are bias terms. In order to calculate
the weights of the network, we need to calculate the squared error function as follows:
e=

1X m
2
(t − pm
i )
2 i i

(9)

m
where pm
i is the vector calculated by the neural network according to the Eq.(8) and ti the real
vector of the corresponding target phrase. The weights can be trained via backpropagation by
→
−
→
−
−
−
minimizing the error on the set of collected training instances {(→s , t )}n1 where →s and t arevectors of the source and target side of a phrase pair (s, t). If we do not use any hidden layer inthe projection neural network, the degenerated neural network will exactly learn Mikolov et al.
(2013a)’s linear transformation matrix. Adding a hidden layer with nonlinear activation functions, we enable our projection neural network to model the nonlinear transformation between
the semantic spaces of the source and target language. We will empirically compare the nonlinear against the linear projection in Section 5.Once the projection neural network is trained,
we can learn projected representations of source phrases in the target semantic space using thisneural network.

4

Integrating Bilingual Representations into SMT

A straightforward way to integrate bilingual phrase representations into SMT is to calculatesemantic similarity between representations of this kind for translation equivalents. Given a→−−phrase pair (s, t), let (→s , t ) denote their vector representations on the source and the target→−−
language semantic space and (p(→s ), t ) the learned bilingual distributed phrase representa→−
tions, where p( s ) is the projected vector representation of source phrase s obtained by our
projection neural network as presented above. The semantic similarity between s and t can thenbe calculated as follows:−p(→s ) • ~t−Sim(p(→s ), ~t) =(10)→−kp( s )k × ~t
Given a source sentence c, we can build a new semantic similarity model based on ourlearned bilingual phrase representations according to the following equation:
X→−−
MSim =Sim(p(→s ), t )(11)(s,t)∈P

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 36

SystemBaseline
PPMI
NeuralNIST 06
30.23
30.37
30.46

NIST 08
23.21
23.36
23.40

Table 1: Results of integrating word representations acquired by the two methods integrated
into hierarchical phrase-based SMT
where P denotes all possible phrase pairs that are in use to translate the source sentence c andhave distributed vector representations. The semantic similarity can be used as a feature in a loglinear model and can also be integrated into any SMT system that uses bilingual phrase pairs
during decoding. In this paper, we integrate this new model into a hierarchical phrase-based
SMT system without loss of generality.
Rules in the hierarchical phrase-based SMT can be classified into two types: 1) phrase
rules that only contain terminals and 2) non-terminal rules with at least one non-terminal. For
phrase rules, the similarity score can be easily calculated according to Eq. (10) in a preprocessing step. As for non-terminal rules, we compute their similarity scores in two steps. Let us
take a specific non-terminal rule X → < X1 > ÔX X2 >L >,X1 the election committee X2 hold election,0-0 1-2 2-3 3-4 4-5 5-6 > as an example to show how we compute
their similarity. First, we can find phrase pairs (> ÔX, the election committee )and ( >L > , hold election ) via word alignments. The similarity values of thesetwo phrase pairs are estimated according to Eq. (10). In order to ensure decoding speed, thesemantic similarities of these phrases are also calculated in a preprocessing step so that they canbe quickly retrieved during decoding. Second, we sum up all similarity values, including thesimilarity values of phrases within nonterminals X1 and X2 , according to Eq. (11) by means ofdynamic programming.

5

Experiments

We have carried out a number of experiments on Chinese-to-English translation to validate theeffectiveness of the proposed framework for learning bilingual phrase representations. Various
combinations of different word representation models, composition methods, and projection
strategies presented above are tested. Particularly, we intend to explore answers to the following
questions:• Which word representation is better, PMI-based vector space representation or neural representation?
• Would phrase representations be better than word representations when used to calculatesemantic similarity scores? Furthermore, would RAE provide more efficient phrase representations than weighted vector addition?
• Is it necessary to project phrase representations in a non-linear fashion?
5.1

Experiment Setup

Our baseline system is hierarchical phrase-based system (Chiang,2007), where translation candidates are scored by a set of features. Our training data consists of 4.1M sentence pairs with
98.9M Chinese words and 112.6M English words from LDC corpora, including LDC 2003E07,
LDC 2003E14, LDC 2004E12, LDC 2004T07, LDC 2005T06 and LDC 2005T10. We use the
NIST evaluation set of 2005 (NIST 05) as development set, and sets of NIST 06/NIST 08 as ourtest sets.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 37

SystemBaseline
Word Representations
weighted vector addition
Phrase Representations
RAE

NIST 06
30.23
30.46
30.59
30.76∗NIST 08
23.21
23.40
23.45+
23.51+

Table 2: Results of integrating word and phrase representations into hierarchical phrase-based
SMT with “+” and “*” to mark the statistically significant performance improvement over thebaseline with p < 0.05 and p < 0.01
SystemBaseline
Linear Projection
Nonlinear Projection

NIST 06
30.23
30.48
30.76∗NIST 08
23.21
23.42+
23.51+

Table 3: Comparison of linear and nonlinear projection, with “+” and “*” to mark the statistically significant performance improvement over the baseline with p < 0.05 and p < 0.01.

Word alignments of training data were obtained by running GIZA++ (Och,2003b) in both
directions of our bilingual language source and applying refinement rule grow-diag-final-and
Koehn et al. (2003). A 4-gram language model was trained on the Xinhua section of Gigaword
by SRILM toolkit Stolcke et al. (2002). We also extracted SCFG rules from the word-aligned
training data. The translation performance was measured by case-insensitive BLEU Papineni
et al. (2002). We used minimum error rate training (MERT) (Och.2003a) to tune the log-linear
feature weights. As MERT is normally instable, we ran the tuning process three times for all
our experiments and presented the average BLEU scores on the three MERT runs as suggested
by Clark et al (2011).
The open source toolkit DISSECT1 was applied to obtain PMI-based vector space word
representations with a context window of 5 words, and Word2Vec2 to acquire neural word representations, with each word represented as a 50-dimensional vector. When adopted Word2Vec,
we just set the context window of size 5 and using continuous bag-of-words model. DISSECT
was also adpoted to train weights in semantic composition of weighted vector addition. Unsupervised greedy RAE was trained in the way following Socher et al. (2011). In the bilingual
projection neural network, 50 hidden units were used in the hidden layer.
5.2

PMI-Based Word Representations vs. Neural Word Representations

Our first series of experiments were carried out to compare PMI-based vector space word representations obtained by DISSECT against neural word representations obtained by Word2Vec.
Note that we did not perform semantic composition in this series of experiments as we focus
on word representations. Source word vector representations were projected onto the target
semantic space via the projection neural network described in Section 3. Experimental results
are presented in Table 1, from which we find both PMI-based vector space and neural word
representations can improve translation quality in terms of BLEU. Since vector space representations obtained by neural word representations are better than PMI-based word representations,
we used neural word representations in experiments hereafter.
1 http://clic.cimec.unitn.it/composes/toolkit/index.html
2 https://code.google.com/p/word2vec/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 38

Source
Reference
Example 1

Baseline
NWR
PRR
SourceReferenceBaseline

Example 2
NWR
PRR
SourceReference
Example 3Baseline
NWR
PRRpÞ i÷@ 1 [‰b P÷ \ ú ã Êê à ¨E Ç'
Lanzhou Price Bureau gives explanation of price controls on
beef noodles : It is only because the raises have been too large .
Lanzhou explained beef noodles reduce : only because of theexcessive increase .
Lanzhou explained that beef noodles reduce only because of excessive price .
Lanzhou gives explanation of beef noodles reduce : only because of the excessive raises .
Ø 6e / ¸ º Î‹ ö å |L „ Í• Ÿà K
High wages are one of the major reasons for many people to getsecond jobs as hourly workers .
High income many people engage in hourly workers outside oneof the major causes .
High income is one major cause many people engage in hourly
workers outside .
High wages are one important reason many people engaged inhours for part-time workers .
½A ÅÒ *ù - ( '2 ú°ºÃöö
Panic strikes as signs of bird flu virus continue to emerge inEurope .
Traces of avian flu virus keeps on Europe, causing panic .
Traces of avian flu virus continued there, causing panic in Europe .
Traces of avian flu continue to emerge, causing panic in Europe.

Table 4: Translation examples to illustrate the advantage of RAE composition based phrase
representations over others. NWR = neural word representations. PPR = phrase representations
with RAE composition.
5.3

Comparison of Different Composition Methods

The second series of experiments were aimed at investigating whether we should learn semantic
representations for phrases and at examining which composition method, i.e., either weighted
vector addition or recursive autoencoder, is a better approach to create phrase vector representations from word representations. Table 2 presents the experimental results, from which we candraw the following observations:• Integrating bilingual distributed phrase representations leads to a substantial improvement
up to 0.53 BLEU points over the baseline.
• Phrase representations gave a better performance than word representations by up to 0.3
BLEU points.
• The RAE composition outperforms the weighted vector addition by up to 0.17 BLEU
points.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 39

5.4

Nonlinear vs. Linear Projection

As mentioned in Section 3, a linear variation of the bilingual projection can be derived byremoving the hidden layer. In the third series of experiments, we investigated whether linear
transformation is sufficient to project source language representations onto the target language
semantic space. Our experimental results presented in Table 3 show that the nonlinear projection
outperforms the linear one by 0.28 BLEU points. This suggests that the former is effective thanthe latter for transformation between the semantic spaces of the source and the target language
even though they are learned by the same method from the same sets of data.
5.5

Translation Examples

Experimental results presented in the last four subsections show that the nonlinearly projectedcompositional phrase representations based on RAE give a better performance than the others.
In this section, we examine a number of translation examples extracted from the test set, aspresented in Table 4, to see the difference that the proposed method makes. These examples
illustrate that the decoder equipped with the proposed semantic composition and bilingual nonlinear projection is able to select better translations for both continuous (Example 1 & 2) andnon-continuous phrases (Example 3).

6

Related Work

Previous studies related to our research can be categorized into three strands as follows:
Distributed representations in monolingual settings Various methods are explored to learn
distributed vector representations for words and phrases. Among them, vector space models arewidely used, creating a vector to represent the co-occurrence relations between a target word
and its contextual words (Bullinaria and Levy,2007; Pado and Lapata,2007). Topic models
can be also used to construct distributed representations for words over topics that are learned
from data Xiao et al. (2012). Recently, a variety of deep neural networks are applied to learn
neural representations for both words and phrases in a continuous semantic space (Bengio etal.2003; Collobert and Weston,2008; Turian et al.2010; Socher et al.2012). All these methods
can be used to create monolingual word representations for use in our framework to underlay
the composition and projection operations.
Distributed representations for SMT In addition to the already mentioned three methods (Mikolov et al.2013b; Zou et al.2013a; Gao et al.2013) in the Introduction section, very
recently Zhang et al. (2014) have proposed a bilingually-constrained recursive autoencoder inthis strand, which extends the traditional semi-supervised recursive autoencoders Socher et al.
(2011) to learn semantic phrase representations. They learn representations of one languagewith constraints from the counterpart language and share learned representations for phrases in
the other language while we learn representations for the source and target language separately.
Semantic similarity models Our work is also related to semantic similarity models used invarious NLP tasks. Bullinaria et al. (2007) carried out a word-based semantic similarity task to
exam the degree of correlation between human judgments for two individual words and vector
based similarity values. Xiao et al. (2012) introduced a topic similarity model to measure thesimilarity of translation rules to a document in terms of topics. We differ from them in that
we calculate semantic similarity scores based on bilingual phrase representations learned via
semantic composition and bilingual projection.

7

Conclusion and Future Work

We have presented a flexible framework above, which learns bilingual distributed phrase representations for machine translation. In this framework, vector representations of phrases are

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 40

obtained by weighted vector addition or recursive autoencoder composition over words, which
are represented as PMI-based vectors or continuous-valued vectors. We adopt a bilingual projection neural network to build nonlinear transformations between the source and the target
language semantic space that are separately learned.
We integrate learned bilingual phrase representations into a hierarchical phrase-based SMT
system. Our experimental results suggest the following:• A semantic similarity model built on phrase representations is better than one built on word
representations.
• Recursive autoencoder is superior to simple weighted vector addition in creating phrase
vector representations from word vectors via composition.
• Nonlinear transformation is effective than linear transformation between the source andtarget language semantic space.
Our future work along this direction is to build stronger RAEs to construct vector representations for sentences.

Acknowledgment
The work was sponsored by the National Natural Science Foundation of China (Grants No.
61403269, 61432013 and 61333018) and Natural Science Foundation of Jiangsu Province(Grant No. BK20140355). We would like to thank three anonymous reviewers for their insightful comments.

References
Baroni, M.; and Zamparelli, R. 2010. Nouns are Vectors, adjectives are matrices: Representing adjectivenoun constructions in Semantic space. In Proceedings of the 2010 Conference on Empirical Methods
in Natural Language Processing. Association for Computational Linguistics, 1183-1193.
Bullinaria, J. A.; and Levy, J. P. 2007. Extracting semantic representations from word co-occurrence
statistics: A computational study. Behavior research methods, 39(3): 510-526.
Bengio, Y.; Ducharme, R.; Vincent, P.; Jauvin, C. 2006. Neural probabilistic language models. Journal of
Machine Learning Research, 3:1137-1155.
Collobert, R.; and Weston, J. 2008. A unified architecture for natural language processing: Deep neuralnetworks with multitask learning. In Proceedings of the 25th international conference on Machine
Learning, 160-167.
Chiang, D. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201-228.
Clark, S.; and Pulman, S. 2007. Combining Symbolic and Distributional Models of Meaning. In Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence (AAAI-07), 52-55. Spring
Symposium: Quantum Interaction.
Clark, J.H.; Dyer, C.; Lavie, A.; and Smith, N. A. 2011. Better hypothesis testing for statistical machinetranslation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the
Association for Computational Linguistics: Human Language Technologies, 176-181.
Coecke, B.; Sadrzadeh, M.; and Clark, S. 2010. Mathematical Foundations for Distributed Compositional
Model of Meaning. Lambek Festschrift. Linguistic Analysis, 36:345-384.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 41

Gao, J.; He, X.; Yih, W.; and Deng, L. 2013. Learning semantic representations for the phrase translation
model. arXiv preprint arXiv:1312.0482.
Harris, Z. 1968. Mathematical Structures of Language. Wiley.
Huang, P. S.; He, X.; Gao, J.; Deng, L.; Acero, A.; and Heck, L. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international
conference on Conference on information and knowledge management. ACM, 2333-2338.
Kartsaklis, D. 2014. Compositional operators in distributional semantics. Springer Science Reviews, 1-17.
Koehn, P.; Och, F. J.; and Marcu, D. 2003. Statistical phrase-based translation. In Proceedings of the2003 Human Language Technology Conference of the North American Chapter of the Association for
Computational Linguistics, 58-54.
Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.; and Dean, J.; 2013a. Distributed representations ofwords and phrases and their compositionality. Advances in Neural Information Processing Systems,
3111-3119.
Mikolov, T.; Le, Q. V.; and Sutskever, I. 2013b. Exploiting similarities among languages for machinetranslation. arXiv preprint arXiv:1309.4168.
Mitchell, J.; and Lapata, M.; 2008. Vector-based Models of Semantic Composition. In Proceedings of the
46th Annual Meeting of the Association for Computational Linguistics,236-244.
Och, F. J. 2003a. Minimum error rate training in statistical machine translation. In Proceedings of the 41st
Annual Meeting of the Association for Computational Linguistics, 160-167. Sapporo, Japan: Association for Computational Linguistics.
Och, F. J.; and Ney, H. 2003b. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51.
Pado’, S.; and Lapata, M. 2007. Dependency-based Construction of Semantic Space Models. Computational Linguistics, 33(2): 161-199.
Papineni, K,; Roukos, S.; Ward, T.; and Zhu, W. J. 2002. Bleu: a method for automatic evaluation ofmachine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, 311-318.
Socher, R.; Pennington, J.; Huang, E. H.; Andrew, Y. Ng.; and Manning, D. C. 2011. Semi-supervised
recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 151-161.
Socher, R.; Huval, B.; Manning, C. D.; and Andrew, Y. Ng. 2012. Semantic compositionality throughrecursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in
Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, 1201-1211.
Stolcke, A. 2002. Srilm-an extensible language modeling toolkit. In proceedings of the 7th International
Conference on Spoken Language Processing, 901-904.
Turian, J.; Ratinov, L.; Bengio, Y. 2010. Word representations: a simple and general method for semisupervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational
Linguistics. Association for Computational Linguistics, 384-394.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 42

Turney, P. D.; and Pantel, P. 2010. From frequency to meaning: Vector space models of semantics. Journalof artificial intelligence research, 37(1): 141-188.
Xiao, X.; Xiong, D.; Zhang, M.; Liu, Q.; Lin, S. 2012. A topic similarity model for hierarchical phrasebased translation. In Proceedings of the 50th Annual Meeting of the Association for Computational
Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 750-758.
Yang,D.; and Powers, D. M. 2008. Automatic thesaurus construction. In Proceedings of the Thirty-First
Australasian conference on Computer science-Volume 74. Australian Computer Society, Inc., 147-156.
Zou, W. Y.; Socher, R.; Cer, D.; and Manning, D. C. 2013. Bilingual Word Embeddings for PhraseBased Machine Translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural
Language Processing, 1393-1398.
Zhang, J.; Liu, S.; Li, M.; Zhou, M.; Zong, C. 2014. Bilingually-constrained phrase embeddings formachine translation. In Proceedings of the 52th Annual Meeting on Association for Computational
Linguistics. Association for Computational Linguistics. Baltimore, Maryland, USA, 111-121.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 43

Learning Bilingual Phrase Representations with
Recurrent Neural Networks
Hideya Mino
Andrew Finch
Eiichiro Sumita

hideya.mino@nict.go.jp
andrew.finch@nict.go.jp
eiichiro.sumita@nict.go.jp
National Institute of Information and Communications Technology, 3-5 Hikaridai, Seika-cho,
Soraku-gun, Kyoto 619-0289, JAPAN

Abstract
We introduce a novel method for bilingual phrase representation with Recurrent Neural Networks (RNNs), which transforms a sequence of word feature vectors into a fixed-length phrase
vector across two languages. Our method measures the difference between the vectors ofsource- and target-side phrases, and can be used to predict the semantic equivalence of source
and target word sequences in the phrasal translation units used in phrase-based statistical machine translation. Our experiments show that the proposed method is effective in a bilingualphrasal semantic equivalence determination task and a machine translation task.

1 Introduction
In recent years, continuous vector representations of word, phrase, and sentence which alleviate issues of sparsity have successfully been used in a number of natural language processing
tasks. Language models with continuous word representations (Bengio et al., 2003; Mikolov
et al., 2010; Mikolov, 2012) based on neural networks have outperformed the previous stateof-the-art approaches. These language models map each word to a dense, low-dimensional,
real-valued vector, and estimate the probability of words in a continuous space. Representations for phrases have been used in the context of Statistical Machine Translation (SMT). Zouet al. (2013) used phrasal representations for computing the distance between phrase pairs andadded a feature based on this distance into the log-linear model of a phrase-based SMT system(Koehn et al., 2003). Their method learned bilingual word representations, and subsequently
obtained the phrase-level representations by simply averaging word vectors. Continuous representations for phrases or sentences with neural networks – such as a feed-forward, recursive,or recurrent neural networks – have also been used in SMT. A phrase representation modelusing a feed-forward neural network for phrase-based SMT was proposed by Schwenk (2007,2012), and achieved significant BLEU score improvements. Since the model directly projects
feature vectors not from words but from phrases or sentences onto a continuous vector space,
the representations can contain more global semantic information.
In this paper, we propose a new method to learn bilingual phrase representations for phrasebased SMT using two Recurrent Neural Networks (RNNs) (source- and target-side) combinedin a simple linear architecture. We follow the idea of Cho et al. (2014) that the last hidden
state of the RNN is a summary representation of the whole input phrase, and the summary
representations with the same meaning are trained to be the same vector representation. The
procedure is similar to that used in the RNN of Cho et al. (2014), which predicts the next word

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 44

in the sequence with a conditional probability. In contrast to this, our model uses an objectivewith a similarity distance instead of a conditional probability, and learns to minimize the error
distance. Furthermore, we developed a novel extension of the model that uses an autoencoder,which is an architecture trained to provide a latent representation of its input by means of anonlinear encoder and an associated decoder. The objective involves three kinds of errors: anext symbol error for predicting the next word in a phrase, a semantic error for the comparisonof the summary phrase representations, and a reconstruction error for the autoencoder. The
prediction error represents how well the intermediate hidden states can predict the next word
in a sequence. The semantic error represents the dissimilarity between the final hidden stateson the source- and target-side. The reconstruction error represents how well the hidden states
represent the words in a phrase.
We introduce a bilingual phrase similarity feature derived from our proposed method asa new feature into the log-linear model of a phrase-based SMT system applied to a EnglishJapanese translation task, and confirm the effectiveness of our method on this task. The resultsof the experiments show that our model is able to indicate the effective phrase pairs for machinetranslation. In this paper, though our model is symmetric and does not differentiate between
source- and target-side, we use the following notation: the left-side RNN is referred to as thesource-side and the right-side RNN (we use overbars on the symbols to differentiate it) is referred to as the target-side.

2 Related work
In this section, we review recent work on neural network phrase representation models.
Continuous phrase representation models with a feed-forward neural network were studied
in Schwenk (2007, 2012). The models estimated translation probabilities for unseen phrases
with a continuous vector space of phrases. Le et al. (2012) proposed a similar approach toscore phrase pairs using fixed-size inputs and outputs. Devlin et al. (2014) proposed a neuralnetwork joint model (NNJM) as an extension of the NNLM (Bengio et al., 2003). The NNJM
calculates the target-side word probability by using a target-side language model in combination
with a context from the source-side. The NNJM requires a maximum length for the source-side
phrases. These approaches employ feed-forward neural networks and are constrained to operateon phrases of limited length.
The use of recursive neural networks addresses the fixed-size issue by using a tree structureof phrases and sentences. The recursive neural network maps features from subsequences of aphrase to a continuous vector on each node of the tree recursively. Li et al. (2013) described an
ITG reordering classifier which predicted phrase reorderings in SMT that was able to exploit
syntactic and semantic information, Zhang et al. (2014, 2015) proposed bilingually-constrained
recursive autoencoders, which generated phrasal embeddings for machine translation by learning to minimize the semantic distance between translation equivalents, and maximizing the
distance between non-translation pairs.
In contrast to the work on recursive networks, it is also possible to create continuous phrase
representations with RNNs. Here, simpler models are possible that do not need take the tree
structure of their input into account. Kalchbrenner and Blunsom (2013) proposed recurrent
continuous translation models based on recurrent language models (RLMs), which predict targetwords from an unbounded history of both source and target words with a conditional probability.
In their implementation convolutional neural networks were used to model the source-side. Cho
et al. (2014) proposed a gated recurrent unit which adaptively remembers and forgets its statebased on the input signal to the unit. This model was used to score each phrase pair in the phrase
table for SMT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 45

$!
!"#!

$!!!

%!

%!

#!
!"#!

)*+!#!!!

!!

!!

$!
$"#!

$!$!

%!

%!

)*+!#!$!

#$"#!

!!

!!

,-.$/0)*-""("!

'!

&

'!

'!

&

'!

"!!"#!

(!!!

"!!!

"!$! (!
$!

"!
$"#!

Figure 1: Bilingual Phrase Representation Model Architecture

3 Bilingual Phrase Representation Model
3.1

Phrase Representations

Our bilingual phrase representation model comprises two RNNs: one for source phrases, and theother for target phrases. Each RNN reads a sequence of word representations, and transforms
it into a fixed-length vector that holds the semantic content of the whole input sequence. We
call this vector the phrasal representation. Then, the model identifies phrase pairs with thesame meaning on both source- and target-side, by computing the similarity distance between
the respective source and target side phrase representations.
Figure 1 shows the framework of the bilingual phrase representation model, where rk (0 ≤k ≤ l) and r̄k (0 ≤ k ≤ m) ∈ Rn×1 are word representations of the phrases r and r̄ in aphrase pair (r, r̄), hk (0 ≤ k ≤ l) and h̄k (0 ≤ k ≤ m) ∈ Rq×1 are hidden layers, ok (1 ≤k ≤ l) and ōk (1 ≤ k ≤ m) ∈ Rn×1 are output layers, ak (1 ≤ k ≤ l) and āk (1 ≤ k ≤m) ∈ Rn×1 are autoencoder layers, c and c̄ ∈ Rq×1 , which are also the last hidden layers,
are the summary layers which contain the summary representations of the phrases r and r̄, andfour types of transformation matrices: I and Ī ∈ Rq×n , the input vocabulary transformation
matrices, F and F̄ ∈ Rn×q , the autoencoder transformation matrices, R and R̄ ∈ Rq×q therecurrent transformation matrices, and O and Ō ∈ Rn×q , the output transformation matrices.
The parameter q indicates the size of the summary representations. In Figure 1, n := 2 andq := 3 for the purposes of illustration.
Each RNN minimizes the error distance over continuous word representations of eachphrase by being trained to predict the inputs, the next inputs, and the summary representationof the whole input sequence which is shared between both source- and target-sides. Hence,
the hidden layer activation vectors H = {h0 , h1 , ..., hl } and H̄ = {h̄0 , h̄1 , ..., h̄m } contain
information of the previous input words and the next word in the input sequence. The summary
layers c and c̄, which are the last hidden layer of each RNN, contain semantic information incommon with each phrase. Any language-specific information is weakened by optimizing tojointly predict source and target. The source-side RNN learns according to following steps: first,
the activations in the hidden layers H are calculated from the word representations recursively

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 46

as follows:{hk =σ (I · r0 )σ (R · hk−1 + I · rk )

k=0
1≤k<l

(1)

where σ is a nonlinear function such as tanh. r0 and r̄0 are the representations of the source
and target start symbols. When there are no constraints on the hidden layers H, the RNNs areable to minimize the error distance by making H → [0], which is undesirable. To prevent such
behavior, the hidden layers H are normalized to have unit length. Then, the output layers o thatpredict the next word of the input, the summary layer c that predicts the target-side summary c̄,and the autoencoder layers a that predict the vector rk (representing the source word at positionk) are calculated from the hidden layer activations H in Equation (1) as follows:
ok =
c =

σ(O · hk−1 ) (1 ≤ k ≤ l)
hl

(2)
(3)

ak

σ(F · hk )(4)=

(1 ≤ k ≤ l)

Bias values for Equations (1), (2), (3), and (4) are included in the computation. To avoid overfitting, we trained each layer using the dropout method (Srivastava et al., 2014). There are threekinds of prediction error, which we denote: the next symbol Eo error, the semantic error Ec ,and the reconstruction error Ea . These were calculated by using Euclidean distance:
1 ∑||rk − ok ||22ll

Eo (r|o; θ) =

(5)

k=1

Ec (c̄|c; θ) =Ea (r|a; θ) =1||c̄ − c||22l
1 ∑||rk − ak ||2
2l

(6)
(7)

k=1

where θ = {I, R, F, O} is the set of source-side parameters to be learned, together with thebias parameters. Equation (5) represents the sum of the next symbol error distance between
each input rk in the source-side phrase and the output prediction ok . The output from the lasthidden layer hl is the summary representation c (c ≡ hl ). A shared semantic representation ofboth source and target is required, and therefore c and c̄ are trained jointly using error signals
based on the distance between them. The error is calculated using the semantic error defined in
Equation (6). Equation (7) is the sum of the reconstruction error distance between each input
rk in the source-side phrase and the autoencoder’s reconstruction ak . The autoencoder is used
for learning representations of words (Chandar A P et al., 2014), phrases (Zhang et al., 2015),
and sentences (Socher et al., 2011; Li et al., 2013). The target-side errors were calculated in thesame manner. The objective function J is the sum of the total error distance from the source
and target RNNs, and is represented by using Equations (5), (6), and (7) as:J= αEo (r|o; θ) + Ec (c̄|c; θ) + βEa (r|a; θ) + λ∥θ∥
+αEo (r̄|ō; θ̄) + Ec (c|c̄; θ̄) + βEa (r̄|ā; θ̄) + λ∥θ̄∥= 2 · Ec (c̄|c; θ, θ̄)+α(Eo (r|o; θ) + Eo (r̄|ō; θ̄))+β(Ea (r|a; θ) + Ea (r̄|ā; θ̄))+λ(∥θ∥ + ∥θ̄∥)

Proceedings of MT Summit XV, vol.1: MT Researchers' Track(8)

Miami, Oct 30 - Nov 3, 2015 | p. 47

Data
IWSLT 2007
NTCIR-10

Training
word types
en
ja
40K
9.5K 10K
720K 119K 85K
sent

Development
sent
word types
enja
0.5K 1.2K 1.3K
2.0K 5.0K 4.4K

sent
0.5K
0.5K

Test
word types
enja
0.8K 0.9K
2.4K 2.1K

Monolingual
sent
en
ja
41M 81M

Table 1: Data sets
where θ̄ is the set of target-side parameters, and α and β are the hyper-parameters for the balanceof each error. We also use an L1 regularization term in the objective function. In Equation (8),we group the semantic error Ec terms of source- and target-side (which use the summary vectors
c and c̄) into one term, and arrange the terms according to error type.
The parameters θ and θ̄ are optimized to minimize Equation (8) using the AdaGrad stochastic adaptive subgradient algorithm (Duchi et al., 2011; Green et al., 2013):
∂J
−1/2
G
∂θi−1 i

(9)

∂J 2
∂θi−1

(10)

θi

= θi−1 − ηGi= Gi−1 +where η is the learning rate, i is the number of the training iterations, and G is the sum of thesquares of the past gradients. θ and θ̄ are learned and updated in every iteration through thetraining data of phrase-pairs. The number of training iterations was determined using development data.
3.2 Word representations
Word representations, in which words are represented as real-valued vectors (Bengio et al.,
2003; Mikolov et al., 2013), serve as the inputs to our model. The word representations r are
calculated as:
ri = Lui ∈ Rn×1

(11)

where n is the number of dimensions of the vector, L ∈ Rn×|V | is a word embedding matrix,|V | is the vocabulary size, and ui is a binary vector which is zero in all positions except for the
ith index. Given a phrase which is a sequence of l words, each word has a vocabulary index iinto the columns of the word embedding matrix L. The ith column of the embedding matrix isthe word’s representation vector. The matrix L is pre-trained by training a neural network on
unlabeled monolingual data. In our experiments, we trained the matrices L and L̄ for source
and target word representations using the Word2Vec toolkit (Mikolov et al., 2013). The size of
the word representation vector n is usually determined empirically.

4 Experiments
We conducted two experiments with the Bilingual Phrase Representation Model: a phrase-pairextraction task and a phrase-based SMT task.
4.1 Data and model parameters
Both experiments were conducted on two English-Japanese (en-ja) corpora. One was fromIWSLT 2007 (Fordyce, 2007) which is in the domain of spoken travel conversation and the otherwas a patent translation corpus from NTCIR-10 (Goto et al., 2013). The Japanese sentenceswere tokenized using KyTea (Neubig et al., 2011).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 48

Table 1 provides statistics on each corpus. The “sent” column indicates the number ofsentence pairs, and the “word types” columns of “en” and “ja” indicate the number of English
and Japanese unique words. The “Monolingual” column indicates the size of the monolingual
data for the training of the word representations described in Section 3.2. For IWSLT 2007,
we used the training data for the training of the word representations. For NTCIR-10, we
used about 723K sentence pairs belonging to the physics domain, which contains the mostdocuments among the domains, according to International Patent Classification (IPC) code 1 .
720K sentence pairs from the documents published between 1993 to 2005 were used as thetraining data, and 2.0K and 0.5K sentence pairs randomly sampled from the 2006 and 2007
documents were used as the development and test data respectively. We also used the 2006 and2007 documents to extract the similar phrases. Furthermore, we used the English and Japanesemonolingual corpus in NTCIR-10 for the training of word representations.
For the extraction of phrase pairs, we used MGIZA++ (Gao and Vogel, 2008) and growdiag-final-and heuristics of the Moses toolkit (Koehn et al., 2007). To facilitate effective learning, we used only phrase pairs that contained content words (i.e. had at least one noun or verb
word in the phrase) and had a high translation probability (a threshold on the source-given-target
conditional probability was used). We extracted phrase pairs from the training, development andtest data. The training phrase-pairs were used for training the neural network models. The development phrase-pairs were used to control the training of the models. The model was trainedfor 4,000 iterations, and estimated parameters θ and θ̄ by evaluating the highest accuracy of
the top-1 phrase pair extracted with the development data. The evaluation of the accuracy was
performed as follows: for each source-side phrase in a 100-phrase pair development set, thesystem was requested to choose the top-n candidate target word sequences from the target-side
100 phrases. The minimum error distance defined in Eq. (8) was used to produce the top-nlist. The test phrase-pairs were used for the experiment on the phrase-pair extraction. Consequently, we extracted about 316K phrase pairs from IWSLT 2007 and 23M phrase pairs fromthe NTCIR-10 training set. Due to the computation time, we randomly selected 10K phrase
pairs for IWSLT 2007 and 100K phrase pairs for NTCIR-10 as the training set from the full setof phrase pairs. For the development set, we randomly selected another 300 phrase pairs. We
also extracted 100 phrase pairs from the test data for the experiment on phrase-pair extraction.
For the parameters of the model, the input and output vector-size n was set to 200. The
summary vector-size q was also set to 200. The activation function σ was tanh. The dropout
rate was 0.9 for the hidden layers and 0.5 for the other layers. The learning rate η was set to0.01 for the experiments with IWSLT 2007 and 0.02 for the experiments with NTCIR-10. The
regularization rate λ was set to 0.01. The hyperparameters α and β in Equation (8) were set to0.01. All weight parameters θ and θ̄ were randomly initialized, and all bias parameters were
initialized to zero.
4.2

Experimental design

4.2.1 Phrase-pair Extraction
We did two sub-experiments for the phrase-pair extraction: the evaluation of the accuracy andthe extraction of phrases with similar meaning. The accuracy was calculated with the test
phrase-pairs. In order to mitigate the issue of the training process terminating in a local minimum, we evaluated the accuracy at each iteration on four data sets: three different development
data sets (DEV.1, DEV.2, and DEV.3) and a fourth (closed) set which was the test data itself.
This resulted in four different models, each defined by the estimated parameters θ and θ̄ at theiteration that gave rise to the highest accuracy on the respective data set. Each of the development data sets contained 100 phrase pairs sampled randomly without replacement from the full
1 SECTION G of IPC code indicates the physics domain

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 49

IWSLT 2007

NTCIR-10

BPRM (without autoencoder)
1-best10-bestBPRM
1-best10-bestBPRM (without autoencoder)
1-best10-bestBPRM
1-best10-best

DEV.1

DEV.2

DEV.3

closed

0.03
0.23

0.06
0.25

0.04
0.24

0.10
0.32

0.03
0.27

0.03
0.28

0.03
0.28

0.08
0.28

0.20
0.45

0.17
0.43

0.18
0.44

0.21
0.48

0.20
0.43

0.24
0.41

0.20
0.45

0.24
0.47

Table 2: Accuracy of the phrase-pair extraction: 1-best and 10-best on three development sets

300-pair development set.
The data for extracting the similar phrases was obtained by searching for English phrases
that were close to their Japanese counterpart phrases in NTCIR-10. We used the English phrases
in unseen sentences published in 2006 and 2007 and the Japanese phrases from sentences randomly selected from the training sets. We calculated the error distance between the Japaneseand the English phrases with the model terminated using the accuracy on DEV.1. To limit the
number of the English phrase candidates, we only used the English sentences that were similar
to the Japanese sentences. The similarity was assessed using the number of lemmatized words
in the Japanese sentences, that could be translated to lemmatized words in the English sentencesby using a Japanese-English dictionary.
4.2.2

Phrase-based SMT

The phrase-based SMT experiments were performed with the two models, in which the parameters θ and θ̄ were estimated on DEV.1 and DEV.2 of the phrase-pair extraction experiment,using the phrase-pairs extracted using the Moses toolkit. We added the inverse of the error
distance used in the ranking experiments, as a feature into the log-linear model of the Moses
decoder. The 5-gram language models were built using the SRILM toolkit (Stolcke, 2002) withmodified Kneser-Ney smoothing (Chen and Goodman, 1996). For word and phrase alignments,
we used MGIZA++ and grow-diag-final-and heuristics. To tune the weights with respect to the
BLEU score (Papineni et al., 2002), we used n-best batch MIRA (Cherry and Foster, 2012). The
distortion limit parameter was set to 10. We evaluated each model on BLEU using the NIST’s
mteval-v13a.pl 2 script. Statistical significance testing of the BLEU differences was performed
using paired bootstrap resampling (Koehn, 2004).
For both experiments, we tested with two models. The first was the proposed BilingualPhrase Representation Model (BPRM). The second was the same BPRM model with the autoencoder layer removed.

5 Results and Analysis
Tables 2 and 3 present the results of the phrase pair extraction task, and Table 4 presents theresults of the phrase-based SMT task.
2 ftp://jaguar.ncsl.nist.gov/mt/resources/mteval-v13a.pl

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 50

English Phrase in training data
as was described above
a series of operations areeach processors
an exposing device

English Phrases from unseen data
as has been described above
and the sequence of operations
each of the processors
the exposure device / the exposure apparatus

Table 3: Examples of the new phrase pairs extracted with BPRM in NTCIR-10

IWSLT 2007

NTCIR-10PBMT+BPRM (without autoencoder)
+BPRM
PBMT
+BPRM (without autoencoder)+BPRM

DEV.1 DEV.2
46.55
47.75
47.65
48.19
48.08
32.13
32.14
32.35
32.49
32.29

Table 4: BLEU scores on the IWSLT 2007 task (PBMT denotes phrase-based SMT)

5.1

Phrase-pair Extraction

In Table 2, the results of the experiments on NTCIR-10 data show higher levels of accuracy thanthe experiments on IWSLT 2007 data. The reason for this may be indicated in Table 1 whichshows that the number of the word types on IWSLT 2007 is smaller than on NTCIR-10, and the
proportion of words that appear multiple times in the corpus was 70% for IWSLT 2007 and 30%
for NTCIR-10. The set of phrases from the IWSLT 2007 data is likely to contain many similarphrases, thereby making the discrimination more difficult. The differences between DEV.1,
DEV.2 and DEV.3 were small. It is likely that there is no local minimum problem in these threemodels. Table 3 shows examples of phrase pairs in the training data and the English phrases
which were extracted from the English monolingual documents with BPRM, illustrating thekinds of semantically similar phrases our model is capable of identifying.
5.2

Phrase-based SMT

The results of the phrase-based SMT experiments on IWSLT 2007 data show that the proposed
method was able to improve machine translation quality. The statistical significance tests between PBMT and the other models shows a significant improvement on both DEV.1 and DEV.2
at p < 0.05. Although the results were not statistically significant, the full BPRM approachachieved higher BLEU scores than the BPRM without the autoencoder. Therefore we believe
it is likely that the autoencoder is effective for the improvement of translation quality. For
NTCIR-10, the improvements in performance were smaller than on the IWSLT 2007 data set.
In terms of computational time for training the model, training with 10K phrase pairs on
IWSLT 2007 took about 40 seconds for one iteration, and training with 100K phrase pairson NTCIR-10 took about 3 minutes for one iteration. Training was performed on an 8-core
2.00GHz Intel Xeon CPU.
In summary, our model was capable of identifying phrase-pairs with semantically source
and target word sequences, and this knowledge could be exploited to yield an respectable improvement in machine translation quality.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 51

6 Conclusion
In this paper, we proposed a Bilingual Phrase Representation Model which learns phrase representations by using source- and target-side Recurrent Neural Networks. We demonstrated theeffectiveness of the proposed model on an English-Japanese corpus on two tasks: phrase-pairextraction and statistical machine translation. Future avenues of research include investigating
hyper-parameter tuning for the objective function, and discovering a method to select appropriate initial values of the weights which were set randomly in this work.

Acknowledgements
We are deeply grateful to Taro Watanabe, Atsushi Fujita and anonymous reviewers for theirsuggestions and insightful comments on the early version of this paper.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 52

References
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model. J.
Mach. Learn. Res., 3:1137–1155.
Chandar A P, S., Lauly, S., Larochelle, H., Khapra, M., Ravindran, B., Raykar, V. C., and Saha, A. (2014).
An autoencoder approach to learning bilingual word representations. In Ghahramani, Z., Welling, M.,
Cortes, C., Lawrence, N., and Weinberger, K., editors, Advances in Neural Information Processing
Systems 27, pages 1853–1861. Curran Associates, Inc.
Chen, S. F. and Goodman, J. (1996). An empirical study of smoothing techniques for language modeling.
In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages
310–318, Santa Cruz, California, USA. Association for Computational Linguistics.
Cherry, C. and Foster, G. (2012). Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages 427–436, Montréal, Canada. Association for Computational Linguistics.
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y.
(2014). Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP), pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust neuralnetwork joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pages 1370–1380, Baltimore,
Maryland. Association for Computational Linguistics.
Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121–2159.
Fordyce, C. S. (2007). Overview of the 4th international workshop on spoken language translation iwslt
2007 evaluation campaign. In In Proceedings of IWSLT 2007, pages 1–12, Trento, Italy.
Gao, Q. and Vogel, S. (2008). Parallel implementations of word alignment tool. In Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 49–57, Columbus, Ohio.
Association for Computational Linguistics.
Goto, I., Chow, K., Lu, B., Sumita, E., and Tsou, B. K. (2013). Overview of the patent machine translation
task at the NTCIR-10 workshop. In Proceedings of the 10th NTCIR Conference on Evaluation ofInformation Access Technologies, NTCIR-10, National Center of Sciences, Tokyo, Japan, June 18-21,
2013.
Green, S., Wang, S., Cer, D., and Manning, C. D. (2013). Fast and adaptive online training of feature-richtranslation models. In Proceedings of the 51st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 311–321, Sofia, Bulgaria. Association for Computational
Linguistics.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings of the2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709, Seattle,
Washington, USA. Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 53

Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Lin, D. and Wu,
D., editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source
toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association
for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages
177–180, Prague, Czech Republic. Association for Computational Linguistics.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In In Proceedings of
HLT-NAACL, pages 48–54, Edmonton, Canada.
Le, H.-S., Allauzen, A., and Yvon, F. (2012). Continuous space translation models with neural networks.
In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 39–48, Montréal, Canada. Association for
Computational Linguistics.
Li, P., Liu, Y., and Sun, M. (2013). Recursive autoencoders for ITG-based translation. In Proceedings ofthe 2013 Conference on Empirical Methods in Natural Language Processing, pages 567–577, Seattle,
Washington, USA. Association for Computational Linguistics.
Mikolov, T. (2012). Statistical language models based on neural networks. PhD thesis, Brno Universityof Technology.
Mikolov, T., Karafit, M., Burget, L., Cernock, J., and Khudanpur, S. (2010). Recurrent neural network
based language model. In Kobayashi, T., Hirose, K., and Nakamura, S., editors, INTERSPEECH, pages
1045–1048. ISCA.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations ofwords and phrases and their compositionality. In Advances in Neural Information Processing Systems
26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting
held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111–3119.
Neubig, G., Nakata, Y., and Mori, S. (2011). Pointwise prediction for robust, adaptable japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies, pages 529–533, Portland, Oregon, USA. Association for
Computational Linguistics.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation ofmachine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Schwenk, H. (2007). Continuous space language models. Comput. Speech Lang., 21(3):492–518.
Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine translation.
In Proceedings of COLING 2012: Posters, pages 1071–1080, Mumbai, India. The COLING 2012
Organizing Committee.
Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., and Manning, C. D. (2011). Semi-supervised recursive
autoencoders for predicting sentiment distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 151–161, Edinburgh, Scotland, UK. Association
for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 54

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: A simple
way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–
1958.
Stolcke, A. (2002). Srilm an extensible language modeling toolkit. In In Proceedings of the SeventhInternational Conference on Spoken Language Processing, pages 901–904.
Zhang, J., Liu, S., Li, M., Zhou, M., and Zong, C. (2014). Bilingually-constrained phrase embeddings
for machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 111–121, Baltimore, Maryland. Association for
Computational Linguistics.
Zhang, J., Liu, S., Li, M., Zhou, M., and Zong, C. (2015). Towards machine translation in semantic vector
space. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 14(2):9:1–9:26.
Zou, W. Y., Socher, R., Cer, D., and Manning, C. D. (2013). Bilingual word embeddings for phrasebased machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural
Language Processing, pages 1393–1398, Seattle, Washington, USA. Association for Computational
Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 55

A Pilot Study Towards End-to-End MT Training
Feifei Zhai

ffzhai2012@gmail.com

Queens College, City University of New York, Queens, NY 11367

Liang Huang∗lianghuang.sh@gmail.com

School of EECS, Oregon State University, Corvallis, OR 97330

Abstract
Typical MT training involves several stages, including word alignment, rule extraction, translation model estimation, and parameter tuning. In this paper, different from the traditional
pipeline, we investigate the possibility of end-to-end MT training, and propose a framework
which combines rule induction and parameter tuning in one single module. Preliminary experiments show that our learned model achieves comparable translation quality to the traditional
MT training pipeline.

1

Introduction

bitext(c)

GIZAbitext

(b)

GIZA

(a)

bitextTypically, as shown in Figure 1(a), traditional machine translation (MT) training involves a longpipeline of several stages, including word alignment (by GIZA++), rule extraction, translation
model (TM) estimation (by max-likelihood), and parameter tuning by M ERT (Och, 2003), P RO
(Hopkins and May, 2011), or M IRA (Watanabe et al., 2007; Chiang et al., 2008). This cascaded
procedure inevitably propagates errors downstream, while at the same time making MT training
overly complicated.
hard wordalignment

rule rule TM rule
full
tune
extr. set est. probs
model

ideal framework for rule induction &parameter tuning

full
model

joint framework for rule
soft alignprobs induction & parameter tuning

full
model

Figure 1: Three approaches to MT training. (a) the standard pipeline; (b) ideal end-to-end MT
training; (c) our work, in between (a) and (b), combines rule induction, TM estimation, andparameter tuning in one module.
In this paper, instead of following the traditional training pipeline, we explore the possibility of end-to-end MT training, which would ideally induce a full model with translation∗ Work done while Prof. Liang Huang was in City University of New York.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 56

rules at the same time (as shown in Figure 1 (b)). Practically, it is a big challenge, because thesearch space would be too prohibitive if we consider all possible rules from scratch without any
constraints from word alignment.
To make this idea computationally affordable, we take a step back and propose a joint
learning framework to do rule induction and parameter tuning together, shown in Figure 1(c). Instead of knowing nothing at the beginning, we use word translation probabilities from
GIZA++ to compute the lexical translation probabilities of phrase pairs, which is an effective
feature guiding us to select good translation rules. Note that even with these probabilities, thisis still a non-trival problem, since the joint learner still needs to deal with the entire space of all
possible translation rules and all possible decoding derivations.
For simplicity reasons we use phrase-based translation. Final experiments show that thejoint learned model achieves comparable performance to the conventional training pipeline in asmall scale. To our best knowledge, this is the first time, although on small data, verifying that
it is possible to do effective end-to-end MT training. The most significant contribution of thispaper lies in this point. We believe this will be a promising direction to MT research.

2

Joint Framework for Rule Induction and Parameter Tuning

Algorithm 1 gives our joint framework in detail. We start with an empty rule set R = ∅, andfor each sentence pair (x, y) where x is the source input and y the target translation, we try
forced decoding finding all derivations that can map x to y (line 5), using all possible phrase
pairs from (x, y). This is because without a rule set to start with, theoretically any portion of x
could map to any portion of y and the learner needs to figure out which phrase pairs make more
sense. Here f (·) and e(·) return the source and target projection of a derivation respectively.
This forced decoding, although much more constrained than real decoding, is still intractable,
and we have to resort to beam search similar to those employed in real decoding, i.e., the set D
in line 5 is the “best-scoring” subset of all possible forced derivations .
Algorithm 1 Joint rule induction and param. tuning.
1: R ← ∅2: w ← 0. initial rule set. initial parameters

3: repeat4:
for each sentence pair (x, y) in bitext do
5:
D(x, y) ← {d | f (d) = x, e(d) = y}
6:
R ← R ∪ {r | r ∈ d, d ∈ D(x, y)}. forced decoding. add new rules

7:
8:
9:

recalculate conditional probs for each r ∈ R
for each sentence pair (x, y) in bitext do
d0 ← argmax w · Φ(x, d). real decoding

d:f (d)=x

10:
11:

0

if e(d ) 6= y thend∗ ← argmax w · Φ(x, d). wrong translation?
. best gold deriv.

d∈D(x,y)12:w ← w + Φ(x, d∗ ) − Φ(x, d0 ). update model

13: until convergedAfter forced decoding we collect translation rules in the forced derivations and add them tostanding rule set R (line 6). Then we recalculate the rule conditional probabilities (line 7, TM
estimation). After that we perform real decoding, trying to find the best translation (line 9) 1 .
1 At first, since all the weights are set to 0, all derivations have the same score 0. Both the best-scoring forced
derivation and real decoding derivation are selected randomly

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 57

If this translation e(d0 ) is different from the reference translation y, then an update is needed to
reward the highest-scoring forced (or “gold”) derivation d∗ and to penalize the highest-scoring
non-gold (Viterbi) derivation d0 (line 12). We apply a latent-variable max-violation perceptron(Huang et al., 2012), which has been successfully used in MT training (Yu et al., 2013; Zhao
et al., 2014), to do update.
Technically, this framework is similar to (Xiao and Xiong, 2013). The main differenceis that we try to do end-to-end MT training, and combine rule induction, TM estimation, andparameter tuning together, while they only focus on rule induction. Moreover, to accommodate
the vast amount of search errors in decoding, we update weights by max-violation perceptron,
performing prefix instead of full-sequence updates, whereas the updates in Xiao and Xiong(2013) are still full-sequence updates which are insensitive to search errors. For simplicity
reasons we do not make this difference explicit in line 12.

3

Phrase-based Forced Decoding

Forced decoding generates those derivations that can produce the exact reference translation.
For example, given the following sentence pair,
Source: Bùshı́ yǔ Shālóng jǔxı́ng le huı̀tán
Target: Bush held a talk with Sharon
One possible derivation created by forced decoding is as follows:) : (0, “”)
r1(•1) : (s1 , “Bush”)
r2(• •••6 ) : (s2 , “Bush held a talk”)
r3(•••3 •••) : (s3 , “Bush held a talk with Sharon”)(0

where each hypothesis is in form (v) : (s, p), in which v is the coverage vector (a • indicatesthe source word at this position is already “covered (or translated)”), and (s, p) is the score andpartial translation of each state.
Generally, we can employ a traditional phrase-based decoder to do forced decoding. However, the distortion limit in the decoder will prohibit long-distance reorderings, and exclude
many sentence pairs from getting forced derivations, especially for language pairs with very
different word orders, such as Chinese and English.
Hence, in order to do better forced decoding, we use a more flexible limit to constrain
the number of gaps during decoding (also known as “IBM constraint” in (Zens et al., 2004)),
rather than distortion limit. Here, a gap refers to a consecutive of positions that are not coveredin the coverage vector. For example, consider the third hypothesis of the above derivation,(• •••6 ) : (s2 , “Bush held a talk”), its coverage vector has one gap, i.e., the two untranslated
words. Also, we don’t want the decoding process to be too flexible on reordering, so we demandthat there are at most two gaps in a specific coverage vector (See Figure 2).
gap
gap

gap
gap

Figure 2: Two possible scenarios in gap-based decoding. The gray boxes denote covered segments.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 58

Obviously, compared to distortion limit, gap limit is more flexible for decoding, especiallywhen there is only one gap in the coverage vector, we can jump to any uncovered position of
the source sentence for translation. Hereafter, we use dl-based decoding to denote the decoding
algorithm using distortion limit, while gap-based decoding denotes the algorithm using gap
limit.
In addition, same as forced decoding, real decoding can also be dl-based or gap-based. We
use the same choice for forced and real decoding.

4

Rule Generation and Collection

As described in § 2, since there is no predefined rule set, we need to consider all possible
phrase pairs from the training sentence pair. To reduce the search space of phrase pairs, we set
the maximum phrase length to 3 2 . As an example, considering the above sentence pair, weenumerate all possible source and target phrases from it for forced decoding:possible source phrases
Bùshı́Bùshı́ yǔBùshı́ yǔ Shālóngyǔ
yǔ Shālóngyǔ Shālóng jǔxı́ng...

possible target phrases
Bush
Bush held
Bush held aheldheld aheld a talk...

From these phrases, we can pick a source phrase and a target phrase arbitrarily to forma phrase rule, and apply it to forced decoding. For example, as a start, if we select Bùshı́ andBush respectively, we can create hypothesis (•1) : (s1 , “Bush”) in the above derivation.
We consider all possible combinations of these phrases as rules for forced decoding.
In addition, since training sentence pairs do not always contain equal information on bothsides, we introduce two null rules hf, nulli and hnull, ei to capture the redundant information
for forced decoding. hf, nulli deletes a source word, and hnull, ei inserts a target word to thetranslation.
After forced decoding, we collect the rules used in forced derivations and add them to thestanding rule set R (line 6 in Algorithm 1). 3 Based on the rule counts in R, we recalculate therule conditional probabilities by the formula from (DeNero et al., 2006):
φ(e|f ) =c(f, e)c(f ) + k l−1(1)where f and e are the source and target phrase, c(·) is the count of phrase or phrase pair, lis the length of phrase f , and k is a tuning parameter. The formula boosts the probability ofshort phrases, and results in better translation quality. After some validation experiments, we
set k = 4.0 finally.
Similar to our rule generation process, Wuebker and Ney (2013) has proposed a lengthbased training method to do rule induction by EM algorithm. The major difference between ourframework and theirs is that their algorithm only relates to rule induction, and still need M ERT
to do parameter tuning, while we combine rule induction and parameter tuning together. In thisway, the two step in our framework can help each other, but the two step in (Wuebker and Ney,
2 We have also tried longer length limit, but the performance becomes worse, because with longer limit, the learner

greatly prefers longer phrases, which are not good at generalization.
3 We count all rules in all unpruned forced derivations, which are stored in a lattice. The rules are counted based on
its contribution to the derivation, i.e., the score it added to the derivation.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 59

2013) are isolated. We think it is possible to integrate their method into our framework. We
leave it as our future work.

5

Parameter Tuning

We apply a max-violation perceptron, which has successfully scaled the MT tuning processfrom dev set to training data (Yu et al., 2013), to do parameter tuning. We skip the details here.
The basic idea is to find the step where the difference (violation) between the best forced decoding derivation and the best real decoding derivation is maximal, and then update parametersat this step so that most information can be learned.
Theoretically, the max-violation perceptron allows arbitrary features. However, in our joint
learning scenario, we find it is very easy to get overfitting with sparse features. We conjecturethat this is because sparse features have a tight connection to rules. Once some bad rules are
introduced, it is difficult for the learner to correct them. We will make more effort on this in
future.
Here, we only use dense features, which are the same as the ones for phrase-based translation, including bidirectional translation probabilities (computed by Formula (1) in § 4) 4 ,bidirectional lexical translation probabilities (estimated based on (Koehn et al., 2003) by wordtranslation probabilities from Moses (Koehn et al., 2007) based on GIZA++), language model,
rule penalty, length penalty, and distortion cost. To simplify the system, we haven’t used thelexicalized reordering model here.

6

Related WorkOn a high level, this work is a combination of two different research directions. One direction
is to induce translation rules directly from bitext, rather than using word alignment (Marcu andWong, 2002; Cherry and Lin, 2007; Zhang et al., 2008; DeNero et al., 2008; Blunsom et al.,2009; Neubig et al., 2011; Levenberg et al., 2012; Xiao and Xiong, 2013). They can learn
better translation rules, but don’t care about parameter tuning of SMT. Another direction is
discriminative training for MT parameter tuning (Liang et al., 2006; Arun and Koehn, 2007;
Blunsom et al., 2008; Flanigan et al., 2013; Green et al., 2013; Yu et al., 2013; Zhao et al.,2014). Both the two directions have achieved promising results. We differ from these works inthat we make efforts to combine their spirit together, and try to do rule induction and parametertuning in one step.

7

Experiments

To evaluate our method, we conduct experiments on Chinese-to-English transaltion. Since weneed to do forced decoding and real decoding on training corpus each iteration, to guarantee thetraining efficiency, here we use a small scale data. It includes about 100K sentence pairs from
FBIS, where the length of each sentence is less than 30 words. We use GIZA++ and grow-diagfinal-and strategy to create symmetric word alignment. We train a trigram language model on
1.5M English sentences.
We base our experiments on Cubit, a state-of-the-art phrase-based system in Python Huang
and Chiang (2007). For the joint learning method, we set the beam size for forced decoding as
10, real decoding as 30. A maximum phrase length of three was used for both baseline and our
joint system. The beam size for final test decoding is set to 50.
We take the newswire portion of NIST MT 2006 data as our dev set, and the NIST MT
03-05 data and the newswire portion of 2008 data as the test set. For baseline, we use M ERT
Och (2003) to tune weights.
4 For the newly generated rules which do not have any counts, we assign a very small probability for them.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 60

force decodable percentage

90%
80%

dl=10 w/ Null Rulegap w/ Null Rule70%

dl=6 w/ Null Rule60%
50%
gap w/o Null Rule40%
dl=10 w/o Null Rule30%

dl=6 w/o Null Rule

20%
10%
1

2

3

4

5
6
iteration

7

8

9

10

Figure 3: The percentage of reachable sentences in forced decoding at each iteration, withvarious decoding options.
7.1

DL-based Decoding vs. Gap-based Decoding

We use dl-based decoding and gap-based decoding respectively to promote our joint learning
framework. Since we only use dense feature for training, the learning process peak very fast,
and we can reach the best BLEU score on dev set in 4 - 5 iterations for both dl-based decoding
or gap-based decoding.
Figure 3 compares the forced decoding reachability, and Table 1 shows the corrspondingtranslation result of different decoding settings. From the Figure and Table, we can first certify
the necessity of Null Rule, which improves translation quality significantly. Also, distortion
limit has a big influence. Big limit (dl=10) allows more flexibility on forced decoding, and thus
extract more effective rules, and get better translation quality than smaller limit (dl=6). Finally,
gap limit gets the best translation quality among all the three, verifying the effectiveness of
gap-based decoding.

w/o Null Rulew/ Null Rule

decoding
dl=6
dl=10
gap ≤ 2dl=6
dl=10
gap ≤ 2# rule
0.44M
0.55M
0.59M
0.93M
0.94M
1.02M

dev
22.04
23.86
23.85
23.05
24.19
24.42

test
21.80
22.56
22.85
22.17
23.47
23.67

Table 1: Rule set size and B LEU results of our joint learning method with different decoding
algorithms.
An interesting phenomenon in Figure 3 is that the gap-based decoding algorithm gets moreforced decodable sentences than dl-based decoding (dl=10) without Null Rule, but gets fewer
ones after adding Null Rule. This is because when we use Null Rule, once the jump exceeds

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 61

system
dl=6
baseline
gap
dl=6
joint
gap

# rule
1.18M
1.18M
0.93M
1.02M

03
21.72
21.19
20.45
21.93

04
25.47
25.18
24.37
25.92

05
22.01
21.60
20.52
21.86

08
21.59
21.57
20.12
21.85

All
23.37
23.05
22.17
23.67

Table 2: B LEU scores of different translation systems. The baseline system uses conventional
pipeline training. “All” is a combination of the 4 test sets.

distortion limit (bigger than 10), the dl-based forced decoder will not jump, but use Null Rule togenerate the corresponding target word, and adopt another Null Rule to delete the source word
later. In this way, the dl-based decoder could handle any long-distance jump. Since dl-based
decoding is also more flexible than the gap-based one in short-distance jump, it is reasonable
that dl-based decoding gets more forced decodable sentences after adding Null Rule. This isalso the reason why dl-based decoding collects fewer translation rules (Table 1) with moreforced decodable sentences. Since it utilizs many Null Rules on building forced derivations, thenumber of useful rules decreases in these derivations, resulting in fewer useful translation rules.
7.2

Translation Results

Table 2 shows the final translation results. First, it is interesting that gap-based decoding isbetter than dl-based in our joint framework, but slightly worse in baseline. This is because withtarget translation as a constraint in forced decoding, gap limit is more flexible to get better forced
derivation (3), and thus more effective rules for better translation. But with a fixed rule set inbaseline, its flexibility will introduce more noise into the beam, leading to worse performance.
Moreover, our gap-based joint learned model is better than gap-based baseline by 0.6
BLEU points, and better than dl-based baseline by 0.3 BLEU points.
7.3

Large Data

As previous experiment only trains word alignment on the small 100K corpus, which might
create bad alignment and be unfair to baseline, we try another experiment and train word alignment on 2M sentence pairs, but translation model on the original 100K data. The result on the“All” test set is 24.46 for dl-based baseline (dl=6), and 24.57 for our gap-based joint learned
model.
7.4

Discussions

The above two experiments have shown that our joint learning framework is comparable to the
traditional MT training pipeline.
Based on the experiments, we conclude that three factors influence translation quality.
First, in the training corpus, only about 82.9% of all sentence pairs are forced decodable, meaning that 17.1% sentence pairs are excluded from extracting useful rules. Second, due to theoverfitting problem, we only use dense features here. Yu et al. (2013) has shown that if onlyuse dense features, max-violation perceptron MT training cannot get good translation quality.
At last, we need to handle the space of all possible phrase pairs, and the space of all possible
decoding derivations. The search space is too large to optimize effectively.
However, even with these problems, we can still get comparable results with the baseline
system, indicating that end-to-end MT training has a great potential to improve, and would be a
promising direction for MT research.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 62

8Conclusion and Future Work

We present a pilot study on end-to-end MT training, and propose a joint framework combining
rule induction, TM estimation, and parameter tuning in one module. Preliminary experiments
show comparable translation quality to the conventional training pipeline.
Before we can fulfill the ambitious goal of end-to-end MT training, there are still a lot ofthings to do. In future, we will explore how to effectively use sparse features to improve thetranslation quality. We also plan to investigate the solution of scaling this framework to large
data set. Currently, we need to solve two problems for scaling. The first one is how to deal withlong sentences. The number of candidate phrase pairs is exponential to the length of training
sentence pairs, making the learning process intractable for long sentence pairs. The practicalsolution is to segment them into several short ones. We can use IBM model 1 to do that, which
has been demonstrated to be effective for SMT (Xu et al., 2005). The second problem is how
to use the large data set with millions of sentence pairs. Currently, parallelization seems to be
the only solution. After segmentation, we can get millions of short sentence pairs, and thenparallelize them to clusters for training.

9

Acknowledgement

We thank the three anonymous reviewers for the valuable comments, and Kai Zhao for discussions. This project was supported in part by DARPA FA8750-13-2-0041 (DEFT), NSF
IIS-1449278, and a Google Faculty Research Award.

References
Arun, A. and Koehn, P. (2007). Online learning methods for discriminative training of phrase based
statistical machine translation. Proc. of MT Summit XI, 2(5):29.
Blunsom, P., Cohn, T., Dyer, C., and Osborne, M. (2009). A gibbs sampler for phrasal synchronous
grammar induction. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL
and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 Volume 2, ACL ’09, pages 782–790, Stroudsburg, PA, USA. Association for Computational Linguistics.
Gibbs sample initialized by GIZA alignments.
Blunsom, P., Cohn, T., and Osborne, M. (2008). A discriminative latent variable model for statisticalmachine translation. In ACL, pages 200–208.
Cherry, C. and Lin, D. (2007). Inversion transduction grammar for joint phrasal translation modeling.
In Proceedings of SSST, NAACL-HLT 2007 / AMTA Workshop on Syntax and Structure in Statistical
Translation, pages 17–24, Rochester, New York. Association for Computational Linguistics.
Chiang, D., Marton, Y., and Resnik, P. (2008). Online large-margin training of syntactic and structuraltranslation features. In Proceedings of EMNLP 2008.
DeNero, J., Bouchard-Côté, A., and Klein, D. (2008). Sampling alignment structure under a Bayesiantranslation model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language
Processing, pages 314–323, Honolulu, Hawaii. Association for Computational Linguistics.
DeNero, J., Gillick, D., Zhang, J., and Klein, D. (2006). Why generative phrase models underperformsurface heuristics. In Proceedings on the Workshop on Statistical Machine Translation, pages 31–38,
New York City. Association for Computational Linguistics.
Flanigan, J., Dyer, C., and Carbonell, J. (2013). Large-scale discriminative training for statistical machinetranslation using held-out line search. In Proceedings of NAACL 2013.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 63

Green, S., Wang, S., Cer, D., and Manning, C. D. (2013). Fast and adaptive online training of feature-richtranslation models. In Proceedings of the 51st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 311–321, Sofia, Bulgaria. Association for Computational
Linguistics.
Hopkins, M. and May, J. (2011). Tuning as ranking. In Proceedings of EMNLP.
Huang, L. and Chiang, D. (2007). Forest rescoring: Fast decoding with integrated language models. In
Proceedings of ACL, Prague, Czech Rep.
Huang, L., Fayong, S., and Guo, Y. (2012). Structured perceptron with inexact search. In Proceedings of
NAACL.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: open source
toolkit for statistical machine translation. In Proceedings of ACL: Demonstrations.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of NAACL,
pages 127–133.
Levenberg, A., Dyer, C., and Blunsom, P. (2012). A bayesian model for learning scfgs with discontiguous rules. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language
Processing and Computational Natural Language Learning, pages 223–232, Jeju Island, Korea. Association for Computational Linguistics.
Liang, P., Bouchard-Côté, A., Klein, D., and Taskar, B. (2006). An end-to-end discriminative approach tomachine translation. In Proceedings of COLING-ACL.
Marcu, D. and Wong, D. (2002). A phrase-based, joint probability model for statistical machine translation. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing,pages 133–139. Association for Computational Linguistics.
Neubig, G., Watanabe, T., Sumita, E., Mori, S., and Kawahara, T. (2011). An unsupervised model forjoint phrase alignment and extraction. In Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics: Human Language Technologies, pages 632–641, Portland, Oregon,
USA. Association for Computational Linguistics.
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of ACL,
pages 160–167.
Watanabe, T., Suzuki, J., Tsukada, H., and Isozaki, H. (2007). Online large-margin training for statisticalmachine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural
Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 764–
773, Prague, Czech Republic. Association for Computational Linguistics.
Wuebker, J. and Ney, H. (2013). Length-incremental phrase training for SMT. In Proceedings of theEighth Workshop on Statistical Machine Translation, pages 309–319, Sofia, Bulgaria. Association for
Computational Linguistics.
Xiao, X. and Xiong, D. (2013). Max-margin synchronous grammar induction for machine translation.
In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages
255–264, Seattle, Washington, USA. Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 64

Xu, J., Zens, R., and Ney, H. (2005). Sentence segmentation using ibm word alignment model 1. In
Proceedings of EAMT 2005 (10th Annual Conference of the European Association for Machine Translation), pages 280–287.
Yu, H., Huang, L., Mi, H., and Zhao, K. (2013). Max-violation perceptron and forced decoding for
scalable MT training. In Proceedings of EMNLP.
Zens, R., Ney, H., Watanabe, T., and Sumita, E. (2004). Reordering constraints for phrase-based statisticalmachine translation. In Proceedings of Coling 2004, pages 205–211, Geneva, Switzerland. COLING.
Zhang, H., Quirk, C., Moore, R. C., and Gildea, D. (2008). Bayesian learning of non-compositional
phrases with synchronous parsing. In Proceedings of ACL-08: HLT, pages 97–105, Columbus, Ohio.
Association for Computational Linguistics.
Zhao, K., Huang, L., Mi, H., and Ittycheriah, A. (2014). Hierarchical mt training using max-violation
perceptron. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 785–790, Baltimore, Maryland. Association for Computational
Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 65

Automatic Detection of Antecedents ofJapanese Zero Pronouns Using a
Japanese-English Bilingual Corpus
Dong Zhan

1301211064@pku.edu.cn
School of Software and Microelectronics, Peking University, Beijing, 100871, China

Hiromi Nakaiwa

nakaiwa@is.nagoya-u.ac.jp
Graduate School of Information Science, Nagoya University, Aichi, 464-8601, Japan

Abstract
In this paper we present a method of detecting zero pronouns in Japanese clauses andidentifying their antecedents using aligned sentence pairs from a Japanese-English
bilingual corpus and open resource tools. We use syntactic and semantic structures
and the alignment of words and phrases in the sentence pairs to automatically detectzero pronouns and determine their antecedents using English translations. We build
rules to link antecedents with zero pronouns and create filters to remove problematic
sentence pairs. Experimental results confirm the effectiveness of our method. The
proposed method allows the construction of an annotated corpus of zero pronoun
sentences in which the antecedents of the missing pronouns are flagged. This would
be very useful for machine translation (MT), because zero pronoun detection is avital problem when translating languages which allow zero pronouns.

1

Introduction

In many languages, such as Japanese and Chinese, elements which can be easily deduced by the reader are frequently omitted from subsequent expressions in discourses.
Omissions related to obligatory subject and object cases are often referred to as zeropronouns. A zero pronoun can be thought of as a noun phrase which is obligatory but
which is not expressed explicitly. Zero pronoun resolution is more complicated thanovert pronoun resolution because a natural language processing (NLP) system has todetect zero pronouns before it can identify their antecedents. Determination of theantecedents of zero pronouns is vital in many NLP applications, including information
extraction, question answering, machine translation, etc.
Many algorithms have been proposed for pronominal anaphora resolution [9, 2, 10].
However, resolving the anaphora problem is much harder in languages which allow zeropronouns compared to languages which do not allow zero pronouns, such as English,
since detection of zero pronouns should be done before anaphora resolution. Several
methods have been proposed to solve this problem [12, 3, 14]. Some of these methods
use semantic and pragmatic constraints, such as semantic constraints on cases, modal
expressions, or verbal semantic attributes, to determine the referents of zero pronouns[12]. Other methods use a machine learning approach for anaphora resolution, such as
using semantic attributes as computable features to perform identification and resolution

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 66

of anaphoric zero pronouns in Chinese [14] or using syntactic patterns of zero pronouns
and their antecedents as features for zero-anaphora resolution in Japanese [3, 5].
Most of these methods focus on zero pronoun resolution and detection of antecedents of zero pronouns as a crucial part of zero pronoun resolution. All of thesemethods employ either an annotated corpus containing zero pronoun sentences, with
anaphoric relationships tagged by annotators [3, 4], or an annotated corpus provided
by other persons or institutes [14, 1]. There are many ways to construct such an annotated corpus. Most of these corpora are monolingual and are annotated by hand.
Manual annotation of these corpora can be very time consuming and may require relevant knowledge, so in this study we attempt to find a way to automatically detect theantecedents of zero pronouns.
Different languages contain different types of anaphoric expressions. Since similar
languages often contain similar anaphoric expressions, languages which are very different
from each other, such as English and Japanese, generally differ more markedly in regard
to how they use pronouns and how these pronouns are linked to their antecedents.
Thus, if we use a bilingual corpus, we are able to contrast how each language handles
corresponding occurrences of subjects, objects and pronouns in the same sentence, andwe can utilize the translation to identify the antecedents of zero pronouns.
Some research on automated corpus annotation using a bilingual corpus has been
conducted [11]. The purpose of that study was to formulate rules for the anaphoric
resolution of Japanese zero pronoun sentences using aligned sentence pairs. The same
method was also used in another study, in which a valency transfer dictionary whichcontained 16,000 pairs of Japanese case-frame patterns was used for zero pronoun detection [6]. It showed the relationship between corresponding Japanese verbs and theirEnglish translations using different semantic models. For example, ”yomu” in the patterns ”N1(human)-ga N2(abstract thing)-o yomu” should translate as ”N1 read N2” inEnglish, but in the pattern ”N1(subjects)-ga N2(”vote”)-o yomu” should translate as”N1 predicts (the outcome of the vote will be) N2”. This dictionary makes it easy tofind the obligatory elements of Japanese sentences and it can also be used to automatically recognize intransitive verbs. But since there were no open source parsers for this
dictionary, this method of zero pronoun resolution could not be widely implemented.
Several years have passed since this method was first proposed however, and in thattime many useful tools for natural language processing have been created which haveachieved good accuracy in tasks such as morphological analysis, parsing and alignment.
These tools are widely used in the field of NLP and most of them are open source
software. Therefore, in this study we propose a method of automatically annotating
Japanese zero pronouns and their antecedents into a Japanese-English bilingual corpus,
using open source tools.

2

Task Definition

For the purpose of machine translation, it is important to recognize that some grammatical elements not present in one language could be obligatory elements in anotherlanguage. In particular, subjects and objects are often omitted in Japanese, but are
obligatory in English. So it is natural to consider using the differences between Japaneseand English to resolve the issue of missing pronouns in Japanese. A Japanese-English
bilingual corpus could be useful if the antecedents of zero pronouns are difficult to recognize in Japanese sentences but these missing pronouns have equivalent antecedents
in English sentences. For example, subject is zero pronoun in Japanese sentence of example (1) and it has a equivalent antecedent ”I” in English sentence, then zero pronoun

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 67

can be aligned with equivalent antecedent in English.
(1) 私は疲れているとき、本を読みたい。
Literally translated, ”When I am tired, want to read book.” In grammatically correct English, it would be translated as, ”When I am tired, (I) want to read a book.”
The first, bold ”I” is the antecedent, and in this sentence it allows us fill in the ”I” inparenthesis, which is the English equivalent of the missing zero pronoun in Japanese.
So in this study, we propose a method to detect the antecedents of Japanese zeropronouns using English translations of Japanese sentences. This method involves two
steps. First, we need to detect whether or not the original sentence contains zeropronouns, and if it has zero pronouns we want to know where the zero pronouns occur.
Then we try to detect the antecedents in the English translation and decide whichantecedent corresponds to the missing zero pronoun. The translation process may resultin some errors and unsuitable sentences, however, so we have to detect and remove
unsuitable sentence pairs. Zero pronouns can be divided into three types, based on thelocation of the antecedent:
The antecedent of the zero pronoun is located in sentence futher back, mostly in theprevious sentence:(2) 彼はとてもいい⼈です。昨⽇、(ϕ-ga) 私に⼿伝ってくれました。
He is a good person. Yesterday helped me.
The antecedent of the zero pronoun is located in the same sentence:(3) 彼は宿題を終わって、(ϕ-ga) テレビを⾒ました。
He finished homework, watched TV.
The antecedent of the zero pronoun does not exist at all in context:(4)(ϕ-ga) 東京にいきたい。
Want to go to Tokyo.
In this paper we focus on both simple sentences and complex sentences, whichcontain clauses in addition to one simple sentence. But to detect the antecedent of azero pronoun within a clause of a complex sentence or in a previous sentence, we willneed to combine our translation comparison method with other anaphora resolution
methods. For example, our method can identify that the antecedent of the zero pronounin the second clause of sentence (2) is ”he”, but it does not know that ”he” refers to thecharacter “彼”in the previous clause. If we can find the antecedent of a zero pronoun
in an English translation, we can also determine its referent using anaphora resolution
on the translation, and then align the referent and the corresponding Japanese word.
This will allow us to resolve even intra-sentential zero pronoun anaphora. There havebeen many studies on anaphora resolution in English, but in this study we only focus ondetecting the antecedents of zero pronouns in English translations of Japanese sentences.

3

CorpusFor this study, we used the Japanese-English Scientific Paper Excerpt Corpus (ASPECJE)1 , which contains 3,008,500 parallel sentence translations collected from JapaneseEnglish scientific paper abstracts. There are no syntactic or semantic structures associated with the sentences in this corpus, so we need to do segmentation, Part-Of-Speechtagging and parsing. We removed especially long sentences from the corpus (more than150 characters in one sentence) because the syntactic structures of these sentences were
too complex to be used. We removed 486,958 long sentences, and then used the remaining 2,521,542 sentence pairs to train the Japanese to English alignment model. To
examine the effectiveness of this method, only 1,000 sentences were used for a closed
test and 200 sentences for an open test, because this seems to be a reasonable number of
1 http://lotus.kuee.kyoto-u.ac.jp/ASPEC/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 68

sentences which could be evaluated manually. All 1,200 of these sentences were chosenrandomly.

4

Linking zero pronouns and their antecedents by aligning
Japanese-English sentence pairs

Our method can be divided into several steps:
1) morphological analysis of the JapaneseEnglish corpus (MeCab2 [8] for Japanese andStanford segmentation3 for English); 2) alignment of Japanese words and English words(GIZA++4 [13]); 3) syntactic and semantic
analysis of Japanese sentences; 4) syntactic
analysis of English sentences; 5) identification of unsuitable sentence pairs; and 6) linking of Japanese zero pronouns with their antecedents. For Japanese syntactic and semantic analysis, we use two tools, CaboCha5 [7]for syntactic analysis and SynCha6 [5] forJapanese predicate-argument structure analysis. These tools allow us to recognize whichpart of a sentence is the subject, object orpredicate and to identify the dependency relationships between word phrases. For English syntactic analysis, the Stanford parser7
is used. There are two types of output;
context-free, phrase structure grammar representations and universal dependencies. We
Figure 1: Overview of the proposed
use the latter to build syntactic relationshipmethodstructures. Overview of the process is shownin Figure 1.
4.1

Identification of antecedent candidates in English sentences

Before linking Japanese zero pronouns with their antecedents as indicated in English
translations, we need to specify a range of antecedent candidates. Anaphoric expressions
are common in English, and there is a set of special identity words which tend tooften appear as antecedents. In order to not miss these common antecedents, we must
consider not only definite nouns and noun phrases, such as ”the company”, but also theseidentifying words, as possible translation equivalents. These identifying words include
personal pronouns (I, you, he, she, they, we, it); impersonal words (one, everyone, noone, men, women, human beings, people) and demonstratives (this, that, these, those,each, every).
2 http://taku910.github.io/mecab/
3 http://nlp.stanford.edu/software/tokenizer.shtml
4 http://www.statmt.org/moses/giza/GIZA++.html. We use alignments of translations in both directions ( from Japanese sentences to English sentences and vice versa) and combine them for the finalalignment. This means Japanese words or phrases and their English counterparts are only linked ifthey are aligned in the translation results in both directions.
5 http://taku910.github.io/cabocha/
6 http://www.cl.cs.titech.ac.jp/ ryu-i/syncha/
7 http://nlp.stanford.edu/software/lex-parser.shtml

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 69

4.2

Identification of unsuitable sentence pairs

There are situations where the aligned sentence pairs are not suitable for annotating
antecedents of zero pronoun. These sentences need to be identified and moved out. For
example, when English parser was unable to make a full syntactic structure, we cannot
use it for identification of antecedent candidates. Considering the accuracy rate of tools
we used and the quality of the bilingual corpus, aligned sentence pairs are identified as
unsuitable if they have the problems caused by corpus, such as freely translation, or
errors caused by tools, like parser and alignment errors. For identifying these unsuitablesentences, we have made some rules to filter these sentences out automatically after
having analyzed the result of original rules. The details of these filters will be explained
in section 5.3.
4.3

Zero pronoun detection

Zero pronoun sentences occur frequently in Japanese and zero pronouns usually appear
where either the grammatical subject or object would appear. Moreover, understanding
the referent of the zero pronouns in these positions is vital for sentence comprehension.
Hence, we focus on zero pronouns which appear as subjects and objects. Normally, a
subject is an essential component of a complete sentence, unless the sentence is formulated in a passive voice (”The mailman was attacked.”) or in the imperative mood(”Attack the mailman!”). If a Japanese sentence does not have a subject, and the sentence is not in the passive voice or imperative mood, we assume there is a zero pronounin subject position. Regarding objects, our assumption is based on whether the predicate contains a transitive or intransitive verb. If the predicate of a Japanese sentence
contains a transitive verb but there is no object, then it is assumed that there must
be a zero pronoun in object position. These assumptions are the basis of zero pronoun
detection in Japanese sentences. However, we are using an open source parser whichcannot recognize intransitive verbs in Japanese sentences. Furthermore, the same verbs
tend to be transitive or intransitive in English as in Japanese. But with the help of an
English translation, we can recognize intransitive verbs in Japanese sentences by noting
if the corresponding English verb in the sentence has no direct object.
4.4

Linking Japanese zero pronouns with their antecedents

Now that we have ways to filter out unsuitable sentences and to identify zero pronouns,
we need a way to link an antecedent which appears in an English sentence to a zeropronoun in a Japanese sentence. Nakaiwa proposed ten rules for this purpose (1999),
and we used these rules as our starting point. However, based on an analysis of theresults and the features of the corpus, we decided to formulate new rules to improve
performance. The ten rules proposed in Nakaiwa’s paper can be described as follows:
Rule 1: Alignment between subjects
Rule 2: Alignment between objects
Rule 3: Alignment of Japanese subjects and English possessive pronouns
Rule 4: Unalignment of Japanese subject when the passive voice is used in English
Rule 5: Non-alignment of Japanese subjects, and alignment of Japanese objects withEnglish subjects, when a passive voice is used in English
Rule 6: Alignment of both subjects and objects
Rule 7: First default alignment rule for remaining unaligned zero pronoun
Rule 8: Second default alignment rule for remaining unaligned zero pronouns
Rule 9: Default unalignment rule for unaligned zero pronouns
Rule 10: Identification of intra-sentential antecedents of any aligned zero pronouns

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 70

These rules are applied from 1 to 10 in numeric order. If a rule is satisfied for azero pronoun, the process will stop. Each rule has its own application conditions, anda rule is applied only if all of the conditions for the rule are satisfied. Details of Rule4, Rule 5, Rule 7 and Rule 9 are shown below, since the results of these rules will beanalyzed in the following section:
Rule 4: Non-alignment of Japanese subjects when a passive voice is used in English
IF Sj of Pj is an unaligned ϕ & Oj of Pj is alignedwith Se of USe & Pe of USe is in a passive voice
THEN there are no antecedents of Sj because a passive voice is used in the English translation: unalignable
For example, in sentence pair (5), the process forRule 4 is as follows:(5) (ϕ-ga) 内部リレーの使い⽅を説明した。
The usage of the inside relay was explained .
Oj: 使い⽅ <-> Se: usage & Pe: ”was explained” isa passive voice => Sj: ϕ -ga = unalignable

Figure 2: Rule4

Rule 5: Non-alignment of Japanese subject, and alignment of Japanese object withEnglish subject, when a passive voice is used in English
IF Sj of Pj is an unaligned ϕ & Oj of Pj is an unaligned ϕ & Pe of USe is in a passive voice & Se of
USe is an antecedent candidate Ai
THEN there are no antecedents of Sj because a passive voice is used in the English translation: unalignable & align ϕ-Oj with Se(6) (ϕ-ga)(ϕ-o) 外⽤剤で治療した。
It was treated by the external preparation .

Figure 3: Rule5

Pe: ”was treated” is a passive voice & Se: it is anantecedent candidate Ai => Sj: ϕ-ga = unalignable & Oj: ϕ-o <-> Se: it
Rule 7: Default alignment rule for remaining unaligned zero pronouns
IF there is only one unaligned zero pronoun case in Japanese and there are one or moreunaligned candidate antecedents
THEN we determine the antecedent of zero pronoun based on the following priority:
personal pronoun > one > demonstratives > definite NP
Rule 9: Default non-alignment rule for unaligned zero pronouns
IF there are any remaining cases of unaligned zero pronouns in Japanese
THEN these are determined to be ϕ-Cj’s whose antecedents are not explicitly translated
in USe and are marked as unresolved

5

Experiment

5.1

Evaluation

We evaluated the proposed method using aligned sentence pairs from the ASPEC-JE
corpus. A closed test was conducted to evaluate the performance of the rules proposed by[11]. After analyzing the results, we added several filters to remove unsuitable sentencepairs with errors caused by processing tools or which had inaccurate translations. We
then added additional rules to detect the antecedents of zero pronouns. A closed test

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 71

was again performed on the same data sets, and the process was repeated several times.
Finally, an open test was conducted on unseen data to evaluate the effectiveness of the
proposed method with an unknown corpus.
5.2

Closed test of Nakaiwa method

We first evaluated the effectiveness of Nakaiwa’s original 10 rules for detecting of zeropronoun antecedents. Before the evaluation, sentences without zero pronouns should be
removed, and according to the parsing results, 95 sentences without zero pronouns were
removed automatically, since these types of sentences are not our target. The remaining
sentences were then examined using Nakaiwa’s original ten rules. Some rules, such as
Rule 5, Rule 6 and Rule 9, determine the alignment or non-alignment of zero pronouns
more than once at the same time. Rule 5 and Rule 6 process zero pronouns in boththe subject and object positions of the same sentence, for example. Rule 9 identifies
sentences which are unalignable. In order to evaluate our results, we had to check theoutput manually. The results are shown in Table 1.
RuleTimes rule applied correctly/Times rule applied (Accuracy)

1
2
3
4
5*
6*
7
8
9
10
Totals95/96 (99.0%)
10/11 (90.9%)
0/0 (-)
333/333 (100%)
17/104 (16.3%)
1/2 (50%)
28/59 (47.5%)
9/12 (75.0%)85/424 (20%)
0/0 (-)
578/1041 (55.5%)

Number of zero pronouns determined andlinked correctly/Number of zero pronouns determined (Accuracy)
95/96 (99.0%)
10/11 (90.9%)
0/0 (-)
333/333 (100%)
121/208 (58.2%)
2/4 (50%)
28/59 (47.5%)
9/12 (75.0%)85/502 (16.9%)
0/0 (-)
683/1225 (55.8%)

Table 1: Results using Nakaiwa’s original rules* Correct results means both subject and object zero pronouns were correctly determined.

5.3

Analysis of results

From these results we can see that only 55.5% of applied rules determined zero pronouns
and their antecedents correctly, and that the overall accuracy rate of zero pronoun andits antecedent determination was 55.8%. Rule 1, for determining zero pronouns in thesubject position, achieved a precision rate of 99.0%,with the one incorrect result caused
by a Japanese parser error. Rule 2, for determination of zero pronouns in the object
position, achieved 90.9% precision. Rule 4, for detection of passive voices, achieved
100% precision. These results show that the original method is successful for many
sentences. However, there were no or very limited cases in which Rule 3, Rule 6 andRule 10 could be applied.
Rule 5, for both detection of zero pronouns in the object position and detection ofpassive voices, achieved only 58.2% precision for zero pronoun determination and Rule
9 achieved very poor precision (16.9%) for zero pronoun determination, which differs
markedly from Nakaiwa’s own results (1999), in which accuracy rates of 92.9% and44.8% for Rule 5 and Rule 9 were achieved, respectively. After checking the results forRule 5, we found that in most cases the English translations were in a passive voice.
These cases should have been processed by Rule 4, but went unrecognized because there
are no objects of Japanese sentences which do not satisfy the condition of Rule 4. As

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 72

for Rule 9, its poor performance was due to its inability to detect intransitive verbs.
Thus, we clearly need to adjust the rules in order to improve accuracy.
We also want to determine the recall of our method. To do this, we need to know theactual number of zero pronouns in the sentences used in the closed test. As mentioned
in section 4.3, we can determine whether zero pronouns exist or not based on whether
subjects or objects exist in the sentences. The number of zero pronouns in the subject
position is counted automatically, but since our system cannot determine whether aJapanese verb is transitive or intransitive automatically, we need to check the numberof zero pronouns in the object position manually. Based on whether or not the subject
or object is missing, all of the sentences in the closed test can be divided into one offour types:
Type of sentence (automatically determined)

No zero pronoun
Subject is missing, object exists
Subject is missing, object unknown
Subject exists, object unknown

Number of simple sentences

Zero pronouns
in subject position
0
490
231
0

95
490
231
278

Number of zeropronouns in object position
0
0
33
8

Table 2: Distribution of zero pronouns
The total number of zero pronouns in the closed test corpus can be calculated asfollows: 490+231+33+8 = 762. We can then calculate the recall of the result: 676/762= 88.7%. Since there were many incorrect determinations of zero pronouns in our
experiment, we tried to analyze these errors to improve accuracy. In our analysis ofincorrect determinations of default results for Rule 5 (87 cases), Rule 7 (31 cases) andRule 9 (339 cases), we found the following types of errors:(A) There is no explicit antecedent in the English sentence due to faulty translation.
(B) Japanese parser error. Syntactic and semantic analysis may have caused errors in
detecting semantic relationships.
(C) English parser error, such as syntactic and semantic analysis errors.
(D) Alignment error. Words that should be aligned cannot be found or were mismatched.
(E) No antecedent candidates.
(F) Cases where Japanese verbs were translated into continuous tenses or as gerunds
in English, leading to incorrect predicate classification.
(G) Incorrectly detected zero pronouns in the object position.
(H) Intransitive verbs which act as predicates in Japanese sentences. In such cases,
there is no object in the sentence but the system cannot detect this automatically.
This is caused by the absence of an open source tool to determine if a Japaneseverb is transitive or intransitive.
Rule5
Rule7
Rule9

Total
87
31
339

A
0
15
7

B
0
3
11

C
0
4
4

D
0
0
8

E
0
0
1

F
0
9
6

G
87
0
0

H
0
0
302

Table 3: Distribution of errors by type of error for Rule 5, Rule 7 and Rule 9
The statistical results are shown below in Table 3. According to the results, allof the Rule 5 errors were caused by incorrect detection of zero pronouns in the object
position, and most of the Rule 9 errors were caused by faulty intransitive verb detection.
Poor translations (type A) and parser errors (Type B and Type C) were responsible

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 73

for many of the errors. By analyzing these errors, we can adjust the rules to improve
performance.
5.4

Modification of original rules

We found that all of the Rule 5 errors were caused by false detection of zero pronouns in
the object position which did not exist, and that most of these sentences were translated
into a passive voice. Such sentences should have been processed using Rule 4, but sincethese sentences do not have objects, they do not satisfy the conditions of Rule 4. Thisis the result of a syntactic difference between Japanese and English.
(9) 標記課題について検討した。
The problem with the title was examined .
For example, in sentence pair (9), the corresponding English translation is in a
passive voice, and the noun phrase before “について”should be the object of the
English sentence (“the problem with the title”). In Japanese “について”often
connects noun phrases and predicates using the pattern “noun + について + verb”.
The noun acts as an object in this case, but the Japanese parser cannot recognize it.
So we developed a new rule to handle these situations, hoping this would improve ourresults. This new rule is not for the detection of antecedents of zero pronouns, but isinstead a complement of tools to recognize whether there is a zero pronoun in the object
position.
Rule11: Detection of zero pronouns as objects
IF Oj of Sj are unaligned ϕ & Cj in Noun+ について modifies the predicate
THEN replace ϕ-Oj with Cj(10) 標記課題について検討した。
The problem with the title was examined .
Oj: ϕ-o & Nj (標記課題) について exists & Nj (標記課題) modifies 検討した。=>
Replace ϕ-o with Nj (標記課題)
RuleTimes rule applied correctly/Times rule applied (Accuracy)

1
2
3
4
5
6
7
8
9
10
Totals95/96 (99.0%)
10/11 (90.9%)
0/0 (-)
471/471 (100%)
17/17 (100%)0/1 (0%)35/64 (54.7%)
4/5 (80.0%)61/360 (16.8%)
0/0 (-)
693/1025 (67.6%)

Number of zero pronouns determined andlinked correctly /Number of zero pronouns determined (Accuracy)
95/96 (99.0%)
10/11 (90.9%)
0/0 (-)
471/471 (100%)
34/34 (100%)0/1 (0%)35/64 (54.7%)
4/5 (80.0%)61/371 (16.4%)
0/0 (-)
710/1053 (67.4%)

Table 4: Result of method with added rule
We insert this new rule bewteen Rule 2 and Rule 3 (not showed in Table 4). We
repeated the experiment. The results of this experiment are shown in Table 4. We cansee that the total number of sentences decreased from 1,041 to 1,025. This is because
in the 16 sentences in which a subject existed but an object did not exist, the object
was added by applying Rule 11, so that no zero pronouns remain in these sentences.
Thus, they will not be processed by the other rules and will not be counted in the total
number of detected zero pronoun sentences. A total of 173 sentences were processedby Rule 11. The number of sentences processed by Rule 4 increased from 333 to 471,
and the largest increase in processing occurred with Rule 5 (87 sentences), even though
there are sentences which Rule 5 cannot be applied to, because they do not satisfy theProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 74

condition of having antecedents in English, in which case the default rules are applied(51 cases). These sentences are now processed by Rule 4 after Rule 11 was added. As
a result, sentences being processed by Rule 5 and by the default rules have decreased.
We can also see that the total precision of zero pronoun determination increased from
61.6% to 67.4% and that recall increased from 88.7% to 93.2% (710/762).
5.5

Importation of filters

From our analysis of Table 3, we could see that many errors originated before processing,such as errors caused by incorrect translations. It is hard to avoid these types of errors,
but we can add filters to detect problematic sentences and remove them in advance.
In some cases, part of the original Japanese sentence is translated into a nounphrase or gerund. Sentence pairs which had different numbers of unit sentences were
rejected as unsuitable in Nakaiwa’s research (1999). We use a similar rule, but weonly filter out unsuitable sentence pairs when English translation contains fewer unit
sentences than the Japanese original so that if the English translation contains moreunit sentences than the Japanese original, we still try to find the antecedent of a zeropronoun by trying to align the zero pronoun with possible antecedents in each of theunit sentences.
Filter 1: Filter for sentence pairs when English translation contains fewer unit sentences
than the Japanese original. If English translation contains fewer unit sentences thanthe Japanese original, then mark the sentence pair as unsuitable.
(11) 柔軟性を重視してＪＡＶＡを⽤いた。JAVA was used because of its flexibility.
Pj: 2 unit sentences & Pe: 1 unit sentence => unsuitable
A large number of errors were caused by faulty detection of intransitive verbs,
described as Type H errors in Table 3, thus we built a filter to determine if there is azero pronoun in the object position.
Filter 2: Filter for zero pronoun in the object position with intransitive verb
If there is no object in the English sentence, and a subject and predicate exist in theJapanese sentence which are aligned with the English predicate, and no unaligned words
remain, then there is no zero pronoun in object position.
(12) 画素構造の改造が（ϕ-o）進んでいる。Modification of the pixel structure is progressing .
Pj: 進んでいる <-> Pe: has advanced .& Sj: 改造がexist & object not exist in English sentence => Sj:
Obj is not zero pronoun
Filter 3 is used to detect errors caused by thelimitations of Japanese language analysis tools. For
example, in sentence(13), the object“事実”(”fact”)
does exist in the Japanese sentence, but the Japaneseparser cannot determine that the word before “は”is an object.

Figure 4: Filter2

Filter 3: Filter for non-alignment of zero pronouns in the object position
If subject, object and predicate all exist in the English sentence and are aligned withJapanese words, and no words are remain to serve as antecedents of the object zeropronoun, then there is no zero pronoun in object position.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 75

(13) この事実は著者が既にこのシリーズで (ϕ-ga)
論じてきた。
The author has already discussed
this fact in this series . Pj: 論じてきた <-> Pe: hasalready discussed & Sj: 著者が <-> Se: The author& Oe: fact <-> other words in Japanese & no object
left in English => unsuitable sentence pair &Sj: Obj
is not a zero pronoun

Figure 5: Filter3

Filter 4: Detecting mismatched sentence pairs. In acase where both the Japanese and English sentences contain sub-sentences, if the predicate and subject of the Japanese sub-sentences are aligned with elements of different
sub-sentences in the English sentence, this means the sentence pairs are mismatched.
If this occurs, we can reject the sentence pair as unsuitable and avoid searching forantecedents for zero pronouns.
FilterNumber of times filters were applied
Number of zero pronouns detected using each filter

1
8
18

2
214
214

3
34
34

4
1
2

Total
256
267

Table 5: Number of sentence pairs processed and zero pronouns detected by each filter
These filters were applied in numerical order from Filter 1 to Filter 4. By applying
these filters before applying our rules, we were able to remove some unsuitable sentences
successfully, as shown in Table 5. We then reapplied our modified rules, and our new
results are shown in Table 6. Due to filtering, the number of rules which were applied
during processing was reduced, while also achieving a large increase in the precisionof zero pronoun determination (from 67.4% to 92.2%). Recall remains unchanged at
93.2% because the number of correctly determined zero pronouns was unchanged.
Rule

1
2
3
4
5
6
7
8
9
10
TotalsWith added ruleRules applied corZero pronouns derectly/Rules applied
termined and linked(Accuracy)
correctly/Zero pronouns
determined(Accuracy)
95/96 (99.0%)
95/96 (99.0%)
10/11 (90.9%)
10/11 (90.9%)
0/0(-)
0/0 (-)
471/471 (100%)
471/471 (100%)
17/17 (100%)
34/34 (100%)0/1 (0%)0/1 (0%)35/64 (54.7%)
35/64 (54.7%)
4/5 (80.0%)
4/5 (80.0%)61/360 (16.9%)
61/371 (16.4%)
0/0 (-)
0/0 (-)
693/1025 (67.6%)
710/1053 (67.4%)

With added rule and filters
Rules applied corZero pronouns derectly/Rules applied
termined and linked(Accuracy)
correctly/Zero pronouns
determined(Accuracy)
95/96 (99.0%)
95/96 (99.0%)
10/11 (90.9%)
10/11 (90.9%)
0/0(-)
0/0 (-)
471/471 (100%)
471/471 (100%)
17/17 (100%)
34/34 (100%)0/1 (0%)0/1 (0%)35/59 (59.3%)
35/59 (59.3%)
4/5 (80.0%)
4/5 (80.0%)61/83 (73.5%)
61/93 (65.6%)
0/0 (-)
0/0 (-)
693/743 (93.3%)
710/770 (92.2%)

Table 6: Results with added rule vs. results with added rule and filters
In Table 7, we analyze the distribution of different types of zero pronoun antecedents(based on the results of the added rule and filters, and only including correctly identified
antecedent results). The majority of zero pronoun sentences are sentences written in a
passive voice, which is due to the nature of the corpus. However, we can also see the
distribution of explicit antecedents and specific nouns, the latter of which are especiallyvaluable for accurately translating Japanese sentences into English.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 76

Rule
1
2
4
5
7
8
TotalsNumber of zero pronoun
95
10
471
34
35
4
632

They/they
27
0
0
0
1
0
28

It/it
20
0
0
7
7
1
35

I
0
0
0
0
1
0
1

We/we
11
0
0
0
4
0
15

noun
37
10
0
10
22
3
82

Passive voice
0
0
471
17
0
0
488

Table 7: Distribution of zero pronoun antecedents determined using selected rules
5.6Open test

We then tested the proposed method using 200 unknown sentence pairs. We conducted
the test twice, using Nakaiwa’s original rules and using our new rule with the added
filters.
Rule

1
2
3
4
5
6
7
8
9
10
TotalsOriginal rules
Rules applied corZero pronouns derectly/Rules applied
termined and linked(Accuracy)
correctly/Zero pronouns
determined(Accuracy)
15/16 (93.8%)
15/16 (93.8%)
3/3 (100.0%)
3/3 (100.0%)
0/0 (-)
0/0 (-)
53/53 (100.0%)53/53 (100.0%)
1/12 (8.3%)
23/24 (95.8%)
0/0 (-)
0/0 (-)
3/11 (27.3%)
3/11 (27.3%)
1/3 (33.3%)
1/3 (33.3%)
9/105 (8.6%)
9/123 (7.3%)
0/0 (-)
0/0 (-)
95/203 (46.8%)
107/233 (45.9%)

Added rules with filters
Rules applied corZero pronouns derectly/Rules applied
termined and linked(Accuracy)
correctly/Zero pronouns
determined(Accuracy)
15/15 (100.0%)
15/15 (100.0%)
3/3 (100.0%)
3/3 (100.0%)
0/0 (-)
0/0 (-)
71/71 (100.0%)71/71 (100.0%)
4/5 (80.0%)
8/10 (80.0%)
0/0 (-)
0/0 (-)
4/10 (40.0%)
4/10 (40.0%)
1/2 (50%)
1/2 (50%)
9/16 (56.2%)
9/19 (47.4%)
0/0 (-)
0/0 (-)
107/122 (87.7%)
111/130 (85.4%)

Table 8: Open test results: using original rules and using added rule with filters
The filters removed 67 unsuitable sentences containing 69 zero pronouns. To calculate recall, we counted the number of zero pronouns, as in Table 2. The total number ofzero pronouns in each category was 129. Using the original rules, the precision of zeropronoun determination was 45.9% and recall was 107/129 = 82.9%. Using the added
rule with filters, the precision of zero pronoun determination was 85.4% and recall was
111/129 = 86.0%. From these results, we can see that the improved method achieved
higher precision and recall compared to the original method. Furthermore, our method
was shown to be effective for detecting and resolving zero pronouns of Japanese sentences in unknown corpus. By applying this method to many aligned sentence pairs,
it will be possible to collect large amounts of data on recognition of antecedents ofzero pronouns, and that data can then be used in many natural language processing
applications.

6

Conclusion

This paper proposes a method to detect and identify missing pronouns in written
Japanese so they can be inserted into English translations. The proposed method usesaligned sentence pairs from a Japanese-English bilingual corpus and open source tools.
Experimental results showed that our proposed method achieved a rule application accuracy rate of 93.3% for all sentence pairs and a 92.2% precision rate for zero pronoun

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 77

identification in a closed test (Table 6). We also achieved a rule application accuracy
rate of 87.7% and a zero pronoun identification rate of 85.4% in an open test (Table 8). The effectiveness of this method will allow us to automatically construct anannotated, Japanese-English corpus which links zero pronouns in Japanese with theirantecedents, which will greatly improve the accuracy of machine translations. In the
future, we plan to combine our method with a machine learning technique to investigate
Japanese anaphora resolution. We also plan to apply our method to Japanese-English
bilingual corpora with other genres, allowing us to examine the full range of zero pronoun phenomena, and to Japanese-Chinese bilingual corpora, allowing us to examine
the effectiveness of the proposed method with a bilingual corpus in which both languages have many zero pronouns. The effect of using different syntactic and semantic
parsers will also be examined.

7

Acknowledgements

This research was carried out while Zhan Dong was an exchange student at Nagoya
University, from September 2014 to July 2015. We would like to thank Prof. Kazuya
Takeda for his helpful comments regarding this research.

References[1] Susan P Converse. Pronominal anaphora resolution in Chinese. 2006.
[2] Niyu Ge, John Hale, and Eugene Charniak. A statistical approach to anaphora resolution.
In Proceedings of the sixth workshop on very large corpora, volume 71, 1998.
[3] Ryu Iida, Kentaro Inui, and Yuji Matsumoto. Exploiting syntactic patterns as clues inzero-anaphora resolution. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational
Linguistics, pages 625–632. Association for Computational Linguistics, 2006.
[4] Ryu Iida, Mamoru Komachi, Kentaro Inui, and Yuji Matsumoto. Annotating a Japanese
text corpus with predicate-argument and coreference relations. In Proceedings of the Linguistic Annotation Workshop, pages 132–139. Association for Computational Linguistics,
2007.
[5] Ryu Iida and Massimo Poesio. A cross-lingual ILP solution to zero anaphora resolution. In
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:
Human Language Technologies-Volume 1, pages 804–813. Association for Computational
Linguistics, 2011.
[6] Satoru Ikehara, Masahiro Miyazaki, Akio Yokoo, Satoshi Shirai, Hiromi Nakaiwa, Kentaro
Ogura, Yoshifumi Ooyama, and Yoshihiko Hayashi. Nihongo goi taikei–a Japanese lexicon.
Iwanami Shoten, 5, 1997.
[7] Taku Kudo and Yuji Matsumoto. Fast methods for kernel-based text analysis. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1,
pages 24–31. Association for Computational Linguistics, 2003.
[8] Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. Applying conditional random fields
to Japanese morphological analysis. In EMNLP, volume 4, pages 230–237, 2004.
[9] Shalom Lappin and Herbert J Leass. An algorithm for pronominal anaphora resolution.
Computational linguistics, 20(4):535–561, 1994.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 78

[10] Ruslan Mitkov. Anaphora resolution. Routledge, 2014.
[11] Hiromi Nakaiwa. Automatic extraction of rules for anaphora resolution of Japanese zeropronouns in Japanese–English machine translation from aligned sentence pairs. Machinetranslation, 14(3-4):247–279, 1999.
[12] Hiromi Nakaiwa and Satoshi Shirai. Anaphora resolution of Japanese zero pronouns with
deictic reference. In Proceedings of the 16th conference on Computational linguisticsVolume 2, pages 812–817. Association for Computational Linguistics, 1996.
[13] Franz Josef Och and Hermann Ney. Giza++: Training of statistical translation models,
2000.
[14] Shanheng Zhao and Hwee Tou Ng. Identification and resolution of Chinese zero pronouns:
A machine learning approach. In EMNLP-CoNLL, volume 2007, pages 541–550, 2007.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 79

METEOR for Multiple Target Languages
using DBnary.
Zied Elloumi
Hervé Blanchon
Gilles Serasset
Laurent Besacier
LIG - GETALP - Univ. Grenoble Alpes

zied.elloumi@imag.fr
herve.blanchon@imag.fr
gilles.serasset@imag.fr
laurent.besacier@imag.fr

Abstract
This paper proposes an extension of METEOR, a well-known MT evaluation metric, formultiple target languages using an in-house lexical resource called DBnary (an extraction
from Wiktionary provided to the community as a Multilingual Lexical Linked Open Data).
Today, the use of the synonymy module of METEOR is only exploited when English is thetarget language (use of WordNet). A synonymy module using DBnary would allow its use
for the 21 languages (covered up to now) as target languages. The code of this new instanceof METEOR, adapted to several target languages, is provided to the community. We also
show that our DBnary augmented METEOR increases the correlation with humanjudgements on the WMT 2013 and 2014 metrics dataset for English-to-(French, Russian,German, Spanish) language pairs.

1. Introduction
Machine translation (MT) is the process of automatically translating a text in a source
language into a corresponding text in a target language. In order to evaluate and compare thequality of several MT systems, we need to rate the translation hypothesis produced by each
MT system either with the help of human experts (subjective evaluation) or compare it to preexisting human translations (using automatic evaluation metrics, objective evaluation). In
practice, subjective evaluation considers various aspects to grade the translation quality, suchas adequacy, fluency, and intelligibility. However, subjective evaluation conducted aposteriori often costs too much (in term of human resources) and, thus, objective evaluation
metrics (fast and cheap as long as references are available) are often preferred nowadays.
One drawback with automatic evaluation metrics is that they compare the MT
hypothesis with few (and sometimes only one) reference translations. This is definitely notenough to capture lexical variation in translation. For this reason, metrics which exploit
synonymy or stem similarities, such as METEOR (Banerjee and Lavie, 2005), exhibit better
correlation with human judgement. METEOR maps words with the same stem or the same
synset using lexico-semantic resources. However, so far, the full potential of METEOR is only
exploited when English is the target language (use of WordNet).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 80

Contribution This paper proposes an extension of METEOR for multiple target languages
using a lexical resource called DBnary (Sérasset, 2015). DBnary is an extraction in RDF of thelexical data of multiple editions of Wiktionary. It has several millions of triples describing
lexical entries of the extracted languages, and more than 4.6 million translations from 21
languages to more than 1500 target languages. The modified code allowing to call METEOR
for new target languages (French, Russian, German, Spanish) is made available to the researchcommunity. More target languages (today 21 in total) could be plugged very quickly byinterested users using the same lexical resource (DBnary notably includes Bulgarian, Dutch,
English, Finnish, French, German, (Modern) Greek, Indonesian, Italian, Japanese, Latin,
Lithuanian, Malagasy, Norwegian, Polish, Portuguese, Russian, Serbo-Croat, Spanish,
Swedish, Turkish). We also present initial experiments on the WMT 2013 and 2014 metrics
dataset and show that our new METEOR slightly increases correlation with human judgmentsof translation quality, for language pairs with a target language different than English.

2. State of the art
Since METEOR was first introduced in 2005, it has been improved and extended to include
more features and accommodate more languages for a subset of its features.
2.1.

METEOR: the basics

Banerjee and Lavie (2005) introduced METEOR to overcome several weakness of BLEU
(Papineni, 2002) and NIST (Doddington, 2002) they identified as: the lack of recall, an
indirect only measure of level of grammatical wellformedness, the lack of explicit wordmatching between translation and reference, and the use of geometric averaging of n-grams.
The goal of METEOR was to aim for better correlation with human judgments oftranslation quality using not only word-to-word alignment between the translation hypothesis
and the reference translation(s). The alignment is incrementally produced by a three-leveled
mapping approach between the hypothesis and the reference, using additional resources if
needed: exact match of the surface forms of the words, exact match of the computed stems of
the words, synonymy overlap through shared WordNet “synset” of the words. The second
mapping level uses a stemmer and the third mapping level uses Enlgish WordNet.
While no free WordNets are available for languages such as French, Spanish or
German, current implementation of METEOR for such languages do not support the third
mapping level.
2.2.

METEOR: the recent extensions

METEOR-NEXT (Denkowski and Lavie, 2010a), was introduced to better correlate withhuman-targeted Translation Edit Rate (HTER) (Snover et al., 2006), a semi-automatic postediting based metric which measures the distance between a MT hypothesis and its post-edited
version. The goal was to go beyond a strictly world-level metric with a new aligner supporting
phrases (multi-word) matches. Thus, a fourth mapping level was added to implement this newfeature using a paraphrase database. For English, the database was developed by Snover(2009a). Later, Denkowski and Lavie (2010b), released paraphrase databases for Czech,
German, Spanish and French.
In 2014, METEOR Universal was released (Denkowski and Lavie 2014) that enabled
the construction of the paraphrase database using only the parallel corpora used to develop the
MT system (which was not the case in 2010).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 81

In order to prevent synonyms/paraphrases corresponding to different senses to betreated as semantically equivalent, Apidianaki and Marie (2015) proposed METEOR-WSD.
The English references are further disambiguated and annotated using Babelfly (Moro et al.,2014) for several language pairs (French, Hindi, German, Czech and Russian to English). For
their experiment, Apidianaki and Marie (2015) got a better segment-level Kendall’s τcorrelation than METEOR for 4 language pairs when the paraphrase module was activated.
2.3.

Lexical resources

2.3.1.

WordNetWordNet is a large lexical database for English, developed by linguists of Princeton University(Fellbaum, 1998). Nowadays, it has become an important and a very useful resource for NLP
applications, such as machine translation, word sense disambiguation, cross-lingual
information retrieval etc. WordNet links nouns, verbs, adjectives and adverbs to sets ofcognitive synonyms (called synsets), where each synset represent a specific concept. Synsets
are interconnected through conceptual semantic and lexical relations, including synonymy,
antonymy, hyponymy etc. Note that words with multiple meanings belong to several synsets,
and their senses are arranged by order of frequency. There are different versions of WordNet inlanguages other than English, such as Arabic WordNet, French WordNet, etc. However, theselexical resources in other languages are not freely available. As already said, METEOR uses
WordNet to increase the chance of the MT output words to match the reference words.
The latest version of WordNet 3.0, contains in total 117 659 synsets: 82 115 noun
synsets, 13 767 verb synsets, 18 156 adjective synsets and 3621 adverb synsets.
To Lemmatize forms, METEOR uses the Morphy-7WN1 function included in WordNet.
This function uses a two-step process to find lemma of a particular word W. Firstly, Morphy
checks for exceptions in a list (containing morphological transformations that are not regular).
If W is not in exception list, Morphy uses the rules of detachment for NOUN, VERB and ADJ
categories (no rules applied to ADV). After each transformation, WordNet is searched for the
resulting string in the syntactic category specified.

2.3.2.

DBnaryDBnary is a multilingual lexical resource in RDF (Klyne & Carroll, 2004) collected at LIG(Sérasset, 2015). The lexical data is represented using standard vocabularies. The lexicon
structure is defined using the LEMON vocabulary (McCrae et al., 2011). Most parts of speech
informations are mapped to the Lexinfo or OliA standard vocabularies (Cimiano et al. 2011,
Hellman et al. 2015), making it highly reusable in many contexts. It is available either as a setof downloadable files or as Linked Open Data directly accessible to browsers or applications.
It may also be queried online using a public SPARQL endpoint.
The available lexical data is automatically extracted from 21 different language
editions2 of Wiktionary, the dictionary counterpart of Wikipedia. Among available lexical data,
one may find 2.9M lexical entries (with parts-of-speech, canonical form for all of them, alongwith pronunciations when available and inflected forms for some languages). Lexical entries
are subdivided into 2.5M lexical senses (with their definitions and some usage example).
1 MORPHY(7WN) manual page : https://wordnet.princeton.edu/man/morphy.7WN.html
2 Bulgarian, Dutch, English, Finnish, French, German, (Modern) Greek, Indonesian, Italian, Japanese, Latin,
Lithuanian, Malagasy, Norwegian, Polish, Portuguese, Russian, Serbo Croat, Spanish, Swedish, Turkish

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 82

DBnary also contains more than 4.6M translations going from the 21 extracted sources
languages to more than 1500 different target languages. Additionaly, DBnary contains lexicosemantic relations (syno/antonymy, hypo/hypernymy and mero/holonymy and troponymy).
Table 1 shows the size of the data for languages involved in the experiments later
reported in this paper. Additional figures are available on the DBnary public web site3.
EnglishFrenchRussianGermanSpanish# of entries

620,369

322,018

185,910

104,505

86,388# of senses

498,451

416,323

176,335

116,290

126,411# of synonyms

35,43736,01931,34533,282

21,024

Table 1. Number of entries, senses, and lexico-semantic relations in DBnary for the target
languages considered in this study.

3.

METEOR-DBnary for multiple target languages

The principal goal of this study is to propose an evaluation metric that uses synonyms in orderto improve MT evaluation for target languages other than English.

3.1.

Resources prepared

As mentioned in the section 1, METEOR package uses the synset dictionary of WordNet,
which is a rich resource of 147 306 unique synsets belonging to four categories (nouns, verbs,
adjectives and adverbs) for English. To gather new external resources for our augmented
METEOR, we downloaded and installed Dbnary dataset 4 and set up a virtuoso-opensource5
server in order to interrogate Dbnary locally. Then, we launched SPARQL queries on DBnaryin order to extract every synonymy relations in the database for English, French, Russian,German and Spanish. The result of the extraction is in the format of: lemma -> synonym. Next,
we performed a processing to match the same format as WordNet (which is alreadycompatible with METEOR). This treatment is to assign an ID for each lemma and to build alist of synonyms for each lemma under format ID eg lemma -> ID_syn1 ID_Syn2 ID_Syn3.
METEOR computes its scores using WordNet as an external lexical resource. In order
to measure the difference between the use of DBnary synonyms and the use of WordNet
synonyms, we built two English synonym dictionaries extracted automatically from Dbnary:
one that contains the same four categories available in WordNet (called METEOR-DB-4-catg)
and another containing all the existing categories in DBnary for English (called METEORDB-All-catg).

3 data, docs and examples are available at http://kaiko.getalp.org/about-dbnary/
4 http://kaiko.getalp.org/about-dbnary/dataset/
5 https://github.com/openlink/virtuoso-opensource

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 83

METEORBaseline

METEOR-DB-4- METEOR-DB-ALLcatg
catg

online A

36.97 %

36.91 %

37.13 %

rbmt-133.74 %

33.60 %

33.89 %

Table 2 . METEOR-Baseline vs METEOR-DBnary for 2 systems picked up randomy from
WMT14 data (French-English MT)
The results of Table 2 above show that METEOR-DBnary-4-catg and METEORBaseline (based on WordNet) both obtain very similar scores while the size of the WordNet
dictionary is 2,5 times larger than that of DBnary (4-catg). Moreover, using all existing
categories in DBnary, we notice an increase of +0.20% in the final score. In other words,
slightly more synonym matches are obtained with the latter metric based on the full English
DBnary.

3.2.

Lemmatisation issues

For English, METEOR uses the Morphy-7WN function as well as an exception list, attempting
to find the lemma of a given word. However, for other target languages, it is very difficult to
identify rules in order to lemmatize an inflected form. Thus, for the moment, we use
TreeTagger (Schmid, 1994) to lemmatize words for other languages.
In order to avoid relaunching TreeTagger for each new entry, we adopt a preprocessing
step that is needed before launching our modified METEOR. During this step, TreeTagger is
run on the full evaluation corpus to collect a list of unique words with their respective lemmas.
We compare the impact of the lemmatization tool used (Morphy vs TreeTagger) by
METEOR on the same two systems of WMT 2014 (see Table 3 results).
METEOR-Morphy

METEOR-TTG

online A

36.97 %

37.00 %

rbmt-133.74 %

33.76 %

Table 3. Impact of lemmatization; METEOR-Morphy vs METEOR-TTG for 2 systems picked
up randomy from WMT14 data (French-English MT)
In Table 3, METEOR-TTG shows a slight increase in the score, compared to METEORMorphy, because TreeTagger lemmatizes all categories (including Adverbs), whereas Morphy
lemmatizes only three categories (Noun, Verb and Adjective).

4.

Correlation with human judgements

In order to evaluate the correlation of our proposed METEOR-DBnary with humanjudgements of machine translation outputs, we used the data from the WMT13 Metrics Shared
Task (Machacek and Bojar, 2013) for English-to-Spanish MT, and from the WMT14 Metrics
Shared Task (Machacek and Bojar, 2014) for French-English, English-French, EnglishGerman and English-Russian MT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 84

We present the results in a similar fashion as in the WMT metrics task methodology
using the following metrics. More details and formulas can be found in (Machacek and Bojar,2013) or (Machacek and Bojar, 2014).
•

System-level using Pearson correlation coefficient between system ranking
based on human judgments versus METEOR (we will use our augmented
metric and compare it to the baseline METEOR).

•

Segment-level using Kendall’s τ correlation between system ranking, at thesentence level, based on human judgments versus METEOR (we will use ouraugmented metric and compare it to the baseline METEOR).

Our results were obtained with two different configurations of METEOR:•METEOR-Baseline: currently available METEOR Universal tool with thesynonym module activated for English only (using the WordNet resource) see table 4.

METEOR-DBnary : our augmented-METEOR with the synonym module
activated for English, French, Spanish, German and Russian, using our lexical
resource DBnary - see table 4.
It is worth mentioning that, in order to use our new synonym dictionaries and evaluate
our approach, we activated the synonym module in METEOR for the following languages:
French Spanish Russian and German, by assigning a weight of 0.8 to each languages (sameweight as for the English module).
•

WMT14

WMT13

FR-EN

EN-FR

EN-RU

EN-GE

EN-ES

METEOR-Baseline

.975

.941

.923

.263

.886

METEOR-DBnary

.973

.943

.928

.320

.895

Table 4. System-level correlations (Pearson Correlation Coefficient) between METEORBaseline (or METEOR-DBnary) and the WMT13/WMT14 human rankings.
WMT14 τWMT13 τ

FR-EN

EN-FR

EN-GE

EN-RU

EN-ES

METEOR-Baseline

.406

.280

.238

.427

.184

METEOR-DBnary

.406

.284

.240

.435

.187

Table 5. Segment-level correlations (Kendall’s τ) between METEOR-Baseline (or METEORDBnary) and the the WMT13/WMT14 human rankings.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 85

Table 4 shows that the use of DBnary slightly improves the system-level correlations of
the METEOR score to human judgments in all language pairs except for French-English.
Table 5 shows the same trend for segment-level correlations which confirms that DBnary canbe a useful resource for MT evaluation. The use of DBnary seems very promising for Russianand German as target languages.
Finally, Table 6 shows the absolute values of both METEOR (Baseline vs DBnary) for
the same language pairs and for a system randomly chosen in the WMT datasets (systemrbmt-1 is a rule-based machine translation system). As expected, METEOR score increases
when used with DBnary since in that case the metric maps more words with the same
meaning, using DBnary as lexical resource for synonymy.
WMT 14

WMT13

EN-FR

EN-RU

EN-DE

EN-ES

METEOR-Baseline

50.94

36.21

38.06

49.88

METEOR-DBnary

52.34

37.60

41.51

51.04

Table 6 : Comparison of METEOR-Baseline vs METEOR-DBnary (for system rbmt-1)
We present below some examples of matches obtained for METEOR-Baseline andfor METEOR-Dbnary.
➢ Example 1 : EN-FR (system rbmt-1)- Reference : Si la personne la plus puissante d'Europe peut être visée, alors lesdirigeants d'entreprise sont sûrement aussi des cibles potentielles.
- Hypothesis: Si la personne la plus puissante de l'Europe peut être visée, alorssûrement les chefs de file des affaires sont également les cibles
potentielles.
 Synonym match : word → lemma → synonym list
• dirigeants → dirigeant → [chef, maître, leader, directeur]•chefs → chef → [tête, maître, cuisinier, leader, maître_queux, patron]=> the lemma “chef” exists in the synonym list of the word “dirigeant”, thus
“dirigeants” and “chefs” are considered as synonyms.
• aussi → aussi → [ainsi, également, itou]•également → également → [aussi, pareillement, de_même, par_ailleurs]=> METEOR considers “aussi” and “également” as synonyms, because “aussi”
belongs in the synonym list of “également” and “également" exists in the synonym list of“aussi” .
 Segment score :METEOR-Baseline : 0.6762
METEOR-DBnary : 0.7290➢ Example 2: EN-FR (system rbmt-1)- Reference : J'estime qu'il est concevable que ces données soient utilisées dans leurintérêt mutuel.
- Hypothesis : Je pense qu'il est concevable que ces données soient employées pour lebénéfice mutuel.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 86

Synonym match : word → lemma → synonym list
• employées → employer → [occuper, utiliser]•utilisées → utiliser → [user]=> During word-to-word alignment, METEOR considers the words “utilisées” (inREF) and “employées” (in HYP) as synonyms, because in the step of synonym match, we findthat the lemma “utiliser” exists in the synonym list of the lemma “employer”.
 Segment score :METEOR-Baseline : 0.6609
METEOR-DBnary : 0.7133➢ Example 3: EN-FR (system rbmt-1)- Reference : Il me parlait, m'encourageait constamment, il habitait mon corps.
- Hypothesis: Il me parlerait, m'encouragent constamment, il a vécu dans mon corps.
Synonym match: word → lemma → synonym list
• habitait → habiter → [occuper]•

vécu → vivre → [ habiter, nourriture ]=> the lemma “habiter” exists in the synonym list of the word “vivre”, thus “habitait” and“vécu” are considered as synonyms.
 Segment score :METEOR-Basline : 0.6743
METEOR-DBnary : 0.7688

5.

Conclusion

We proposed an extension of METEOR, a well-known MT evaluation metric, for multiple
target languages using our in-house lexical resource called DBnary. Our augmented METEOR
obtained a better correlation with human judgements than the baseline METEOR, on the
WMT 2014 metrics dataset for English-to-(French, Russian, German, Spanish) language pairs.
The modified code allowing to call METEOR for new target languages (French,
Russian, German, Spanish) is made available to the research community from the following
link (http://kaiko.getalp.org/about-dbnary/meteor-with-dbnary/).
In a near future, more target languages (today 21 in total) could be plugged very
quickly by us or by interested users (please contact us if you want to contribute) using thesame lexical resource (DBnary). The same adaptation of synonym matches could be done to
TER-Plus (Snover et al., 2009b). Finally, using WSD, such as done in (Apidianaki and Marie,
2015), is another interesting avenue for improving correlation between automatic evaluation
metrics and human judgements.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 87

References
Apidianaki M., Marie B. (2015) METEOR-WSD: Improved Sense Matching in MT Evaluation.
Proceedings of the 9th Workshop on Syntax, Semantics and Structure in Statistical Translation(SSST'9), 4 June, Denver, Colorado, pp. 49--51.
Banerjee, S. and Lavie, A. (2005). "METEOR: An Automatic Metric for MT Evaluation with Improved
Correlation with Human Judgments", Proceedings of the ACL 2005 Workshop on Intrinsic andExtrinsic Evaluation Measures for MT and/or Summarization,
Cimiano, P., Buitelaar, P., McCrae, J., & Sintek, M. (2011). LexInfo : A Declarative Model for the
Lexicon-Ontology Interface. Web Semantics: Science, Services and Agents on the World Wide Web,
9(1), 29–51. doi:10.1016/j.websem.2010.11.001
Denkowski, M. and Lavie, A. (2014). METEOR Universal: Language Specific Translation Evaluation
for Any Target Language, Proceedings of the EACL 2014 Workshop on Statistical MachineTranslation.
Denkowski M., Lavie A. (2010a). "METEOR-NEXT and the METEOR Paraphrase Tables: Improved
Evaluation Support For Five Target Languages", Proceedings of the ACL 2010 Joint Workshop on
Statistical Machine Translation and Metrics MATR.
Denkowski M., Lavie A. (2010b). Extending the METEOR Machine Translation Evaluation Metric to
the Phrase Level. In Human Language Technologies: The 2010 Annual Conference of the North
American Chapter of the Association for Computational Linguistics , pages 250–253, Los Angeles,
California, USA.
Doddington G. (2002). Automatic Evaluation of Machine Translation Quality Using N-gram CoOccurrence Statistics. Proceedings of HLT 2002. San Diego, California. March 24-27, 2002. 138145.
Fellbaum, C. (1998). WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press.
Hellmann, S., Moran, S., Munich, L. M. U., Brümmer, M., Chiarcos, C., & Sukhareva, M. (2015). OLiA
– Ontologies of Linguistic Annotation. Semantic Web Journal - Special Issue on Multilingual
Linked Open Data, 6(4), 379–386.
Klyne, G., & Carroll, J. J. (2004). Resource Description Framework (RDF): Concepts and Abstract
Syntax. (G. Klyne & J. Carroll, Eds.) Structure. World Wide Web Consortium. Retrieved fromhttp://www.w3.org/TR/rdf-concepts/
Sérasset, G. (2015). The DBnary eco-system, data and APIs. 1st Summer Datathon on Linguistic Linked
Open Data (SD-LLOD-15), 15 June, Madrid, Spain.
Machacek, M. and Bojar, O. (2013). Results of the WMT13 Metrics Shared Task. In Proceedings of theNinth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria.
Machacek, M. and Bojar, O. (2014). Results of the WMT14 Metrics Shared Task. In Proceedings of theNinth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland, USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 88

McCrae, J., Spohr, D., & Cimiano, P. (2011). Linking Lexical Resources and Ontologies on the Semantic
Web with Lemon. In G. Antoniou, M. Grobelnik, E. P. B. Simperl, B. Parsia, D. Plexousakis, P. De
Leenheer, & J. Z. Pan (Eds.), ESWC (1) (Vol. 6643, pp. 245–259). Springer.
Moro A., Raganato A., Navigli R. (2014). Entity Linking meets Word Sense Disambiguation: a Unified
Approach. Transactions of the Association for Computational Linguistics (TACL), 2:231–244.
Papineni K., Roukos S., Ward T., & Zhu W.-J. (2002). BLEU: a Method for Automatic Evaluation of
Machine Translation. Proceedings of 40th meeting of the Association for Computational
Linguistics. Philadelphia, USA. July 7-12, 2002. 311-318.
Schmid, N. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. Proceedings ofInternational Conference on New Methods in Language Processing, Manchester, UK.
Snover M., Dorr B., Schwartz R., Micciulla L., Makhoul J. (2006). A Study of Translation Edit Rate withTargeted Human Annotation. Proceedings of AMTA 2006. Cambridge, MA, USA. August 8-12,
2006. 223-231.
Snover M., Madnani N., Dorr B., Schwartz R. (2009a). Fluency, Adequacy, or HTER? Exploring
Different Human Judgments with a Tunable MT Metric. Proceedings of Fourth Workshop on
Statistical Machine Translation at the 12th Meeting of the European Chapter of the Association for
Computational Linguistics. March 30-31, 2009. 259-268.
Snover M., Madnani N., Dorr B., Schwartz R. (2009b). TER-Plus: paraphrase, semantic, and alignment
enhancements to Translation Edit Rate. Machine Translation Journal. September, 2009. 117-127.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 89

Japanese Controlled Language Rules to Improve
Machine Translatability of Municipal Documents
Rei Miyata

rei@p.u-tokyo.ac.jp
The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan

Anthony Hartleya.hartley@rikkyo.ac.jp
Rikkyo University, 3-34-1 Nishi-Ikebukuro, Toshima-ku, Tokyo, 171-8501, JapanCécile Paris
Cecile.Paris@csiro.au
CSIRO, Crn Vimiera and Pembroke Roads, Marsfield NSW, 2122, Australia
Midori Tatsumi

tatsumi@rikkyo.ac.jp
Rikkyo University, 3-34-1 Nishi-Ikebukuro, Toshima-ku, Tokyo, 171-8501, JapanKyo Kageura

kyo@p.u-tokyo.ac.jp
The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan

Abstract
We report on experiments to test the effectiveness of controlled language (CL) rules on texts
from Japanese municipal websites. We compiled a set of rules by trial and error, systematicallyrewriting Japanese source texts and analysing the machine translation (MT) outputs. We thenemployed native English speakers with little knowledge of Japanese as human evaluators andtested the understandability and accuracy of the English machine translated text. Comparing
the results of four MT systems showed that the effectiveness of CL rules varies depending on
the particular MT systems. A preliminary selection of optimal rules for each system showed
more than 15% increase in MT performance. We also assessed the readability of the Japanesesource texts and discuss the way to make them compatible with the quality of the MT outputs.

1 Introduction
It has become increasingly important for Japanese municipalities to provide information in multiple languages. Although combining machine translation (MT) and human post-editing has
increasingly proven to be an effective MT workflow, it may not be an option for the websitesof municipalities, which must offer a large volume of information, such as daily announcements, procedural guidelines, emergency information and policy white papers, with the needfor frequent updates. Having all these documents translated into a number of languages using
human translators or post-editors is simply too expensive and time consuming. In such cases,
raw machine translation output is often provided on-demand.
Translations between languages that have completely different structures such as Japaneseand English are, however, still difficult compared to those between European languages or between Japanese and Korean, and in many cases the quality of raw outputs of MT is of low oreven not understandable quality. When post-editing is not a standard option, and the machinetranslation system is used as a ‘black box’, controlled authoring remains the key to the improvement of translation quality. Controlled authoring involves a suite of technologies and environments that provide document templates, glossary management, grammar and style checkers. In
the current study, we focus on controlled language (CL) rules as a starting point. We chose En-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 90

glish as the target language since English is still an overwhelming choice in translating Japanesemunicipal texts, followed by Chinese, Korean and Portuguese (Carroll, 2010).
In our scenario, we assume that effective CL rules should (i) help to raise the quality of
MT output, and (ii) not degrade the quality of the Japanese source texts. To achieve both ofthese requirements, we devised empirical procedures to formulate CL rules and conducted anevaluation to assess the efficacy of each rule in terms of machine translatability and Japanesereadability.
The remainder of this paper is structured as follows. We describe related work in Section 2.
In Section 3, we discuss how we constructed our CL rules, while Section 4 explains our experimental setup for human evaluation to assess the rules. We present our results accompanied witha discussion in Section 5. Section 6 concludes with implications for future work.

2 Related studies
Controlled (natural) language or C(N)L is ‘a constructed language that is based on a certain
natural language, being more restrictive concerning lexicon, syntax, and/or semantics while
preserving most of its natural properties’ (Kuhn, 2014, p.123). A number of English CL rulesets have been proposed to improve MT performance as well as human comprehension, andthey have been actually implemented, mainly in technical documentation (e.g., Kamprath et al.,
1998; Nyberg et al., 2003). Evaluation experiments on CL for MT have also been undertaken toassess machine translatability and post-editing productivity (Pym, 1990; Bernth and Gdaniec,
2001; O’Brien and Roturier, 2007; Aikawa et al., 2007), showing evidence of the effectivenessof CL.
In the case of Japanese CL, Nagao et al. (1984) devised a controlled grammar to syntactically disambiguate Japanese sentences. Yoshida and Matsuyama (1985) also advocated theneed for Japanese CL in parallel with MT development and conducted pioneering work. Little practical implementation, however, resulted from their efforts. One of the reasons for thissetback in Japanese CL is the difficulty in producing significant results because the machinetranslation task from Japanese to another major language such as English is hard, comparedto the task of automatically translating between European language pairs such as English andFrench.
From the 1990’s to the 2000’s, a number of studies in Japanese computational linguistics have addressed the automatic rewriting (or pre-editing) and paraphrasing for MT (Kim andEhara, 1994; Shirai et al., 1998; Inui and Fujita, 2004). However, fully automatic reformulation of natural language copes badly with highly complex expressions, which tend to requirecontextual information. The scope of variations in linguistic patterns was, therefore, limited.
More recently, Ogura et al. (2010) proposed ‘Simplified Technical Japanese’ (STJ) to improve MT performance. They constructed the rules by (i) identifying linguistic patterns whichappeared related to MT output quality, (ii) defining putative rules, and (iii) conducting a preliminary assessment of their efficacy. They finally formulated the STJ rule set consisting of about 50
rules, while pointing out that it does not comprehensively cover the range of Japanese expression
patterns. Another attempt to create Japanese CL is the on-going ‘Technical Japanese’ project,which focuses mainly on documents for business purposes. It published a ‘Patent Documents
Writing Manual’, which consists of 31 rules designed to improve the clarity and translatabilityof patent texts (Matsuda, 2014).
While recent work focused mainly on technical documents for industry and business, Tatsumi et al. (2013) formulated 22 CL rules for municipal website documents drawing on existing
wisdom about technical writing in Japanese. Since the study chiefly referred to writing guidelines intended for human understandability or readability, the overall efficacy of the rules with
MT was not significant. Thus, there remains much room for investigating other patterns impact-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 91

ing on MT performance within the municipal domain.
O’Brien (2006) argued persuasively for the need to tune CL rule sets to language pairand MT system. The results of evaluation experiments by (Hartley et al., 2012) also suggested
there were differences between rule-based machine translation (RBMT) and statistical machinetranslation (SMT) systems in terms of the impact of specific CL rules on their performance,
although the MT systems as such were not the focus of their investigation. In short, it is still
uncertain to what extent CL rules can be effectively generalised across MT systems or how
much improvement can be attained if we compile rules specifically tuned to a given system.
Practical deployment of CL requires that the readability of the source text (ST) should
not be compromised in the interests of making it more tractable for MT in order to improvetarget text (TT) quality. Indeed, the stated aim of Technical Japanese mentioned above is toimprove both readability and machine tractability (Watanabe, 2010). However, when Hartley
et al. (2012); Tatsumi et al. (2013) investigated the efficacy of CL rules with respect to both thereadability of the Japanese source and the quality of the MT output, they observed for somerules a ‘trade-off’ between ST and TT quality, which needs further exploring.

3 Controlled language rules
3.1

CL formulation protocol

Given that municipal texts should serve readers of Japanese and English equally, any CL rule we
propose here should contribute to the quality of MT outputs without degrading that of source
texts. We assume that comparing original source texts and more machine translatable ones
rewritten by humans enables us to derive insights into how to (re)write texts amenable to MT,
while still guaranteeing source text readability, as long as human authors take charge of thewhole rewriting process.
To materialise our assumption above, we devised the following empirical protocol to detect
linguistic or textual features potentially effective for MT performance:
1. Rewrite a source text aiming at a better quality of MT outputs.
2. Record how the text was changed and assess the quality of the outputs.
3. Repeat steps 1 and 2, until achieving satisfactory quality of the MT outputs.
Examples of the detected linguistic features are shown in Table 1.
ST
電力会社に連絡、使用開始手
続き 完了後、ブレーカーのス
イッチを入れます。
電力会社に連絡 します。使用
開始 の 手続きが 完了した後、ブレーカーのスイッチを入れます。
電力会社に連絡し てください。使用開始の手続きが完了
した後 に、ブレーカーのス
イッチを入れ てください。

MT
You can turn on the contact, theprocedures after completion of
the electric power companies.
Will contact the electric powercompany. Procedures for
activation is complete, you turnon the breaker.
Please contact the electric
power company. Please turn on
the breaker after the procedureof the activation is completed.

Change[original sentence]

Split sentence
Add ‘します’
Add ‘の’Expand ‘完了後’
Change ‘ま す’ to‘てください’Add particle ‘に’Table 1: Example of rewriting the source text
3.2

CL formulation for municipal documents

To formulate CL rules for Japanese municipal documents through this protocol, we first extracted 100 original Japanese sentences from municipal websites and one of the authors con-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 92

ducted the above protocol using three MT systems, i.e., one RBMT system (TransGateway1 )and two SMT systems (Google Translate2 and Minnano Jido Hon’yaku3 ). We then summarised
the linguistic and textual features expected to have an impact on MT quality and classified theminto five categories: Mood/Modal, Structural, Lexical, Textual/Orthographical, and Terminological. In this study, we focus on Structural, Lexical, and Textual/Orthographical categories.
We adopted a total of 38 features (Table 2) which had not been covered in the set of 22 CL rules
proposed by (Tatsumi et al., 2013) and formulated 38 CL rules to regulate these features, suchas ‘Avoid using multiple verbs in a sentence’ (rule 1).
In this phase, we did not take the variability of the systems into account, and extracted
a wide range of features which we assumed could improve MT performance. These rules are
not necessarily effective for every MT system. In formulating them, we observed that a few
rewriting rules intended for one MT system had no effect or even a negative effect on the others.
We thus conducted a quantitative human evaluation to determine which rules were effective forwhich MT system(s).
Structural
20. omission
1. multiple verbs in a sentence
21. suffix
2. lack of subject
22. particle Made (まで)
3. lack of object
23. particle De (で)4. connection
24. particle No (の) to mean ‘by’ or ‘from’
5. particle Ga (が) for object
25. per A
6. enumeration A-Mo, B-Mo (A も、B も) 26. particle Te (て)7. Te-kuru (てくる) / Te-iku (ていく)27. if particle To (と)8. inserted adverbial clause
28. particle He-Ha (へは)9. ending clause with noun
29. particle Ni-Ha (には)10. Sahen-noun4 + auxiliary verb Desu (です) 30. particle No-Ka (のか)11. attributive use of Shika-Nai (しか-ない) 31. demonstrative pronoun (ko-so-a-do)
12. verb + You-ni (ように)32. particle Ni (に)13. A or not
14. Sahen-noun + Wo (を) + Suru (する)Textual/Orthographical
15. Sahen-noun + Sare-ru (される)
33. Japanese Kana / Chinese Kanji
34. bullet markLexical
35. machine dependent character
16. particle Nado (など/等)
36. punctuation (sentence separation)
17. giving and receiving verb
37. square bracket
18. redundant word
38. wave dash (〜)19. compound word
Table 2: A list of features to be regulated

4 Experimental setup
The aim of the evaluation was (1) to gauge how effective our CL rules are to different MT
systems, and (2) to investigate whether the rules which contribute to MT performance also
maintain source readability. Using texts from a municipal website and four MT systems, we
assessed these two parameters through human evaluation.
1 http://www.kodensha.jp/platform/
2 https://translate.google.com
3 https://mt-auto-minhon-mlt.ucri.jgn-x.jp
4 A Sahen-noun is a noun which can be connected to the verb Suru (する) and act as a verb.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 93

4.1

Data

We extracted 11,075 sentences with no more than 70 characters from the Toyohashi City website5 in five categories (public information, Q&A, department information, news articles, andtopical issues), and selected sentences violating our 38 rules – four sentences for each rule –resulting in a total of 152 Japanese-Original (JO) sentences. One of the authors rewrote all 152
sentences in accordance with each rule, so generating 152 Japanese-Rewritten (JR) sentences.
We used four MT systems, two commercial RBMT systems (The Hon’yaku6 and TransGateway, hereafter, systems A and B) and two freely available SMT systems (Google Translate andMinnano Jido Hon’yaku, hereafter, systems C and D), without user dictionaries or any sortof customisation, to translate the 152 JO and 152 JR sentences into English. The result was
1,216 machine translated sentences: 152 sentences for each label AO (system A-Original), AR(system A-Rewritten), and so on – BO, BR, CO, CR, DO and DR.
4.2

MT quality evaluation

Our main interest in evaluating the MT quality was to assess whether or not the translation wasunderstandable in terms of practical use of information. We followed a simple method proposedby (Tatsumi et al., 2013), which focuses on understandability at an acceptable level, disregarding grammatical and lexical errors as long as they do not impair the reader’s comprehension.
Questionnaire design
In order to find out whether or not an MT output was judged understandable and, if so, whether
or not the reader’s understanding was in fact correct, we adopted a two-step evaluation method.
In Step 1, we showed the judges an MT output without telling them that it was the result ofMT, and asked them to indicate how well they understood the text, and how much effort was
required to understand it, by selecting one of the following options:[1] I understood fully what this sentence is saying, after reading it once.
[2] I understood fully what this sentence is saying, after reading it more than once.
[3] I understood partially what this sentence is saying, after reading it more than once.
[4] I have no idea what this sentence is saying even after reading it more than once.
In Step 2, the judges were shown a human translation (HT) corresponding to the MT
output shown in Step 1 as an ‘alternative translation’. The question asked at that point differeddepending on the answer to the question in Step 1. If [1] or [2] had been selected, the judges
were asked to indicate how close the meaning of the new sentence was to the first sentence (theMT output) by selecting one of the following options.
[5] Exactly the same meaning[6] Mostly the same meaning[7] Partly the same meaning[8] Completely different meaning
Considering that the judge’s memory from Step 1 might not last long enough to compare
their understanding at Step 2, we showed the MT output again here. At the same time, in order
to discourage direct comparison between the two texts, the judges were asked to compare only
the general meaning, not focusing on the difference in word choice.
If [3] or [4] had been selected at Step 1, i.e., when the MT output was not understandable,
it was of little use to know whether their understanding was correct or not. Instead, we needed
to know if it was because of the bad quality of the MT or a problem with the content itself. The
judges were shown the HT as an alternative. They were then asked to indicate how much of it
5 http://www.city.toyohashi.lg.jp
6 http://pf.toshiba-sol.co.jp/prod/hon_yaku/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 94

they understood and how much effort was required to do so, with the same options as at Step 1,
i.e., [9]=[1], [10]=[2], [11]=[3], [12]=[4]. In this case, the first sentence (the MT output) was
no longer shown.
Implementation
We employed 24 judges, all adult English native speakers. They were all living in Japan, yet
had little knowledge of Japanese. The judges were thus highly representative of the intended
readers of the targets texts.
Each judge was asked to evaluate 152 MT sentences that corresponded to the 152 Japanesesource sentences but were a mix of translations from eight sources (AO, AR, BO, BR, CO, CR,
DO, DR). The evaluation was conducted online.
4.3

Japanese readability evaluation

The Japanese source texts and their manually rewritten versions were both of understandable
quality from the outset. In order to deal with the subtle differences in the sentence readability,
we adopted the following evaluation method following (Hartley et al., 2012): the judges were
presented with pairs of sentences JO and JR, whose ordering was randomised. Each judge was
asked to evaluate each sentence of the pair on a four-point scale: easy to read; fairly easy toread; fairly difficult to read; difficult to read. We instructed the judges in advance not to focus
on the minute grammatical exactness, but to judge the ease of reading.
For this task, we recruited three Japanese native speaker university graduate students as
judges. Each judge evaluated 152 pairs of Japanese sentences, i.e., all 152 JO and 152 JR.

5 Results and discussions
5.1 MT quality
Overall result
Firstly, we classified the results into four categories (Table 3). The numbers in square brackets
correspond to the choices in each step as described in Section 4.2. An MT output is considered
useful only when either [5] or [6] was chosen at Step 2 (MT–Useful) of the evaluation task
described above. Those classed [7] or [8] are the dangerous instances, as they mean that the MT
output is understandable while conveying inaccurate information (MT–Inaccurate). In the caseof [9] or [10], the MT output is not intelligible and is thus useless. Yet it is less dangerous than
[7] and [8]. Finally, [11] and [12] mean that even the human translation was not understandable.
This may suggest that the problem lies in the content which requires some domain knowledge
or contextual information for full understanding.
Selected option[5][6]

Category
MT–Useful[7][8]

MT–Inaccurate[9][10]MT– Unintelligible[11][12]

HT–Unintelligible

Interpretation
The reader understood the MT output and theirunderstanding was correct
The reader understood the MT output but theirunderstanding was not correct
The reader did not understand the MT output,
but they understood the corresponding HT
The reader did not understand either the MT or
the corresponding HT

Table 3: Result categories
Table 4 shows the percentage of judgements that fell into each category. Overall, applying
the CL rules increased the percentage of [MT–Useful] by around 3–4% for three systems, A, B

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 95

and C, while system D shows a slight decrease. Although our focus is not on the analysis of therules as a whole but on the diagnosis of the efficacy of each rule, it is worth noting that only
about 30% of the MT outputs were deemed useful even after applying each CL rule.
Label
AO
AR
BO
BR
CO
CR
DO
DR

MT–Useful
27.4%
30.9%
23.2%
27.2%
26.5%
30.0%
27.0%
26.3%

MT–Inaccurate
4.6%
5.5%
5.0%
5.7%
3.9%
6.8%
6.4%
6.8%

MT–Unintelligible
62.1%
58.8%
66.0%
63.4%
64.7%
58.3%
61.0%
60.1%

HT–Unintelligible
5.9%
4.8%
5.7%
3.7%
4.8%
4.8%
5.7%
6.8%

Table 4: Overall results of MT quality
A comparison between the four MT systems shows the different effects of the CL ruleson the four MT systems. While in the case of systems A and B the application of the CL
rules mostly resulted in decreasing the number of [MT–Unintelligible] and increasing that of[MT–Useful], in the case of system C, it resulted in a notable increase in the number of [MT–Inaccurate]. The following example shows system C generating an [MT–Inaccurate] output
after the application of rule 14.
Rule 14: avoid using Sahen-noun + Wo (を) + Suru (する)
CO: At the venue, the video for each of the agenda is projected on a large screen, was thedescription of the agenda in an easy-to-understand local residents.
CR: At the venue, the video for each of the agenda is projected on a large screen, I explained the agenda in an easy-to-understand local residents.
HT: At the venue, images related to each topic were projected on a large screen and local
residents explained the topics in an easy to understand manner.
We changed only ‘説明-を-しました’ (‘gave an explanation’) into ‘説明-しました’ (‘explained’) in the source. System C then incorrectly (and unexpectedly) inferred the subject ‘I’though the true subject ‘地域住民’ (‘local residents’) is present in the source, and generated anunderstandable but misleading output. This kind of unpredictable change not directly related to
the feature in question tends to occur with SMT systems.
Importantly, the score of around 5% in the [HT–Unintelligible] category shows that evenhuman translated sentences are sometimes not understandable, implying a fundamental difficulty with evaluating at the sentence-level. We had instructed the human translator to translate
the source sentences without adding explanations or suppressing information to make themcomparable with the MT outputs. Moreover, we did not show the judges the context of eachsentence, so the occasional failure of judges to understand the human translation even though
it was grammatically correct could be due to a lack of knowledge of the Japanese municipal
domain.
Generally applicable CL rules
To diagnose the effectiveness of each CL rule for different MT systems in detail, we focused on
the [MT–Useful] cases, since an increase in [MT-Useful] mostly corresponds to an decrease in[MT-Unintelligible], except for system C, which showed a significant rise in [MT-Inaccurate]
together with [MT-Useful]. We counted the judgements that fell in this category for each ruleand calculated the improvement (or degradation) scores as a percentage, emphasising the improvements in bold (Table 5).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 96

No
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

A
-25.0
25.0
8.3
0.0
0.0
8.3
-16.7
33.3
-8.3
16.7
-8.3
8.3
16.7
-8.3
-16.7
0.0
0.0
0.0
-16.7

B
8.3
25.0
33.3
16.7
8.3
-16.7
0.0
-16.7
0.0
16.7
0.0
-8.3
25.0
-25.0
8.3
-16.7
25.0
0.0
8.3

C
-8.3
25.0
0.0
8.3
-8.3
0.0
8.3
8.3
0.0
25.0
25.0
8.3
8.3
8.3
-8.3
8.3
0.0
8.3
-8.3

D
-16.7
-16.7
-8.3
16.7
-16.7
-16.7
25.0
16.7
25.0
16.7
8.3
16.7
8.3
0.0
25.0
8.3
-8.3
8.3
16.7

No
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

A
16.7
8.3
25.0
0.0
16.7
8.3
-16.7
-16.7
25.0
0.0
-8.3
0.0
0.0
-25.0
8.3
16.7
33.3
16.7
8.3

B
16.7
0.0
-16.7
16.7
-8.3
16.7
8.3
16.7
-16.7
25.0
16.7
0.0
0.0
8.3
-25.0
25.0
-8.3
16.7
-33.3

C
8.3
16.7
16.7
-8.3
0.0
16.7
0.0
-8.3
16.7
-8.3
-16.7
16.7
-33.3
-25.0
25.0
33.3
-8.3
0.0
-16.7

D
0.0
-33.3
-25.0
-8.3
-8.3
8.3
0.0
-25.0
25.0
-33.3
-25.0
0.0
0.0
-16.7
0.0
25.0
-25.0
16.7
-8.3

Table 5: Improvement in [MT–Useful] category
We can see that four rules – 10, 13, 25 and 35 – have a positive effect on all four MT
systems. Some examples of translations of the original sentences and their rewrites are listed
below.
Rule 10: avoid using Sahen-noun + Desu (です)AO: An admission ticket is the 10:00 a.m. sales start on Fri., April 22.
AR: An admission ticket starts sale at 10:00 a.m. on Fri., April 22.
BO: An admission ticket is sales starting on Friday, April 22 at 10:00am.
BR: An admission ticket begins to sell it at 10:00am on Friday, April 22.
CO: Admission ticket is sales start at April 22 (gold) 10 am.
CR: Tickets will start April 22 (Friday) at 10 am selling.
DO: Admission ticket is April 22 (Kim) 10 a.m. launch.
DR: Tickets will be on sale at 10 a.m. on April 22 (Kim).
HT: Ticket sales will start at on Friday, April 22 at 10:00 AM.
In this case, we rewrote Sahen-noun + Desu construction ‘開始-です’ into Sahen-noun +Suru construction ‘開始-します’, which resulted in more natural expressions in the MT outputs.
Rules 2, 4, 8, 12, 20, 28 and 37 show a positive effect on three systems and can also be
regarded as generally applicable rules. We provide below examples of the application of one ofthese rules.
Rule 2: avoid omitting subject
AO: A home and the community are places where a child spends much time daily, andstudy that it is various in a life.
AR: A home and the community are places where a child spends much time daily, and achild studies that it is various in a life.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 97

BO: A house and an area are the place where a child spends much time daily, and various
things will be learned in the life.
BR: A house and an area are the place where a child spends much time daily, and a childwill learn various things in the life.
CO: Home and regions, children are routinely spend place a lot of time, you will learn avariety of things in life.
CR: Home and regions, children are routinely spend place a lot of time, children will learn
a variety of things in life.
HT: Homes and communities are places where children spend a lot of time every day, andwhere they learn many things about life.
In Japanese writing, subjects tend to be omitted. Humans normally have no problems
inferring the subjects from the context. In this case, for example, ‘children’ (子ども) or ‘they’(彼ら) can be inferred in the latter clause. MT systems, however, often have difficulties indealing with null-subject expressions. This is evidenced in AO, BO and CO above: system A
did not insert a subject, but this caused a disagreement between the subject ‘a child’ and the verb
‘study’; system B adopted a passive construction ‘will be learned’; system C wrongly inserted‘you’ as a subject. We can see that inserting ‘子ども’ as a subject in the original Japanese
sentence enhanced the performance of the three systems (see AR, BR and CR).
MT-dependent CL rules
As Table 5 clearly demonstrates, it is also important to note that the effectiveness of each CL
rule is variable. For example, rule 11 had a positive effect on the output of systems C and D.
We look at it in more detail below.
Rule 11: avoid using attributive use of Shika-Nai (しか-ない)
AO/AR: Although it is a plant of a greenhouse, please look at this flower that makes aflower bloom only at this time once.
BO/BR: It’s a plant in a greenhouse, but please see this flower which makes a flower bloom
only at this time once by all means.
CO: Although it is greenhouse plants, please come visit once this flower only at this time
does not bloom.
CR: Although it is greenhouse plants, please come visit once the flowers bloom this time
only to flower.
DO: Is a plant of the greenhouse, you take the time to peruse this flower only during thisperiod not bloom.
DR: Is a plant of the greenhouse, you take the time to peruse the flowers that bloom onlyin this time of the year.
HT: Among the greenhouse plants, please be sure to take a look at this flower, which only
blooms during this time of year.
In this case, we rewrote the attributive expression Shika-Nai (しか-ない) into anotherattributive particle Dake (だけ). For the RMBT systems A and B, both attributive patterns were
linguistically processed in the same manner, using the adverb ‘only’, and this rule shows no
improvement. On the other hand, for the SMT systems C and D, Shika-Nai (しか-ない) is dealt
with as a kind of negative construction, which leads to an unnecessary negation in the output.
Thus, regulating it is effective in improving MT quality.
While this particular rule triggered differing reactions in the RBMT versus SMT systems,
we could not discern a regular correlation between system architectures and the effectiveness of
CL rules. Instead, the results showed the idiosyncrasy of each system.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 98

It should also be added that, in our experimental design to examine the effectiveness ofeach rule, we rewrote only that part of the sentence which was directly related to the applicable
rule. This caused two major issues in our results. First, applying a single CL rule did notnecessarily address the quality of the entire sentence, and thus did not contribute to as great anincrease in [MT–Useful] as we had expected. In particular, there are cases where rewriting thesource text did not change the MT outputs at all, as shown in the examples of rule 11 above.
Even worse, rule 32 produced no improvement in any of the four MT systems. Second, insome cases a CL rule successfully brought a local improvement in the quality of the translation,
yet other critical mistranslations (which, we observed, often stemmed from technical terms andproper nouns) overrode its positive effect and thus led to an overall low grade. This suggeststhat piece-by-piece modifications are not always enough to improve MT output quality.
Optimal rule set for each system
Table 6 shows how much improvement we could see if we select effective CL rules for each
MT system. We preliminarily selected those rules which produced an increase in [MT–Useful]according to Table 5, i.e., 18 rules for system A, 19 for B, 19 for C and 16 for D, and summarised
the results as in Table 4.
For all systems, [MT–Useful] category increases by more than 15 % with no or little increase in the [MT–Inaccurate] category. This result clearly indicates the necessity of tailoring
the selection of rules to a particular MT system. In addition, further improvement can be expected if all applicable optimal rules are applied to a given sentence.
Label
AO
AR
BO
BR
CO
CR
DO
DR

MT–Useful
20.6%
37.3%
20.6%
38.2%
21.1%
36.4%
17.7%
34.4%

MT–Inaccurate
3.9%
3.9%
4.4%
3.5%
4.8%
5.3%
7.8%
6.8%

MT–Unintelligible
68.9%
55.7%
70.6%
54.4%
70.2%
53.1%
69.3%
53.1%

HT–Unintelligible
6.6%
3.1%
4.4%
3.9%
3.9%
5.3%
5.2%
5.7%

Table 6: Overall results of MT quality (after optimal rules were selected)
5.2Japanese readability

The judgements of readability can be separated into two levels: acceptable and unacceptable.
We defined the first two options of the question – easy to read and fairly easy to read – asacceptable and the other two options – fairly difficult to read and difficult to read – as unacceptable, on the assumption that the gap between the former and the latter is significant from thepoint of reading ease by humans.
Columns JO and JR in Table 7 show the percentages of judgements categorised as acceptable, and column JR-JO shows the improvement or deterioration. The higher the score in
JR-JO, the greater the reading ease of JR compared to JO. Given our requirement that the CL
should not degrade source text readability, figures greater than or equal to 0.0 (%) in JR-JO arehighlighted in bold.
As a whole, 23 out of 38 the CL rules improved or at least retained the quality of the source
text. In particular, rules 2, 18 and 26 were effective with more than 40% improvement in readability. According to rule 18 (avoid using redundant word), for instance, we deleted redundant
expressions such as ‘ものとする’ and ‘こととする’. After this rewrite human evaluators judged
JR to be more readable than JO. These kinds of periphrastic expressions are commonly used inProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 99

No
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

JO
91.7
50.0
58.3
83.3
58.3
91.7
83.3
75.0
75.0
58.3
83.3
83.3
91.7
75.0
75.0
66.7
66.7
58.3
83.3

JR
75.0
91.7
91.7
58.3
91.7
50.0
100.0
75.0
58.3
91.7
75.0
91.7
75.0
83.3
83.3
66.7
100.0
100.0
75.0

JR-JO
-16.7
41.7
33.3
-25.0
33.3
-41.7
16.7
0.0
-16.7
33.3
-8.3
8.3
-16.7
8.3
8.3
0.0
33.3
41.7
-8.3

No
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

JO
75.0
100.0
100.0
91.7
66.7
66.7
50.0
83.3
66.7
66.7
91.7
58.3
100.0
75.0
83.3
83.3
75.0
66.7
100.0

JR
50.0
50.0
66.7
75.0
91.7
91.7
91.7
91.7
66.7
83.3
83.3
58.3
66.7
100.0
100.0
66.7
91.7
66.7
41.7

JR-JO
-25.0
-50.0
-33.3
-16.7
25.0
25.0
41.7
8.3
0.0
16.7
-8.3
0.0
-33.3
25.0
16.7
-16.7
16.7
0.0
-58.3

Table 7: Improvement in Japanese readability
municipal documents in Japan.
In contrast, rules 6, 21 and 38 resulted in more than 40% degradation in readability. Avoid
using suffix (rule 21) and wave dash ‘〜’ (rule 38) introduce redundancy by replacing the feature
with a longer sequence of words, such as replacing ‘午後 1 時〜4 時’ (‘1:00–4:00 PM’) with
‘午後 1 時から 4 時まで’ (‘from 1:00 to 4:00 PM’).
More detailed analysis revealed there are some rules for which the evaluations of their effectiveness differed depending on the more specific features of sentences. For example, rewriting ‘記念品代相当分’ (‘an appropriate amount of money toward the commemorative item’) as‘記念品代に相当する分’ (‘an appropriate amount of money corresponding to the commemorative item’) according to rule 19 (avoid using compound noun) improves readability, while
rewriting ‘市民提供資料’ (‘materials provided by residents’) as ‘市民が提供した資料’ (‘materials that residents provided’) based on the same rule degrades readability.
5.3

Compatibility of machine translatability and Japanese readability

Comparing the results of the MT quality (Table 5) and Japanese readability (Table 7), we now
discuss the compatibility of the two requirements. Focusing on the generally applicable rules,
we see that rules 2, 8, 10, 12, 25, 28 and 37 improved or retained Japanese readability. In
particular, rule 2 and 10 greatly improved both machine translatability and Japanese readability.
There are, however, some rules which are effective for MT quality in general but have an adverse
effect on human readability, such as rules 4, 13, 20 and 35. Rule 20, for instance, produced abetter MT output for systems A, B and C, but degraded the readability of the source text. We
give an example below.
Rule 20: avoid omission
JO: 月・水・金曜日 の午前 9 時から午後 4 時まで開設しており、3 月末まで開設しています。
JR: 月曜日・水曜日・金曜日 の午前 9 時から午後 4 時まで開設しており、3 月末まで開設しています。

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 100

BO: It’s established from a month and 9:00am of water and Friday to 4:00pm and it’sestablished until the end of March.
BR: It’s established from 9:00am of Monday, Wednesday and Friday to 4:00pm and it’sestablished until the end of March.
HT: It will be open from 9:00 AM to 4:00 PM on Mondays, Wednesdays, and Fridays until
the end of March.
System B failed to recognise ‘月・水・金曜日’ as an elliptic expression and literally translated ‘月’ into ‘month’, ‘水’ into ‘water’, and ‘金曜日’ into ‘Friday’ (BO). Restoring an omitted
element ‘曜日’ according to rule 20 helped the system deal correctly with this expression (BR).
In contrast, human evaluators preferred JO to JR in terms of readability. This is no doubt because too much complementation made the sentence longer and hindered reading. The point
here is that we need to find a way to meet both requirements of the MT quality and human readability. In the case of rule 20, for instance, we should keep JO for human readers, but employ
pre-processing to produce JR for MT purposes.

6

Conclusion and future work

We have proposed an empirical protocol for formulating CL rules with a view to improving
the machine translatability of Japanese municipal documents. Focusing on Japanese to English
translation and using 100 sentences from municipal websites, we derived 38 CL rules different
from the 22 rules that had previously been formulated on the basis of collective wisdom about
technical writing.
We assessed the efficacy of the rules on municipal documents, with respect to both MT
quality and source text readability. We identified a total of 11 rules which are effective for atleast three MT systems. Since previous studies could identify few ‘general’ rules (Hartley et al.,2012; Tatsumi et al., 2013), this result encourages us to pursue our CL formulation protocol. In
addition, we consider the protocol described in Section 3.1 to be generalisable to other language
pairs and text domains, even though the CL rules in this study are formulated particularly forJapanese-to-English translation of municipal documents.
Interestingly, the effectiveness of CL rules was not shown to align with architectural differences between RBMT and SMT. This implies that we need to tune CL rule sets at the levelof particular MT systems rather than at the level of MT types. In addition, a preliminary selection of optimal rules for each system achieved a greater than 15% increase in the [MT–Useful]
category.
On the other hand, the results of the Japanese readability assessment showed that about two
thirds of the CL rules improved or at least maintained source text readability. To achieve bothmachine translatability and human readability, it is important to serve different texts to humans
and machines. We identified that degradations in readability for humans often correlate withredundancy generated by the rules. Thus an effective solution would be, for instance, to unpack
the elliptic expressions and insert linguistic elements such as subjects only for MT. Moreover,this pre-processing for MT can be automated to some extent by employing existing pre-editing
methods (e.g., Shirai et al., 1998), which can reduce the cost of implementing CL rules. In
a future study, we will assess the total productivity of controlled authoring and translation in
combination with post-editing.

Acknowledgement
This work was supported by the Research Grant Program of KDDI Foundation, Japan. The MT
system J-SERVER Professional TransGateway V3 was offered by Kodensha Co.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 101

References
Aikawa, T., Schwartz, L., King, R., Corston-Oliver, M., and Lozano, C. (2007). Impact of controlled
language on translation quality and post-editing in a statistical machine translation environment. In
Proceedings of the Machine Translation Summit XI, pages 1–7, Copenhagen, Denmark.
Bernth, A. and Gdaniec, C. (2001). MTranslatability. Machine Translation, 16(3):175–218.
Carroll, T. (2010). Local government websites in Japan: international, multicultural, multilingual?
Japanese Studies, 30(3):373–392.
Hartley, A., Tatsumi, M., Isahara, H., Kageura, K., , and Miyata, R. (2012). Readability and translatability
judgments for ‘Controlled Japanese’. In Proceedings of the 16th Annual Conference of the European
Association for Machine Translation, pages 237–244, Trento, IT.
Inui, K. and Fujita, A. (2004). A survey on paraphrase generation and recognition. Journal of Natural
Language Processing, 11(5):151–198. (in Japanese).
Kamprath, C., Adolphson, E., Mitamura, T., and Nyberg, E. (1998). Controlled language for multilingual document production: Experience with Caterpillar Technical English. In Proceedings of the 2nd
International Workshop on Controlled Language Applications, pages 51–61, Pittsburgh, PA.
Kim, Y.-B. and Ehara, T. (1994). An automatic sentence breaking and subject supplement method for J /E machine translation. IPSJ Journal, 35(6):1018–1028.
Kuhn, T. (2014). A survey and classification of controlled natural languages. Computational Linguistics,
40(1):121–170.
Matsuda, S. (2014). Efforts for Technical Japanese : Focusing mainly on the ‘Patent Documents Writing
Manual’. Journal of Information Processing and Management, 57(6):387–394. (in Japanese).
Nagao, M., Tanaka, N., and Tsujii, J. (1984). Support system for writing texts based on controlled grammar. IPSJ SIG Technical Reports, 1984(27):33–40. (in Japanese).
Nyberg, E., Mitamura, T., and Huijsen, W.-O. (2003). Controlled language for authoring and translation.
In Somers, H., editor, Computers and the Translator, pages 245–281. Benjamins, Amsterdam.
O’Brien, S. (2006). Controlled language and post-editing. Multilingual, 17(7):17–19.
O’Brien, S. and Roturier, J. (2007). How portable are controlled language rules? In Proceedings of theMachine Translation Summit XI, pages 345–352, Copenhagen, DK.
Ogura, E., Kudo, M., and Yanagi, H. (2010). Simplified Technical Japanese: Writing translation-readyJapanese documents. IPSJ SIG Technical Reports, 2010-DD-78(5):1–8. (in Japanese).
Pym, P. (1990). Pre-editing and the use of simplified writing for MT. In Mayorcas, P., editor, Translating
and the Computer 10, pages 80–95. Aslib, London.
Shirai, S., Ikehara, S., Yokoo, A., and Ooyama, Y. (1998). Automatic rewriting method for internal
expressions in Japanese to English MT and its effects. In Proceedings of the 2nd International Workshop
on Controlled Language Applications, pages 62–75, Pittsburgh, PA.
Tatsumi, M., Miyata, R., Hartley, A., Kageura, K., and Isahara, H. (2013). Towards acceptable quality machine translation without post-editing for municipal websites: An evaluation ofJapanese controlled language rules. In Machine Translation Summit XIV: QTLaunchPad Workshop on Human-Centric Machine Translation and Evaluation, Nice, France. (Slide available:
http://reimiyata.com/materials/mtsummit2013-ws.pdf).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 102

Watanabe, T. (2010). Outline of the ‘Technical Japanese’ project: Activity for acceleration of patenttechnological information utilization. Journal of Information Processing and Management, 53(9):480–
491. (in Japanese).
Yoshida, S. and Matsuyama, A. (1985). Standardizing Japanese: Standardizing dependency relations andtransformation rules. IPSJ SIG Technical Reports, 1985(31):1–6. (in Japanese).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 103

Morphological Constraints
for Phrase Pivot Statistical Machine Translation
Ahmed El Kholy

ame2127@columbia.edu

Center for Computational Learning Systems, Columbia University

Nizar Habash

nizar.habash@nyu.edu

Computer Science, New York University Abu Dhabi

Abstract
The lack of parallel data for many language pairs is an important challenge to statistical machine translation (SMT). One common solution is to pivot through a third language for which
there exist parallel corpora with the source and target languages. Although pivoting is a robust technique, it introduces some low quality translations especially when a poor morphology
language is used as the pivot between rich morphology languages. In this paper, we examine
the use of synchronous morphology constraint features to improve the quality of phrase pivot
SMT. We compare hand-crafted constraints to those learned from limited parallel data between
source and target languages. The learned morphology constraints are based on projected alignments between the source and target phrases in the pivot phrase table. We show positive results
on Hebrew-Arabic SMT (pivoting on English). We get 1.5 BLEU points over a phrase pivot
baseline and 0.8 BLEU points over a system combination baseline with a direct model built
from parallel data.

1

Introduction

One of the main challenges in statistical machine translation (SMT) is the scarcity of parallel
data for many language pairs especially when the source and target languages are morphologically rich. A common SMT solution to the lack of parallel data is to pivot the translation througha third language (called pivot or bridge language) for which there exist abundant parallel corpora with the source and target languages. The literature covers many pivoting techniques. Oneof the best performing techniques, phrase pivoting (Utiyama and Isahara, 2007), builds an induced new phrase table between the source and target. One of the main issues of this techniqueis that the size of the newly created pivot phrase table is very large. Moreover, many of the
produced phrase pairs are of low quality which affects the translation choices during decoding
and the overall translation quality.
In this paper, we focus on improving phrase pivoting. We introduce morphology constraint
scores which are added to the log linear space of features in order to determine the quality ofthe pivot phrase pairs. We compare two methods of generating the morphology constraints.
One method is based on hand-crafted rules relying on the authors knowledge of the source
and target languages; while in the other method, the morphology constraints are induced from

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 104

available parallel data between the source and target languages which we also use to build a
direct translation model. We then combine both the pivot and direct models to achieve better
coverage and overall translation quality. We show positive results on Hebrew-Arabic SMT.
We get 1.5 BLEU points over a phrase-pivot baseline and 0.8 BLEU points over a system
combination baseline with a direct model built from given parallel data.
Next, we briefly discuss some related work. In Section 3, we review the best performing
pivoting strategy and how we use it. In Section 4, we discuss the linguistic differences among
Hebrew, Arabic, and the pivot language, English. This is followed by our approach to usingmorphology constraints in Section 5. We finally present our experimental results in Section 6
and a case study in Section 7.

2

Related Work

Many researchers have investigated the use of pivoting (or bridging) approaches to solve the data
scarcity issue (Utiyama and Isahara, 2007; Wu and Wang, 2009; Khalilov et al., 2008; Bertoldi
et al., 2008; Habash and Hu, 2009). The main idea is to introduce a pivot language, for which
there exist large source-pivot and pivot-target bilingual corpora. Pivoting has been explored for
closely related languages (Hajič et al., 2000) as well as unrelated languages (Koehn et al., 2009;
Habash and Hu, 2009). Many different pivot strategies have been presented in the literature. The
following three are the most common. The first strategy is the sentence translation technique inwhich we first translate the source sentence to the pivot language, and then translate the pivot
language sentence to the target language (Khalilov et al., 2008). The second strategy is basedon phrase pivoting (Utiyama and Isahara, 2007; Cohn and Lapata, 2007; Wu and Wang, 2009).
In phrase pivoting, a new source-target phrase table (translation model) is induced from sourcepivot and pivot-target phrase tables. Lexical weights and translation probabilities are computedfrom the two translation models. The third strategy is to create a synthetic source-target corpus
by translating the pivot side of source-pivot corpus to the target language using an existing
pivot-target model (Bertoldi et al., 2008). In this paper, we use the phrase pivoting approach,which has been shown to be the best with comparable settings (Utiyama and Isahara, 2007).
There has been recent efforts in improving phrase pivoting. One effort focused on improving alignment symmetrization targeting pivot phrase systems (El Kholy and Habash, 2014).
In another recent effort, Multi-Synchronous Context-free Grammar (MSCFG) is leveraged totriangulate source-pivot and pivot-target synchronous Context-free Grammar (SCFG) rule tables into a source-target-pivot MSCFG rule table that helps in remembering the pivot during
decoding. Also, pivot LMs are used to assess the naturalness of the derivation (Miura et al.,2015).
In our own previous work, we demonstrated quality improvement using connectivity
strength features between the source and target phrase pairs in the pivot phrase table (El Kholyet al., 2013). These features provide quality scores based on the number of alignment links
between words in the source phrase to words of the target phrase. In this work, we extend
on the connectivity scores with morphological constraints through which we provide quality
scores based on the morphological compatibility between the connected/aligned source and target words.
Since both Hebrew and Arabic are morphologically rich, we should mention that there
has been a lot of work on translation to and from morphologically rich languages (Yeniterzi and
Oflazer, 2010; Elming and Habash, 2009; El Kholy and Habash, 2010; Habash and Sadat, 2006;
Kathol and Zheng, 2008). Most of these efforts are focused on syntactic and morphologicalprocessing to improve the quality of translation.
Until recently, there has not been much parallel Hebrew-English and Hebrew-Arabic data
(Tsvetkov and Wintner, 2010), and consequently little work on Hebrew-English and Hebrew-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 105

Translation Model
Hebrew-English
English-Arabic
Pivot Hebrew-Arabic

Training Corpora
Size≈1M words≈60M words
N/A

Phrase Table# Phrase Pairs
Size
3,002,887
327MB
111,702,225
14GB> 30 Billion≈2.5TB

Table 1: Translation Models Phrase Table comparison in terms of number of lines and sizes.
Arabic SMT. Lavie et al. (2004) built a transfer-based translation system for Hebrew-English
and so did Shilon et al. (2012) for translation between Hebrew and Arabic. Our previous work
discussed above (El Kholy et al., 2013) was demonstrated on Hebrew-Arabic with English
pivoting.

3

Phrase Pivoting

In this section, we review the phrase pivoting strategy in detail as we describe how we built
our baseline for Arabic-Hebrew via pivoting on English. We also discuss how we overcome
the large expansion of source-to-target phrase pairs in the process of creating a pivot phrase
table. In phrase pivoting (which is sometimes called triangulation or phrase table multiplication), we train a Hebrew-Arabic and an English-Arabic translation models, such as those used in
the sentence pivoting technique. Based on these two models, we induce a new Hebrew-Arabictranslation model. Since our models are based on a Moses phrase-based SMT system (Koehn
et al., 2007), we use the standard set of phrase-based translation probability distributions.1 We
follow Utiyama and Isahara (2007) in computing the pivot phrase pair probabilities. The following are the set of equations used to compute the lexical probabilities (pw ) and the phrasetranslation probabilities (φ):
P
φ(h|a) = φ(h|e)φ(e|a)
eP
φ(a|h) = φ(a|e)φ(e|h)
Pe
pw (h|a) = pw (h|e)pw (e|a)
ePpw (a|h) = pw (a|e)pw (e|h)
eAbove, h is the Hebrew source phrase; e is the English pivot phrase that is common in both
Hebrew-English translation model and English-Arabic translation model; and a is the Arabic
target phrase. We also build a Hebrew-Arabic reordering table using the same technique but wecompute the reordering probabilities in a similar manner to Henriquez et al. (2010).
Filtering for Phrase Pivoting As discussed earlier, the induced Hebrew-Arabic phrase andreordering tables are very large. Table 1 shows the amount of parallel corpora used to train
the Hebrew-English and the English-Arabic and the equivalent phrase table sizes compared tothe induced Hebrew-Arabic phrase table.2 We follow the work of El Kholy et al. (2013) andfilter the phrase pairs used in pivoting based on log-linear scores. The main idea of the filtering process is to select the top [n] English candidate phrases for each Hebrew phrase fromthe Hebrew-English phrase table and similarly select the top [n] Arabic target phrases for eachEnglish phrase from the English-Arabic phrase table and then perform the pivoting process described earlier to create a pivoted Hebrew-Arabic phrase table. To select the top candidates, we
1 Four different phrase translation scores are computed in Moses’ phrase tables: two lexical weighting scores andtwo phrase translation probabilities.
2 The size of the induced phrase table size is computed but not created.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 106

first rank all the candidates based on the log linear scores computed from the phrase translation
probabilities and lexical weights multiplied by the optimized decoding weights then we pickthe top [n] pairs. In our experiments, we pick the top 1000 pairs for pivoting.

4

Linguistic Comparison

In this section we present the challenges of preprocessing Arabic, Hebrew, and English, andhow we address them. Both Arabic and Hebrew are morphologically complex languages. One
aspect of Arabic’s complexity is its various attachable clitics and numerous morphological features (Habash, 2010). Clitics include conjunction proclitics, e.g., +ð w+3 ‘and’, prepositional
proclitics, e.g., + È l+ ‘to/for’, the definite article + È@ Al+ ‘the’, and the class of pronominal
enclitics, e.g., Ñë+ +hm ‘their/them’. All of these clitics are separate words in English. Beyond the clitics, Arabic words inflect for person, gender, number, aspect, mood, voice, stateand case. Additionally, Arabic orthography uses optional diacritics for short vowels and consonant doubling. This, together with Arabic’s morphological richness, leads to a high degreeof ambiguity: about 12 analyses per word, typically corresponding to two lemmas on average(Habash, 2010). We follow El Kholy and Habash (2010) and use the PATB tokenization scheme(Maamouri et al., 2004) in our experiments. The PATB scheme separates all clitics except forthe determiner clitic Al+(D ET). We use MADA v3.1 (Habash and Rambow, 2005; Habash
et al., 2009) to tokenize the Arabic text. We only evaluate on detokenized and orthographicallycorrect (enriched) output following the work of El Kholy and Habash (2010).
Similar to Arabic, Hebrew poses computational processing challenges typical of Semitic
languages (Itai and Wintner, 2008; Shilon et al., 2012). Hebrew orthography also uses optional
diacritics and its morphology inflects for gender, number, person, state, tense and definiteness.
Furthermore, Similar to Arabic, Hebrew has a set of attachable clitics, e.g., conjunctions (suchas +!‫ ו‬w+4 ‘and’), prepositions (such as +!‫ ב‬b+ ‘in’), the definite article (+!‫ ה‬h+ ‘the’), or pronouns(such as !M‫ה‬+ +hm ‘their’). These issues contribute to a high degree of ambiguity that is achallenge to translation from Hebrew to English or to any other language. We follow Singh andHabash (2012)’s best preprocessing setup which utilized a Hebrew tagger (Adler, 2007) andproduced a tokenization scheme that separated all clitics.
English, our pivot language, is quite different from both Arabic and Hebrew. English ispoor in morphology and barely inflects for number and tense, and for person in a limited context.
English preprocessing simply includes down-casing, separating punctuation and splitting off
“’s”.

5

Approach

One of the main challenges in phrase pivoting is the very large size of the induced phrase table.
It becomes even more challenging if either the source or target language is morphologicallyrich. The number of translation candidates (fanout) increases due to ambiguity and richness
which in return increases the number of combinations between source and target phrases. Since
the only criteria of matching between the source and target phrase is through a pivot phrase,many of the induced phrase pairs are of low quality. These phrase pairs unnecessarily increase
the search space and hurt the overall quality of translation. A basic solution to the combinatorial
expansion is to filter the phrase pairs used in pivoting based on log-linear scores as discussed in
Section 3, however, this doesn’t solve the low quality problem.
3 Arabic transliteration throughout the paper is presented in the Habash-Soudi-Buckwalter scheme (Habash et al.,2007).
4 The following Hebrew 1-to-1 transliteration is used (in Hebrew alphabetical order): abgdhwzxTiklmns‘pcqršt. All
examples are undiacritized and final forms are not distinguished from non-final forms.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 107

Similar to factored translation models (Koehn and Hoang, 2007) where linguistic (morphology) features are augmented to the translation model to improve the translation quality, ourapproach to address the quality problem is based on constructing a list of synchronous morphology constraints between the source and target languages. These constraints are used to generatescores to determine the quality of pivot phrase pairs. However, unlike factored models, we do
not use the morphology in generation and the morphology information comes completely fromexternal resources. In addition, since we work in the pivoting space, we only apply the morphology constraints to the connected words between the source and target languages through thepivot language. This guarantees a fundamental level of semantic equivalence before applying
the morphology constraints especially if there is distortion between source and target phrases.
We build on our approach in El Kholy et al. (2013) where we introduced connectivity
strength features between the source and target phrase pairs in the pivot phrase table. Thesefeatures provide quality scores based on the number of alignment links between words in thesource phrase and words in the target phrase. The alignment links are generated by projecting
the alignments of the source-pivot phrase pairs and the pivot-target phrase pairs used in pivoting. We use the same concept but instead of using the lexical mapping between source andtarget words, we compute quality scores based on the morphological compatibility between theconnected source and target words.
To choose which morphological features to work with, we performed an automatic error
analysis on the output of the phrase-pivot baseline system. We did the analysis using AMEANA(El Kholy and Habash, 2011), an open-source error analysis tool for natural language processing
tasks targeting morphologically rich languages. We found that the most problematic morphological features in the Arabic output are gender (G EN), number (N UM) and determiner (D ET).
We focus on those features in addition to (P OS) in our experiments.
Next, we present our approach to generating the morphology constraint features using
hand-crafted rules and compare this approach with inducing these constraints from HebrewArabic parallel data.
5.1

Rule-based Morphology Constraints

Our rule-based morphology constraint features are basically a list of hand-crafted mappings ofthe different morphological features between Hebrew and Arabic. Since both languages aremorphologically rich as explained in Section 4, it is straightforward to produce these mappings
for G EN, N UM and D ET. Note, however, that we also account for ambiguous cases; e.g.,feminine gender in Arabic can map to words with ambiguous gender in Hebrew. We additionallyuse different P OS tag sets for Arabic (47 tags) and Hebrew (25 tags) and in many cases one
Hebrew tag can map to more than one Arabic tag; for example, three Arabic noun tags abbrev,
noun and noun prop map to two Hebrew tags feminine, masculine noun.5 Table 2 shows asample of the morphological mappings between Arabic and Hebrew.
After building the morphological features mappings, we use them to judge the quality ofa given phrase pair in the phrase pivot model. We add two scores Ws and Wt to the log linear
space. Given a source-target phrase pair s̄, t̄ and a word projected alignment a between thesource word positions i = 1, ..., n and the target word positions j = 1, ..., m, Ws and Wt are
defined in equations 1 and 2. F is the set of morphological features (we focus on G EN, N UM,
D ET and P OS). Mf is the hand-crafted rules mapping between Arabic and Hebrew feature
values of feature f ∈ F . In case of ambiguity for a given feature; for example, a word’s gender
being masculine or feminine, we use the maximum likelihood value of this feature given theword. M LEf (i) is the maximum likelihood feature value of feature f for the source word at
5 Please refer to (Habash et al., 2009) for a complete set of Arabic P OS tag set and (Adler, 2007) for Hebrew P OS
tag set.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 108

Features
G EN
N UM

D ET

ArabicFeminine
Masculine
Singluar
Dual
PluralNo Determiner
Determiner

Hebrew
Feminine / Both
Masculine / BothSingluar / Singluar-Plural
Dual / Dual-Plural
Plural / Dual-Plural / Singular-Plural
No Determiner
Determiner

Table 2: Rule-based mapping between Arabic and Hebrew morphological features. Each feature
value in Arabic can map to more than one feature value in Hebrew.
position i, and M LEf (j) is the maximum likelihood feature value of feature f for the target
word at position j. The maximum likelihood feature values for Hebrew were computed from the
Hebrew side of the training data. As for Arabic, the maximum likelihood feature values werecomputed from the Arabic side of the training data in addition to Arabic Gigaword corpus,
which was used in creating the language model (more details in Section 6.1).

Ws =

1 X
|F |

X

1 X
|F |

X

∀f ∈F ∀(i,j)∈a

Wt =∀f ∈F ∀(i,j)∈a

5.2

1[(M LEf (i), M LEf (j)) ∈ Mf ]
n

(1)

1
[(M LEf (i), M LEf (j)) ∈ Mf ]m

(2)

Induced Morphology Constraints

In this section, we explain our approach in generating morphology constraint features from agiven parallel data between source and target languages. Unlike the rule-based approach webuild a translation model between the source and target morphological features and we use
the morphology translation probabilities as metric to judge a given phrase pair in the pivot
phrase table. For the automatically induced constraints, we jointly model mapping between
conjunctions of features attached to aligned words rather than tallying each feature match independently. Writing good manual rules for such feature conjunction mappings would be moredifficult. Table 3 shows some examples of mapping (G EN), number (N UM) and determiner(D ET) in Hebrew to their equivalent in Arabic and their respective bi-directional scores.
Hebrew (H)[Fem+Dual+Det][Fem+Dual+Det][Fem+Dual+Det][Fem+Dual+Det]

Arabic (A)[Fem+Dual][Fem+Dual+Det][Fem+Singular] [Fem+Dual][Masc+Dual+Det]

PF C (A|H)
0.00060.01480.00520.0047PF C (H|A)
0.08330.33330.08330.5000

Table 3: Examples of induced morphology constraints for (G EN), number (N UM) and determiner (D ET) and their respective scores.
As in rule-based approach, we add two scores Ws and Wt to the log linear space which aredefined in equations 3 and 4. PF C is the conditional morphology probability of a given feature

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 109

combination (F C) value. Similar to rule-based morphology constraints, we resort to the maximum likelihood value of a feature combination when the values are ambiguous. M LEF C (i)is the maximum likelihood feature combination (F C) value for the source word at position iwhile M LEF C (j) is the maximum likelihood feature combination (F C) value for the target
word at position j.
Ws =

Wt =

5.3

1
n
1
m

X

PF C (M LEF C (i)|M LEF C (j))(3)

PF C (M LEF C (j)|M LEF C (i))(4)∀(i,j)∈aX∀(i,j)∈a

Model Combinations

Since we use parallel data to induce the morphology constraints, it would make sense to measure
the effect of combining (a) the pivot model with added morphology constraints, and (b) the
direct model trained on the parallel data used to induce the morphology constraints. We performthe combination using Moses’ phrase table combination techniques. Translation options arecollected from one table, and additional options are collected from the other tables. If the sametranslation option (in terms of identical input phrase and output phrase) is found in multiple
tables, separate translation options are created for each occurrence, but with different scores(Koehn and Schroeder, 2007). We show results over a learning curve in Section 6.5.

6

Experiments

In this section, we present a set of experiments comparing the use of rule-based versus induced
morphology constraint features in phrase-pivot SMT as well as model combination to improve
Hebrew-Arabic pivot translation quality.
6.1

Experimental Setup

In our pivoting experiments, we build two SMT models; one model to translate from Hebrew
to English, and another model to translate from English to Arabic. The English-Arabic parallel
corpus is about (≈ 60M words) and is available from LDC6 and GALE7 constrained data. The
Hebrew-English corpus is about (≈ 1M words) and is available from sentence-aligned corpus
produced by Tsvetkov and Wintner (2010). For the direct Hebrew-Arabic SMT model, we use
a TED parallel corpus of about (≈ 2M words) (Cettolo et al., 2012).
Word alignment is done using GIZA++ (Och and Ney, 2003). For Arabic language modeling, we use 200M words from the Arabic Gigaword Corpus (Graff, 2007) together with the
Arabic side of our training data. We use 5-grams for all language models (LMs) implemented
using the SRILM toolkit (Stolcke, 2002).
All experiments are conducted using the Moses phrase-based SMT system (Koehn et al.,2007). We use MERT (Och, 2003) for decoding weight optimization. Weights are optimized
using a tuning set of 517 sentences developed by Shilon et al. (2010).
We use a maximum phrase length of size 8 across all models. We report results on a
Hebrew-Arabic development set (Dev) of 500 sentence with a single reference and an evaluation
set (Test) of 300 sentences with three references developed by Shilon et al. (2010). We evaluate
using BLEU-4 (Papineni et al., 2002).
6 LDC Catalog IDs:

LDC2005E83, LDC2006E24, LDC2006E34, LDC2006E85, LDC2006E92, LDC2006G05,
LDC2007E06, LDC2007E101, LDC2007E103, LDC2007E46, LDC2007E86, LDC2008E40, LDC2008E56,
LDC2008G05, LDC2009E16, LDC2009G01.
7 Global Autonomous Language Exploitation, or GALE, was a DARPA-funded research project.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 110

6.2

Baselines

We compare the performance of adding the connectivity strength features (+Conn) to the phrase
pivoting SMT model (Phrase Pivot) and building a direct SMT model using all parallel He-Ar
corpus available. The results are presented in Table 4. Consistently with our previous effort(El Kholy et al., 2013), the performance of the phrase-pivot model improves with the connectivity strength features. While the direct system is better than the phrase pivot model in general,the combination of both models leads to a high performance gain of 1.7/4.4 BLEU points in
Dev/Test over the best performers of both the direct and phrase-pivot models.
Model
DirectPhrase Pivot
Phrase Pivot+Conn
Direct+Phrase Pivot+Conn

Dev
9.7
8.3
9.1
11.4

Test
20.4
19.8
20.1
24.5

Table 4: Comparing phrase pivoting SMT with connectivity strength features, direct SMT andthe model combination. The results show that the best performer is the model combination in
Dev and Test sets.
6.3

Rule-based Morphology Constraints
Model
DirectPhrase Pivot+Conn
Phrase Pivot+Conn+Morph Rules
Phrase Pivot+Conn+Morph Auto

Single
9.7
9.1
9.6
9.6

Dev
Combined
n/a
11.4
12.2*
12.4*

Single
20.4
20.1
20.9*
21.6*

Test
Combined
n/a
24.5
24.6
25.3*

Table 5: Morphology constraints results. The “Single” columns show the results of a single
model of either the direct model or the phrase pivoting models with additional morphologicalconstraints features. The “Combined” show the results of system combination between the
direct model and the different phrase pivoting models. In the first row, the “Combined” results
are not applicable for the direct model. (*) marks a statistically significant result against both
the direct and phrase-pivot baseline.
In this experiment, we show the performance of adding hand-crafted morphology constraints (+Morph Rules) to determine the quality of a given phrase pair in the phrase-pivot
translation model. The third row in Table 5 shows that although the rules are based on a one-toone mapping between the different morphological features, the translation quality is improved
over the baseline phrase-pivot model by 0.5/0.8 BLEU points in Dev/Test sets.
As expected, the system combination of the pivot model with the direct model improves
the overall performance but the gain we get from the morphology constraints only appears in
the Dev set with 0.8 BLEU points, and not much in the Test set.
6.4

Induced Morphology Constraints

In this experiment, we measure the effect of using induced morphology constraints(+Morph Auto) on MT quality. The last row in Table 5 shows that the induced morphology
constraints improve the results over the baseline phrase-pivot model by 0.5/1.5 BLEU points inProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 111

Parallel Model
Dev
Test
Data Size
Single Combined Single Combined
125K
Direct
2.7
n/a
8.4
n/a
Phrase Pivot+Conn
9.1
10.4
20.1
20.9
Phrase Pivot+Conn+Morph Auto
9.2
10.6
20.6
21.3
500K
Direct
5.9
n/a
15.1
n/a
Phrase Pivot+Conn
9.1
10.7
20.1
22.5
Phrase Pivot+Conn+Morph Auto
9.7
11.2
20.8
22.8
2M
Direct
9.7
n/a
20.4
n/a
Phrase Pivot+Conn
9.1
11.4
20.1
24.5
Phrase Pivot+Conn+Morph Auto
9.6
12.4
21.6
25.3
Table 6: Learning curve results of 100% (2M words), 25% (500K words) and 6.25% (125K
words) of the parallel Hebrew-Arabic corpus.

Dev/Test sets and over the Rule-based morphology constraints by 0.7 BLEU points in the Test
set.
Similar to the Rule-based constraints, the performance did not improve compared to thedirect model in the Dev set; but, again, the Test set showed a great improvement of 1.5 and1.2 BLEU points over the pivot and direct models, respectively. Also the system combinationof the pivot model with the direct model improves the overall performance. The model using
induced morphological features is the best performer with an increase in the performance gain
by 1.0/0.8 BLEU points in Dev/Test sets. This shows that the benefit we get from the induced
morphology constraints were not diluted when we do the model combination given the fact thatthe constraints were induced from the parallel data to start with.
It is important to note here that the induced morphology constraints outperformed therule-based constraints across all settings. This shows that the complex morphology constraints
extracted from the parallel data provide knowledge that can not be covered by simple linguistic
rules. However, the simple rule-based approach comes in handy when there is no data between
the source and target languages.

6.5

Learning Curve

In this experiment, we examine the effect of using less data in inducing morphology constraints
rules and the overall performance when we combine systems. Table 6 shows the results on alearning curve of 100% (2M words), 25% (500K words) and 6.25% (125K words) of the parallel
Hebrew-Arabic corpus.
As expected, The system combination between the direct translation models and thephrase-pivot translation model leads to an improvement in the translation quality across thelearning curve even when there is small amount of parallel corpora. Despite the weak performance (2.7 BLEU) of the direct system built on 6.25% of the parallel Hebrew-Arabic corpus,
the system combination leads to 1.4 BLEU points gain.
An interesting observation from the results is that we always get a performance gain fromthe induced morphology constrains across all settings. This shows that the system combination helps in adding more lexical translation choices while the constraints help in a different
dimension, which is selecting the best phrase pairs from the pivot system.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 112

Hebrew Source.!M‫ לדבר בפומבי על המחירי‬M‫ מסרבי‬M‫ והסוחרי‬M‫המתווכי‬the+middlemen and+the+traders refuse[m.p.] to+speak publicly about the+prices YmÌ '@ PAj.JË@ð ZA¢ñË@ ¯QK
PAªB@ á« AJÊ« IK

Arabic Reference

refuse[m.s.] the+middlemen and+the+traders the+speaking publicly about the+prices JË@ ¯QK !M‫ והסוחרי‬ZA¢ð
PAªB@ á« AJÊ« HYj

Phrase Pivot+Conn

middlemen !M‫ והסוחרי‬refuse[m.s.] the+speaking publicly about the+prices YmÌ '@ @ñ¯P PAj.JË@ð !M‫המתווכי‬PAªB@ úÎ« CÖÏ@ úÎ« IK

Direct!M‫ המתווכי‬and+the+traders refused[m.p.] the+speaking upon the+public about the+prices JË@ àñ¯QK !M‫ והסוחרי‬ZA¢ñË@
Phrase Pivot+Conn+
PAªB@ á« AJÊ« HYj
Morph Autothe+middlemen !M‫ והסוחרי‬refuse[m.p.] the+speaking publicly about the+prices JË@ @ñ¯P PAj.JË@ð ZA¢ð
Direct+Phrase Pivot+
PAªB@ á« AJÊ« HYj
Conn+Morph Auto middlemen and+the+traders refused[m.p.] the+speaking publicly about the+prices

Table 7: Translation examples.

7

Case StudyIn this section we consider an example from our Dev set that captures many of the patterns and themes in the evaluation. Table 7 shows a Hebrew source sentence and its Arabic reference. This is followed by the output from the pivot system, the direct system, thePhrase Pivot+Conn+Morph Auto system and the combined system.
Two particular aspects should be noted. First is the complementary lexical coverage of thedirect and pivot systems. This is seen in how one of each covers half of the phrase middlemen
and traders. The combined system captures both. Secondly, the gender, number and tense of themain verb prove challenging in many ways (and this is an issue for a majority of the sentencesin the Dev set). The Hebrew verb in the present tense is masculine and plural; and naturally
follows the subject. The Arabic reference verb appears at the beginning of the sentence, inwhich location it only agrees with the subject in gender (while number is singular). Arabic
Verbs in SVO order agree in gender and number. All the MT systems we compare leave theverb after the subject. The direct, Phrase Pivot+Conn+Morph Auto, and combination systems
get the number and gender correctly; however, the direct and combined system make the verb
tense past. The Phrase Pivot+Conn+Morph Auto example highlights the value of morphology
constraints; but the example points out that they sometimes are hard to evaluate automatically,
since there are morphosyntactically allowable forms that do not match the translation references.

8

Conclusion and Future WorkIn this paper, we presented the use of synchronous morphology constraint features based on
hand-crafted rules compared to rules induced from parallel data to improve the quality ofphrase-pivot based SMT. We show that the two approaches lead to an improvement in the translation quality. The induced morphology constraints approach is a better performer, however, it
relies on the fact there is a parallel corpus between source and target languages. We show positive results on Hebrew-Arabic SMT. We get 1.5 BLEU points over phrase-pivot baseline and
0.8 BLEU points over system combination baseline with direct model built from given paralleldata.
In the future, we plan to work on reranking experiments as a post-translation step basedon morphosyntactic information between source and target languages. We also plan to work on
word reordering between morphologically rich language to maintain the relationship between
the word order and the morphosyntactic agreement in the context of phrase pivoting.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 113

Acknowledgments
The work presented in this paper was possible thanks to a generous Google Research Award.
We would like to thank Reshef Shilon and Shuly Winter for helpful discussions and support withprocessing Hebrew. We also thank the anonymous reviewers for their insightful comments.

References
Adler, M. M. (2007). Hebrew morphological disambiguation: An unsupervised stochastic word-based
approach. PhD thesis, Ben-Gurion University of the Negev.
Bertoldi, N., Barbaiani, M., Federico, M., and Cattoni, R. (2008). Phrase-based statistical machine translation with pivot languages. Proceeding of IWSLT, pages 143–149.
Cettolo, M., Girardi, C., and Federico, M. (2012). Wit3 : Web inventory of transcribed and translated talks.
In Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT),
pages 261–268, Trento, Italy.
Cohn, T. and Lapata, M. (2007). Machine translation by triangulation: Making effective use of multiparallel corpora. In ANNUAL MEETING-ASSOCIATION FOR COMPUTATIONAL LINGUISTICS,
volume 45, page 728.
El Kholy, A. and Habash, N. (2010). Orthographic and Morphological Processing for English-Arabic
Statistical Machine Translation. In Proceedings of Traitement Automatique du Langage Naturel (TALN10). Montréal, Canada.
El Kholy, A. and Habash, N. (2010). Techniques for Arabic Morphological Detokenization and Orthographic Denormalization. In Proceedings of the seventh International Conference on Language
Resources and Evaluation (LREC), Valletta, Malta.
El Kholy, A. and Habash, N. (2011). Automatic Error Analysis for Morphologically Rich Languages. In
MT Summit XIII.
El Kholy, A. and Habash, N. (2014). Alignment symmetrization optimization targeting phrase pivot statistical machine translation. In Proceedings of the 17th annual conference of the European Association
for Machine Translation, EAMT 2014.
El Kholy, A., Habash, N., Leusch, G., Matusov, E., and Sawaf, H. (2013). Language independent connectivity strength features for phrase pivot statistical machine translation. In Proceedings of the 51st
Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages
412–418, Sofia, Bulgaria. Association for Computational Linguistics.
Elming, J. and Habash, N. (2009). Syntactic Reordering for English-Arabic Phrase-Based Machine Translation. In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages, pages 69–77, Athens, Greece.
Graff, D. (2007). Arabic Gigaword 3, LDC Catalog No.: LDC2003T40. Linguistic Data Consortium,
University of Pennsylvania.
Habash, N. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers.
Habash, N. and Hu, J. (2009). Improving Arabic-Chinese Statistical Machine Translation using English
as Pivot Language. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages
173–181, Athens, Greece.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 114

Habash, N. and Rambow, O. (2005). Arabic Tokenization, Part-of-Speech Tagging and Morphological
Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for
Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan.
Habash, N., Rambow, O., and Roth, R. (2009). MADA+TOKAN: A toolkit for Arabic tokenization,
diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. In Choukri,
K. and Maegaard, B., editors, Proceedings of the Second International Conference on Arabic Language
Resources and Tools. The MEDAR Consortium.
Habash, N. and Sadat, F. (2006). Arabic Preprocessing Schemes for Statistical Machine Translation. In
Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short
Papers, pages 49–52, New York City, USA.
Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic Transliteration. In van den Bosch, A.
and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods.
Springer.
Hajič, J., Hric, J., and Kubon, V. (2000). Machine Translation of Very Close Languages. In Proceedingsof the 6th Applied Natural Language Processing Conference (ANLP’2000), pages 7–12, Seattle.
Henriquez, C., Banchs, R. E., and Mariño, J. B. (2010). Learning reordering models for statistical machinetranslation with a pivot language.
Itai, A. and Wintner, S. (2008). Language resources for Hebrew. Language Resources and Evaluation,42(1):75–98.
Kathol, A. and Zheng, J. (2008). Strategies for building a Farsi-English smt system from limited resources.
In Proceedings of the 9th Annual Conference of the International Speech Communication Association(INTERSPEECH’2008), pages 2731–2734, Brisbane, Australia.
Khalilov, M., Costa-jussá, M. R., Fonollosa, J. A. R., Banchs, R. E., Chen, B., Zhang, M., Aw, A., Li, H.,
Mariño, J. B., Hernández, A., and Q., C. A. H. (2008). The talp & i2r smt systems for iwslt 2008. InInternational Workshop on Spoken Language Translation. IWSLT 2008, pg. 116–123.
Koehn, P., Birch, A., and Steinberger, R. (2009). 462 machine translation systems for europe. Proceedingsof MT Summit XII, pages 65–72.
Koehn, P. and Hoang, H. (2007). Factored translation models. In EMNLP-CoNLL, pages 868–876.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: open source
toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association
for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages
177–180, Prague, Czech Republic.
Koehn, P. and Schroeder, J. (2007). Experiments in domain adaptation for statistical machine translation.
In Proceedings of the Second Workshop on Statistical Machine Translation, pages 224–227. Association
for Computational Linguistics.
Lavie, A., Probst, K., Peterson, E., Vogel, S., Levin, L., Font-Llitjos, A., and Carbonell, J. (2004). A trainable transfer-based machine translation approach for languages with limited resources. In Proceedingsof European Association for Machine Translation Workshop on Broadening horizons of machine translation and its applications, Malta.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 115

Maamouri, M., Bies, A., Buckwalter, T., and Mekki, W. (2004). The Penn Arabic Treebank: Building aLarge-Scale Annotated Arabic Corpus. In NEMLAR Conference on Arabic Language Resources andTools, pages 102–109, Cairo, Egypt.
Miura, A., Neubig, G., Sakti, S., Toda, T., and Nakamura, S. (2015). Improving pivot translation by remembering the pivot. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2:
Short Papers), pages 573–577, Beijing, China. Association for Computational Linguistics.
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the 41st
Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association
for Computational Linguistics.
Och, F. J. and Ney, H. (2003). A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19–52.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of
Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational
Linguistics, pages 311–318, Philadelphia, PA.
Shilon, R., Habash, N., Lavie, A., and Wintner, S. (2010). Machine translation between hebrew and arabic:
Needs, challenges and preliminary solutions. In Proceedings of AMTA.
Shilon, R., Habash, N., Lavie, A., and Wintner, S. (2012). Machine translation between Hebrew andArabic. Machine Translation, 26:177–195.
Singh, N. and Habash, N. (2012). Hebrew morphological preprocessing for statistical machine translation.
In 16th annual conference of the European Association for Machine Translation (EAMT), Trento, Italy.
Stolcke, A. (2002). SRILM - an Extensible Language Modeling Toolkit. In Proceedings of the International Conference on Spoken Language Processing (ICSLP), volume 2, pages 901–904, Denver, CO.
Tsvetkov, Y. and Wintner, S. (2010). Automatic acquisition of parallel corpora from websites with dynamic
content. In Proceedings of the Seventh conference on International Language Resources and Evaluation(LREC’10), pages 3389–3392.
Utiyama, M. and Isahara, H. (2007). A comparison of pivot methods for phrase-based statistical machinetranslation. In Human Language Technologies 2007: The Conference of the North American Chapterof the Association for Computational Linguistics; Proceedings of the Main Conference, pages 484–491,
Rochester, New York. Association for Computational Linguistics.
Wu, H. and Wang, H. (2009). Revisiting pivot language approach for machine translation. In Proceedingsof the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 154–162, Suntec, Singapore. Association
for Computational Linguistics.
Yeniterzi, R. and Oflazer, K. (2010). Syntax-to-morphology mapping in factored phrase-based statisticalmachine translation from english to turkish. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 454–464, Uppsala, Sweden. Association for Computational
Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 116

Using Joint Models for Domain Adaptation in
Statistical Machine Translation
Nadir Durrani Hassan Sajjad Shafiq Joty Ahmed Abdelali Stephan Vogel
Qatar Computing Research Institute – Hamad Bin Khalifa University{ndurrani,hsajjad,sjoty,aabdelali,svogel}@qf.org.qa

Abstract
Joint models have recently shown to improve the state-of-the-art in machine translation (MT).
We apply EM-based mixture modeling and data selection techniques using two joint models,
namely the Operation Sequence Model or OSM — an ngram-based translation and reorderingmodel, and the Neural Network Joint Model or NNJM — a continuous space translation model,
to carry out domain adaptation for MT. The diversity of the two models, OSM with inherit
reordering information and NNJM with continuous space modeling makes them interesting to
be explored for this task. Our contribution in this paper is fusing the existing known techniques(linear interpolation, cross-entropy) with the state-of-the-art MT models (OSM, NNJM). On a
standard task of translating German-to-English and Arabic-to-English IWSLT TED talks, weobserved statistically significant improvements of up to +0.9 BLEU points.

1

Introduction

Parallel data required to train Statistical Machine Translation (SMT) systems is often inadequate, and is typically collected opportunistically from wherever it is available. The conventional wisdom is that more data improves the translation quality. Additional data however, may
not be best suited for tasks such as translating TED talks (Cettolo et al., 2014) or patents (Fujii
et al., 2010) or educational content (Abdelali et al., 2014), and often come with the challenges ofdealing with word-sense ambiguities and stylistic variance of other domains. When additionaldata, later referred as out-domain, is much larger than in-domain, the resultant distribution can
get biased towards out-domain, yielding a sub-optimal system. Domain adaptation aims to preserve the identity of the in-domain data while using the best of the out-domain data. This isdone by selecting a subset from the out-domain data, which is closer to the in-domain (Matsoukas et al., 2009; Moore and Lewis, 2010), or by re-weighting the probability distribution infavor of the in-domain data (Foster and Kuhn, 2007; Sennrich, 2012).
Bilingual sequence models (Mariño et al., 2006) have shown to be effective in improving the quality of machine translation and have achieved state-of-the-art performance recently(Le et al., 2012; Durrani et al., 2013; Devlin et al., 2014). Their ability to capture non-local
dependencies makes them superior to the traditional phrase-based models, which do not consider contextual information across phrasal boundaries. Two such models that we explore inthis paper are (i) the Operation Sequence Model or OSM (Durrani et al., 2011) — a markov
translation model that integrates reordering, and (ii) the Neural Network Joint Model or NNJM
(Devlin et al., 2014) — a continuous space model that learns neural network over augmented
streams of source and target sequences. Both models are used as additional language model(LM) features inside the SMT decoder.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 117

The diversity of the two models, i.e., OSM with embedded reordering information and
NNJM with continuous space modeling, makes them interesting to be explored for domain
adaptation. The LM-like nature of the two models provides motivations to apply methods suchas perplexity optimization for model weighting and cross-entropy-based ranking for data selection. In this paper, we explore both avenues. Firstly, we train models (OSM and NNJM) fromeach domain separately and then interpolate them (i) linearly using Expectation-Maximization
or EM-based weighting, (ii) using log-linear model inside the SMT pipeline. Secondly, weuse cross-entropy difference (Moore and Lewis, 2010) between in- and out-domain models toperform data selection for domain adaptation.
The bilingual property of the OSM and NNJM models gives them an edge over traditional
LM-based methods, which do not capture source and target domain relevance jointly. The
embedded reordering information modeled in OSM helps it to preserve reordering characteristicof the in-domain data. Capturing reordering variation across domains have been shown to bebeneficial also by Chen et al. (2013a). NNJM adds a different dimension to it by semanticallygeneralizing the data using distributed representation of words (Bengio et al., 2003).
We evaluated our systems on a standard task of translating IWSLT TED talks for Germanto-English (DE-EN) and Arabic-to-English (AR-EN) language pairs. Below is a summary of
our main findings:
Model Weighting:• Linearly interpolating OSM models through EM-based weighting gave average BLEU (Papineni et al., 2002) improvements of up to +0.6 for DE-EN and +0.9 for AR-EN.
• Log-linear variant performed better in the case of NNJM giving an average improvementsof +0.4 BLEU points for DE-EN and +0.5 for AR-EN.
• Linear interpolation for NNJM models was slightly behind its log-linear variant.
Data Selection:• OSM-based selection performed better for AR-EN task giving an average improvement of+0.7• NNJM performed better at the DE-EN task giving an average improvement of +0.6 points.
• Both OSM- and NNJM-based selection gave slightly better results than Modified-MooreLewis (MML) selection (Axelrod et al., 2011).
The rest of the paper is organized as follows. Section 2 briefly describes the OSM and the
NNJM models. Section 3 describes mixture model and data selection techniques that we apply
using the OSM and the NNJM models to carry out adaptation. Section 4 presents the results.
Section 5 discusses related work and Section 6 concludes the paper.

2

Joint Sequence Models

In this section, we revisit Operation Sequence and Neural Network Joint models briefly.
2.1

Operation Sequence Model

The Operation Sequence Model (OSM) is a bilingual model that couples translation and reordering by representing them as a sequence of operations. An operation either generates source

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 118

and/or target word(s) or performs reordering by inserting gaps and jumping forward and backward. A bilingual sentence pair (T, S) and its word-alignment A is transformed deterministically to a heterogeneous sequence of translation and reordering operations (o1 , o2 , . . . , oJ ). A
Markov model is then learned over these sequences:Posm (T, S) = P (o1 , .., oJ ) ≈JYP (oj |oj−n+1 ...oj−1 )j=1

For example, the German-English sentence pair shown in Figure 1 can be converted into thefollowing sequence of operations:
Generate (Wir, We) – Generate (haben, have) – Insert Gap – Generate (genommen, taken) –Jump Back (1) – Generate (sie, them) – Generate (aus, out) – Generate (ihrer , of their) –Generate (ursprünglichen, natural) – Generate (Pyramids, pyramid)

Figure 1: Sample German-English Sentence with Alignments
The generation is carried out in the order of target (English in this case). Gaps and jumps areinserted on the source side. Unaligned source and target words are handled through Generate
Source Only and Generate Target Only operations, respectively. Discontinuous source andtarget units are handled through other operations; see Durrani et al. (2011) for details about theoperations and the algorithm to convert a word-aligned corpus into sequences of operations.
Mixing lexical generation and reordering, each (translation or reordering) decision conditions on n−1 previous (translation or reordering) decisions. This allows the model to learn veryrich translation and reordering patterns. Moreover, the model is based on minimal translation
units (MTUs) and considers source and target contextual information across phrasal boundaries, thus addressing phrasal independence assumption and spurious segmentation problems in
traditional phrase-based MT.
2.2

Neural Network Joint Model

In recent years, there has been a great deal of effort dedicated to neural networks (NNs) andword embeddings with applications to MT and other areas in NLP (Bengio et al., 2003; Auli
et al., 2013; Kalchbrenner and Blunsom, 2013; Gao et al., 2014; Schwenk, 2012; Collobert
et al., 2011; Mikolov et al., 2013; Socher et al., 2013; Hinton et al., 2012). A bilingual Neural
Network Joint model for MT was recently proposed by Devlin et al. (2014). It learns a feedforward neural network from augmented streams of source and target sequences. For a bilingual
sentence pair (S, T ), NNJM defines a conditional probability distribution:
P (T |S) ≈|T |YP (ti |ti−1 . . . ti−n+1 , si )i=1where, si is an m-word source window for a target word ti based on the one-to-one alignment
between T and S. Each input word in the context has a D dimensional (continuous-valued)

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 119

vector representation in the shared look-up table L ∈ R|Vi |×D , where Vi is the input vocabulary.
The context of the sequence is represented by a concatenated vector xn ∈ R(m+n−1)D , whichis then passed through non-linear hidden layers to learn a high-level representation. The output
layer is a softmax over the output vocabulary Vo :exp (wkT φ(xn ))P (yn = k|xn , θ) = P|V |oT
m=1 exp (wm φ(xn ))where φ(xn ) defines the non-linear transformations of xn , and wk are the weights from theoutermost hidden layer to the output layer. By setting m and n to be sufficiently large, NNJM
can capture long-range cross-lingual dependencies between words.

3

Domain Adaptation

The ability to learn rich lexical and reordering patterns by OSM, the generalization power of
NNJM, and their strong empirical results in MT gives us a strong motivation to use them for the
problem of domain adaptation. However, the OSM and NNJM models trained on a plain concatenation of in-domain data with large and diverse multi-domain data are suboptimal. Whenother domains are sufficiently larger and/or different than the in-domain, the probability distribution can skew away from the target domain resulting in poor performance. The goal indomain adaptation is to do restrict this drift while still using the best of the available data.
We analyze the operation corpus as generated by the corpus conversion algorithm of Durrani et al. (2011) in OSM training. It provides useful insights on the amount of reordering,number of (source word) insertions and (target word) deletions that are carried out in the bilingual corpus. We use this information to motivate our study. Table 1 shows some statistics
about the operations in several datasets. We report probabilities of Jumps (Jump Forward andJump Back (*) operations), Gaps (Insert Gap operation), Insertions of source words (Generate Source Only (X) operation to handle unaligned source words) and Deletions of targetwords (Generate Target Only (Y) operation to handle unaligned target words) in each domain.
Domain

Jumps

Gaps

Deletions

Insertions

German-to-English
iwslt
news
europarlcommon crawl

0.17
0.21
0.22
0.19

0.09
0.13
0.14
0.11

0.06
0.05
0.07
0.12

0.04
0.07
0.06
0.11

0.07
0.07

0.05
0.08

Arabic-to-English
iwslt
UN

0.17
0.21

0.09
0.12

Table 1: Probabilities of Jumps, Gaps, Insertion and Deletion operations in each domain.
The probabilities of Jumps and Gaps in the in-domain IWSLT data are lower than other domains
in both German-to-English and Arabic-to-English language pairs. This indicates that lesser
amount of reordering is required in the in-domain data. Because other domains are significantly
larger than the in-domain data, the resulting distribution would get biased towards doing morereordering than desired. For example Insert Gap operation in Europral and UN data is muchprobable than IWSLT (compare column Gaps in Table 1). Similarly the probability of insertions
carried out in the in-domain data is less than the other domains. Therefore, the resulting models

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 120

would favor more insertions than preferred by the in-domain data. Table 1 does not show
statistics on different vocabularies, but lexical variance between domains is obviously anothercause of divergence from the in-domain data, which previous methods have also tackled. In
this work, we additionally address the reordering variance across domains. These statistics,
although, collected from the operation corpus on which the OSM model is trained, can bereflected on the NNJM training as well which uses same word-alignments to generate the streamof source and target n-grams.
In this paper we study two directions to perform domain adaptation in MT. We apply mixture modeling, a well-established model weighting technique, to re-weight the models in favorof the in-domain data. More specifically, we first train OSM and NNJM models on different
domains and then use an EM-based interpolation to optimize the weights based on an in-domain
tuning set. We also use the two models to rank sequences for data selection using cross entropy
difference. In the next two subsections we discuss these in detail.
3.1

Model Weighting

We use both OSM and NNJM models as an additional language model feature inside the decoder. A domain-adapted version of the model, biased towards the in-domain data, can help
assigning higher scores to the hypotheses that represent lexical choices and reordering patterns
preferred by the in-domain data. We train OSM and NNJM models from each domain separately and learn the relative weights of the models using linear and log-linear interpolation
methods. For linear interpolation, we compute weights by optimizing perplexity on in-domain
tuning set1 using a standard EM-based algorithm as described below:
Model Weighting by EM: Let θd ∈ {θ1 , . . . θD } represent a model (e.g., OSM, NNJM)
trained on domain d, where D is the total number of domains. The probability of a sequencexn can be written as a mixture of D probability densities, each coming from a different model:
P (xn |θ, λ) =DXP (xn |zn = d, θd ) λdd=1

where P (xn |zn = d, θd ) represents the probability of xn assigned by model θd , and the mixturePDweights λd satisfy 0 ≤ λd ≤ 1 and d=1 λd = 1. In our setting, θ = {θ1 , . . . θD } is known,
and we can use EM to learn the mixture weights. The expected complete data log likelihood isgiven by:E[L(λ)] =N X
DX

rnd log [P (xn |zn = d, θd )λd ]n=1 d=1

where rnd = P (zn = d|xn , θd , λt−1d ) is the responsibility that domain d takes for data pointn given the mixing weight in the previous step λdt−1 . In the E-step, we compute rnd and we
update λ in the M-step. More specifically:
λt−1 P (x |z =d,θ )dt
E-step: Compute rnd= PD d λt−1 Pn(xn |z =d,θ)d0 =1M-step: Update λtd = N1

d0

n

n

d0

PN

t
n=1 rndOnce we have learned the relative weights of the models based on the in-domain tuning data,
we can linearly interpolate the models as:
1 The tuning-set is required to be word-aligned and then converted into a sequence of operations (for OSM) andaugmented streams of source and target strings (for NNJM) to compute model-wise perplexities.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 121

Posm (T, S) ≈J X
Yλd P (oj |oj−n+1 ...oj−1 , θd )j=1 dPnnjm (T |S) ≈

|T |
Y
X

λd P (ti |ti−1 . . . ti−n+1 , si , θd )i=1 d

An alternative way to combine the models is through log-linear interpolation by optimizing
weights, directly on BLEU, along with other features inside of the SMT pipeline.
3.2

Data Selection

An alternative to model weighting is data selection, which attempts to filter out harmful datafrom the training corpus rather than down weighting it. Data selection could be useful in a
scenario with memory constraints. However, a down-side of this approach is that it requires
extensive amount of experimentation to find an optimal cut-off point.
In this paper, we select data using differences in cross entropy as proposed by Moore andLewis (2010). More specifically, we first train a model (OSM or NNJM) on the in-domain
corpus, and then train another model on the out-domain data of equal size. Then we score theout-domain data using:score(x) = HI (x) − HO (x)where x is a sequence of operations (o1 , . . . , on ) in the case of OSM and an augmented streamof source and target sequences (t1 , . . . , tn , si ) in the case of NNJM. HD is the cross-entropy
between a model and the empirical n-gram distribution in the domain D. We train a 5-gram
OSM and a 14-gram NNJM with 5-grams on target-side and 4-grams on each side of the source
word that is aligned with the target word ti . The bilingual characteristic of the models makes it
comparable to the MML method which trains source- and target-side language models from inand out-domains separately and take a sum of cross-entropy differences over each side of the
corpus:score(s, t) = [HI−src (s) − HO−src (s)] + [HI−tgt (t) − HO−tgt (t)]where s and t are sequences of source and target strings respectively. Out-domain models aretrained by randomly selecting corpora of same size as that of the in-domain data.

4

Experiments

Data: We used TED talks (Cettolo et al., 2014) as our in-domain corpus. For German-toEnglish (DE-EN), we used the data made available for WMT’14.2 This contains News, Europarl
and Common Crawl as out-domain data. For Arabic-English (AR-EN), we used the UN corpus
as out-domain data. We concatenated dev- and test-2010 for tuning and used test2011-2013 forevaluation. Table 2 shows the size of the training and test data used.
NNJM Settings: The NNJM models were trained using NPLM3 toolkit (Vaswani et al., 2013)
with the following settings. We used a target context of 5 words and an aligned source windowof 9 words, forming a joint stream of 14-grams for training. We restricted source and target side
vocabularies to 20K and 40K most frequent words. We used an input embedding layer of 150
2 http://www.statmt.org/wmt14/
3 http://nlg.isi.edu/software/nplm/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 122

German-English

Arabic-English

Corpus

Sent.

TokDE

TokEN

Corpus

Sent.

TokAR

TokEN

iwslt
news
ep
cc

177K
200K
1.9M
2.3M

3.3M
5.1M
48.7M
53.9M

3.5M
5.0M
51.0M
57.5M

iwslt
un
-

186K
3.7M
-

2.7M
12.4M
-

1.8M
12.3M
-

Test Set

Sent.

TokDE

TokEN

Corpus

Sent.

TokAR

TokEN

tune
test-11
test-12test-13

2452
1433
1700
1363

42K
22K
25K
19K

44K
23K
26K
20K

tune
test-11
test-12test-13

2456
1199
1702
1169

48K
21K
30K
26K

52K
24K
32K
28K

Table 2: Statistics of the German-English and Arabic-English training corpora in terms of Sentences and Tokens (Source/Target). Tokens are represented in Millions. ep = Europarl, cc =Common Crawl, un = United Nations
and an output embedding layer of 750. Only one hidden layer is used with NCE4 to allow fastertraining and decoding. Training was done using mini-batch size of 1000 and using 100 noise
samples. We train the out-domain NNJM models using the same vocabulary as the in-domain
vocabulary. All models were trained for 25 epochs.
Machine Translation Settings: We followed Birch et al. (2014) to train a Moses system
Koehn et al. (2007) with the following settings: maximum sentence length of 80, Fast-Align(Dyer et al., 2013) for word-alignments, an interpolated Kneser-Ney smoothed 5-gram language
model (Schwenk and Koehn, 2008) with KenLM (Heafield, 2011) for querying, lexicalized reordering (Galley and Manning, 2008) and other default parameters. We used Moses implementations of OSM and NNJM as a part of their respective baseline systems. Arabic OOVs weretranslated using an unsupervised transliteration module (Durrani et al., 2014b) in Moses. We
used k-best batch MIRA (Cherry and Foster, 2012) for tuning.5
4.1

Results: Model Weighting

We first discuss the results of applying mixture modeling approach. The MT systems are trainedon a concatenation of all in- and out-domain data. The OSM and NNJM models used in baseline
MT systems were also trained on the concatenated data.
Linear interpolation (OSMln ) based on EM-weighting shows significant improvementswith average BLEU gains of +0.6 in DE-EN and +0.9 in AR-EN over the baseline system Bcat
(see Table 3).6 One reason for better gains in AR-EN is the fact that the out-domain UN data
4 Training NNJM with backpropagation could be prohibitively slow because for each training instance, the softmax

layer requires a summation over the entire output vocabulary One way to avoid this repetitive computation is to use
a Noise Contrastive Estimation or NCE (Gutmann and Hyvärinen, 2010) of the loss function. NCE has been recently
used in neural language models (Vaswani et al., 2013; Mnih and Teh, 2012).
5 All systems were tuned three times.
6 We carried out additional experiments by linearly interpolating class-based OSM models Durrani et al. (2014a).
We used the mkcls utility in GIZA to cluster source and target vocabularies into 50 classes. Class-based OSM models
were trained on each domain and interpolated in the same way as we did for the word forms. This however, did not yield
any significant improvements on top of what was already achieved from the interpolation of word-based OSM. We also
tried interpolating POS, morph and lemma-based OSM-models but did not gain any further improvement. Results areommitted from the paper.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 123

OSM Interpolation (German-English)
System

test11

test12

test13

Avg.

Bcat

35.8

31.1

27.6

31.5

OSMln
OSMlg

36.6 +0.8
35.4 -0.4

31.9 +0.8
31.1 ± 0.0

27.7 +0.1
27.4 -0.2

32.1 +0.6
31.3 -0.2

OSM Interpolation (Arabic-English)
Bcat

26.4

29.2

29.9

28.5

OSMln
OSMlg

27.3 +0.9
25.8 -0.6

30.0 +0.8
28.7 -0.5

30.8 +0.9
29.4 -0.5

29.4 +0.9
28.0 -0.5

Table 3: OSM Interpolation OSMln = Linear, OSMlg = Log-linear
NNJM Interpolation (German-English)
System

test11

test12

test13

Avg.

Bcat

35.6

31.3

27.4

31.4

NNJMln
NNJMlg36.2 +0.6
36.1 +0.5

31.8 +0.5
32.1 +0.8

27.1 -0.3
27.2 -0.2

31.7 +0.3
31.8 +0.4

NNJM Interpolation (Arabic-English)
Bcat

26.6

29.4

30.1

28.7

NNJMln
NNJMlg26.7 +0.1
26.8 +0.2

30.2 +0.8
30.2 +0.8

30.3 +0.2
30.5 +0.4

29.1 +0.4
29.2 +0.5

Table 4: NNJM Interpolation NNJMln = Linear, NNJMlg = Log-linear

is much harmful for the task at hand. On the contrary additional data in DE-EN is helpful (seealso the results in next section for more information). Log-linear interpolation of OSM models(OSMlg ) performs much worse than Bcat in both language pairs. In the log-linear model, all
sub-models are queried separately. An operation sequence from the out-domain data that is
unknown to the in-domain OSM, gets high probability7 and is ranked higher in the search space.
On the contrary, the same gets down-weighted in a linearly interpolated global model.
Both linear and log-linear interpolation of the NNJM models showed improvements overthe baseline system Bcat (refer to Table 4). Log-linear interpolation (NNJMlg ) performed
slightly better in both cases. Notice that NNJMlg does not face the same problem as OSMlg
because all NNJM models are trained using the in-domain vocabulary with a low probability assigned to the out-domain UNKs.8 See Joty et al. (2015) for more details on our novel handling
7 Due to probability mass assigned to UNK sequences.
8 In order to reduce the training time and to learn better word representations, neural models are trained on mostfrequent vocabulary words only and low frequency words are represented under a class of unknown words, unk. This
results in a large number of n-gram sequences containing at least one unk word and thereby, makes unk a highly
probable word for the model. As a result of this discrepancy, sentences with more number of unk words will beselected. To solve this problem we created a separate class for out-domain unko words. We train the in-domain modelby adding a few dummy sequences containing unko occurring on both source and target sides ensuring that out-domain
unknown words get minimal probabilities.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 124

German-English

%age

0%
5%
10%
20%
40%
100%

Arabic-English

MML

OSM

NNJM

MML

OSM

NNJM

35.4
36.0
36.2
36.4
36.3
35.6

35.4
36.0
36.3
36.8
36.6
35.6

35.4
36.2
36.5
36.9
36.7
35.6

27.2
27.6
26.9
26.8
26.6
26.6

27.2
27.7
27.3
27.0
26.8
26.6

27.2
27.6
27.1
27.0
26.6
26.6

Table 5: MML, OSM and NNJM-based data selection, evaluated using test2011
Data Selection (German-English)
System

test11

test12

test13

Avg.

B100%
B0%

35.8
35.4

31.1
31.3

27.6
25.5

31.5
30.7

MML20%

36.4 +0.6

31.4 +0.3

27.7 +0.1

31.8 +0.3

OSM20%
NNJM20%36.8 +1.0
36.9 +1.1

31.5 +0.4
31.6 +0.5

27.7 +0.1
27.7 +0.1

32.0 +0.5
32.1 +0.6

Data Selection (Arabic-English)
B100%
B0%

26.4
27.2

29.2
30.0

29.9
30.2

28.5
29.1

MML5%

27.6 +0.4

30.5 +0.5

31.0 +0.8

29.7 +0.6

OSM5%
NNJM5%

27.7 +0.5
27.6 +0.4

30.6 +0.6
30.5 +0.5

31.0 +0.8
31.1 +0.9

29.8 +0.7
29.7 +0.6

Table 6: Data Selectionof UNK words in the NNJM model.
4.2

Results: Data Selection

We selected 0%, 2.5%, 5%, 10%, 20%, 40% and 100% out-domain data and evaluated on
test2011 to select the best percentage. See Table 5 for results on each selected percentage.
Table 6 shows that the out-domain data is helpful in the case of DE-EN and harmful in thecase of AR-EN; compare B100% (all data) versus B0% (in-domain data only). MML-selection
improves the baseline by +0.3 and +0.6 in case of DE-EN and AR-EN respectively. OSM and
NNJM-based selection gave similar improvements with slightly better results than MML. We
found that the amount of overlap in data selected by the three models is roughly 63% in DE-EN
and 71% in AR-EN.

5

Related Work

Previous work on domain adaptation in MT can be broken down broadly into two main categories namely data selection and model adaptation.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 125

5.1

Data Selection

Data selection has shown to be an effective way to discard poor quality or irrelevant training
instances, which when included in the MT systems, hurts its performance. The idea is to scorethe out-domain data using model trained from the in-domain data and apply a cut-off based on
the resulting scores. The MT system can then be trained on a subset of the out-domain data thatis closer to in-domain. Selection based methods can be helpful to reduce computational costwhen training is expensive and also when memory is constrained. Data selection was earlier
done for language modeling using information retrieval techniques (Hildebrand et al., 2005) andusing perplexity measure (Moore and Lewis, 2010). Axelrod et al. (2011) further extended thework of Moore and Lewis (2010) to translation model adaptation by using both source side andtarget side language models. Duh et al. (2013) used recurrent neural network language model
instead of an ngram-based language model to do the same. Translation model features wereused recently by Liu et al. (2014); Hoang and Sima’an (2014) to do data selection.
5.2

Model Adaptation

The downside of data selection is that finding an optimal cut-off threshold is a time consuming
process. Therefore rather than filtering less useful data, an alternative way is to down-weight it
and boost the data closer to the in-domain. It is robust than selection since it takes advantageof the complete out-domain data with intelligent weighting towards the in-domain. Matsoukas
et al. (2009) proposed a classification-based sentence weighting method for adaptation. Foster
et al. (2010) extended this by weighting phrases rather than sentence pairs. Other researchers
have carried out weighting by merging phrase-tables through linear interpolation (Finch andSumita, 2008; Nakov and Ng, 2009) or log-linear combination (Foster and Kuhn, 2009; Bisazza
et al., 2011; Sennrich, 2012) and through phrase training based adaptation (Mansour and Ney,
2013). Chen et al. (2013b) used vector space model for adaptation at phrase level. Every phrase
pair is represented as a vector where every entry in the vector reflects its relatedness with eachdomain. Chen et al. (2013a) also applied mixture model adaptation for reordering model. Joty
et al. (2015) performed model weighting by regularizing the loss function towards the in-domain
model directly inside neural network training. They also used NNJM model as their basis.
Other work on domain adaptation includes but not limited to studies that focus on topic
modeling (Eidelman et al., 2012; Hasler et al., 2014), dynamic adaptation where no in-domain
data is available (Sennrich et al., 2013; Mathur et al., 2014) and sense disambiguation (Carpuat
et al., 2013).

6

Conclusion

We targeted an unexplored area of using bilingual language models for domain adaptation. We
applied model weighting and data selection techniques using OSM and NNJM models. Bothmethods were shown to be effective in the target translation tasks. Interpolating multi-domain
models gave an average improvement of up to +0.9 BLEU points using OSM and +0.5 using
NNJM. We also used NNJM and OSM models for data selection using differences in crossentropy and showed improvements of up to +0.6 BLEU points. The code will be contributed toMoses git repository.

References
Abdelali, A., Guzman, F., Sajjad, H., and Vogel, S. (2014). The AMARA corpus: Building
parallel language resources for the educational domain. In LREC’14, Reykjavik, Iceland.
Auli, M., Galley, M., Quirk, C., and Zweig, G. (2013). Joint language and translation modeling

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 126

with recurrent neural networks. In EMNLP-2013, pages 1044–1054, Seattle, Washington,
USA. ACL.
Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection. EMNLP ’11, pages 355–362, Edinburgh, UK. ACL.
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language
model. J. Mach. Learn. Res., 3:1137–1155.
Birch, A., Huck, M., Durrani, N., Bogoychev, N., and Koehn, P. (2014). Edinburgh SLT and MT
system description for the IWSLT 2014 evaluation. In Proceedings of the 11th International
Workshop on Spoken Language Translation, IWSLT ’14, Lake Tahoe, CA, USA.
Bisazza, A., Ruiz, N., and Federico, M. (2011). Fill-up versus interpolation methods for phrasebased SMT adaptation. In Federico, M., Hwang, M.-Y., Rödder, M., and Stüker, S., editors,
IWSLT, pages 136–143.
Carpuat, M., Daume III, H., Henry, K., Irvine, A., Jagarlamudi, J., and Rudinger, R. (2013).
Sensespotting: Never let your parallel data tie you to an old domain. In ACL, pages 1435–
1445, Sofia, Bulgaria.
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., and Federico, M. (2014). Report on the
11th IWSLT Evaluation Campaign. Proceedings of the International Workshop on Spoken
Language Translation, Lake Tahoe, US.
Chen, B., Foster, G., and Kuhn, R. (2013a). Adaptation of reordering models for statisticalmachine translation. NAACL-HLT ’13, pages 938–946, Atlanta, Georgia.
Chen, B., Kuhn, R., and Foster, G. (2013b). Vector space model for adaptation in statisticalmachine translation. In ACL (Volume 1: Long Papers), Sofia, Bulgaria.
Cherry, C. and Foster, G. (2012). Batch tuning strategies for statistical machine translation.
NAACL-HLT ’12, Montréal, Canada.
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011).
Natural language processing (almost) from scratch. volume 12, pages 2493–2537. JMLR.
org.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast androbust neural network joint models for statistical machine translation. In ACL (Volume 1:
Long Papers), pages 1370–1380.
Duh, K., Neubig, Graham, S. K., and Tsukada, H. (2013). Adaptation data selection usingneural language models: Experiments in machine translation. In ACL (Volume 2: Short
Papers), Sofia, Bulgaria.
Durrani, N., Fraser, A., Schmid, H., Hoang, H., and Koehn, P. (2013). Can Markov Models
Over Minimal Translation Units Help Phrase-Based SMT? In Proceedings of the 51st Annual
Meeting of the Association for Computational Linguistics, pages 399–405, Sofia, Bulgaria.
Association for Computational Linguistics.
Durrani, N., Koehn, P., Schmid, H., and Fraser, A. (2014a). Investigating the Usefulness ofGeneralized Word Representations in SMT. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 421–432,
Dublin, Ireland. Dublin City University and Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 127

Durrani, N., Sajjad, H., Hoang, H., and Koehn, P. (2014b). Integrating an Unsupervised
Transliteration Model into Statistical Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the ACL (EACL 2014), Gothenburg, Sweden.
Durrani, N., Schmid, H., and Fraser, A. (2011). A Joint Sequence Translation Model with
Integrated Reordering. In Proceedings of the Association for Computational Linguistics:
Human Language Technologies (ACL-HLT’11), Portland, OR, USA.
Dyer, C., Chahuneau, V., and Smith, N. A. (2013). A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of NAACL’13.
Eidelman, V., Boyd-Graber, J., and Resnik, P. (2012). Topic models for dynamic translation
model adaptation. ACL ’12, pages 115–119, Jeju Island, Korea. ACL.
Finch, A. and Sumita, E. (2008). Dynamic model interpolation for statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, Columbus,
Ohio.
Foster, G., Goutte, C., and Kuhn, R. (2010). Discriminative instance weighting for domain
adaptation in statistical machine translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 451–459, Cambridge, MA. ACL.
Foster, G. and Kuhn, R. (2007). Mixture-model adaptation for smt. In Proceedings of theSecond Workshop on Statistical Machine Translation, StatMT ’07.
Foster, G. and Kuhn, R. (2009). Stabilizing minimum error rate training. In Proceedings of the
Fourth Workshop on Statistical Machine Translation, StatMT ’09, Athens, Greece.
Fujii, A., Utiyama, M., Yamamoto, M., and Utsuro, T. (2010). Overview of the patent translation task at the ntcir-8 workshop. In In Proceedings of the 8th NTCIR Workshop Meeting on
Evaluation of Information Access Technologies: Information Retrieval, Question Answeringand Cross-lingual Information Access, pages 293–302.
Galley, M. and Manning, C. D. (2008). A Simple and Effective Hierarchical Phrase Reordering
Model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language
Processing, pages 848–856, Honolulu, Hawaii.
Gao, J., He, X., Yih, W.-t., and Deng, L. (2014). Learning continuous phrase representations
for translation modeling. In ACL (Volume 1: Long Papers), pages 699–709, Baltimore,
Maryland. ACL.
Gutmann, M. and Hyvärinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Teh, Y. and Titterington, M., editors, Proc. Int.
Conf. on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR W&CP, pages
297–304.
Hasler, E., Blunsom, P., Koehn, P., and Haddow, B. (2014). Dynamic topic adaptation forphrase-based mt. EACL, pages 328–337, Gothenburg, Sweden.
Heafield, K. (2011). KenLM: Faster and Smaller Language Model Queries. In Proceedings of
the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland,
United Kingdom.
Hildebrand, A. S., Eck, M., Vogel, S., and Waibel, A. (2005). Adaptation of the translation
model for statistical machine translation based on information retrieval. In EAMT, Budapest.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 128

Hinton, G., Deng, L., Yu, D., Dahl, G., rahman Mohamed, A., Jaitly, N., Senior, A., Vanhoucke,V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012). Deep neural networks for acoustic
modeling in speech recognition. Signal Processing Magazine.
Hoang, C. and Sima’an, K. (2014). Latent domain translation models in mix-of-domains
haystack. COLING: Technical Papers, Dublin, Ireland.
Joty, S., Sajjad, H., Durrani, N., Al-Mannai, K., Abdelali, A., and Vogel, S. (2015). How
to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models. In
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,
Lisbon, Portugal.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In EMNLP2013, Seattle, Washington, USA.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007).
Moses: Open source toolkit for statistical machine translation. (ACL’07), Prague, Czech
Republic.
Le, H.-S., Allauzen, A., and Yvon, F. (2012). Continuous space translation models with neuralnetworks. In Proceedings of the 2012 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pages 39–48,
Montréal, Canada. Association for Computational Linguistics.
Liu, L., Hong, Y., Liu, H., Wang, X., and Yao, J. (2014). Effective selection of translation modeltraining data. In ACL (Volume 2: Short Papers), pages 569–573, Baltimore, Maryland. ACL.
Mansour, S. and Ney, H. (2013). Phrase training based adaptation for statistical machine translation. NAACL-HLT ’13, Atlanta, Georgia.
Mariño, J. B., Banchs, R. E., Crego, J. M., de Gispert, A., Lambert, P., Fonollosa, J. A. R., andCosta-jussà, M. R. (2006). N-gram-Based Machine Translation. Computational Linguistics,
32(4):527–549.
Mathur, P., Venkatapathy, S., and Cancedda, N. (2014). Fast domain adaptation of smt models without in-domain parallel data. COLING 2014: Technical Papers, pages 1114–1123,
Dublin, Ireland.
Matsoukas, S., Rosti, A.-V. I., and Zhang, B. (2009). Discriminative corpus weight estimation
for machine translation. In Proceedings of the 2009 Conference on Empirical Methods in
Natural Language Processing: Volume 2, EMNLP ’09.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space.
Mnih, A. and Teh, Y. W. (2012). A fast and simple algorithm for training neural probabilistic
language models. In Proceedings of the International Conference on Machine Learning.
Moore, R. C. and Lewis, W. (2010). Intelligent selection of language model training data.
(ACL’10), Uppsala, Sweden.
Nakov, P. and Ng, H. T. (2009). Improved statistical machine translation for resource-poor
languages using related resource-rich languages. In EMNLP’09, Singapore.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 129

Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic
evaluation of machine translation. (ACL’02), Philadelphia, PA, USA.
Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machinetranslation. In Proceedings of COLING 2012: Posters, pages 1071–1080, Mumbai, India.
Schwenk, H. and Koehn, P. (2008). Large and Diverse Language Models for Statistical MachineTranslation. In International Joint Conference on Natural Language Processing, pages 661–
666.
Sennrich, R. (2012). Perplexity minimization for translation model domain adaptation in statistical machine translation. EACL, Avignon, France.
Sennrich, R., Schwenk, H., and Aransa, W. (2013). A multi-domain translation model framework for statistical machine translation. In ACL (Volume 1: Long Papers), pages 832–840,
Sofia, Bulgaria. ACL.
Socher, R., Bauer, J., Manning, C. D., and Andrew Y., N. (2013). Parsing with compositional
vector grammars. In ACL (Volume 1: Long Papers), pages 455–465, Sofia, Bulgaria.
Vaswani, A., Zhao, Y., Fossum, V., and Chiang, D. (2013). Decoding with large-scale neurallanguage models improves translation. In EMNLP-2013, pages 1387–1392.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 130

Machine translation evaluation made fuzzier:
A study on post-editing productivity and evaluation
metrics in commercial settings
Carla Parra Escartı́n
Manuel Arcedillocarla.parra@hermestrans.com
manuel.arcedillo@hermestrans.com

Hermes Traducciones
C/ Cólquide 6, portal 2, 3. I
28230 Las Rozas, Madrid, Spain

Abstract
In this paper, we report on an experiment carried out in the context of a translation company.
Ten translators, with diverse degrees of experience in translation and machine translation postediting (MTPE), were assigned the same task, involving translation from scratch, fuzzy-match
post-editing, and MTPE. We evaluate the MT output using traditional evaluation metrics such
as BLEU and TER, correlate these measures with productivity values and study whether afuzzy score stands up against them. Our main goal was to evaluate whether fuzzy scores canbe used for evaluating MTPE, thus incorporating its familiarity and TM matching analogies toan MTPE workflow. The results of our experiment seem to support this hypothesis.

1 Introduction
In the last years, machine translation post-editing (MTPE) tasks have become ever more common in the translation industry. Translators and translation companies of all sizes use machinetranslation (MT) to increase their productivity and reduce production costs. However, it is still
unclear how to assess MT output in order to verify that those goals have been met and, if applicable, determine a fair compensation for the post-editor.
Traditional automatic evaluation metrics such as BLEU (Papineni et al., 2001) and TER(Snover et al., 2006) have shortcomings such as unproven correlation with productivity gains(Callison-Burch et al., 2006), technical difficulties for their estimation by general users and lackof intuitiveness. Meanwhile, the translation industry has a well-established way of evaluatingtext similarity and establishing discount rates: translation memory (TM) fuzzy match scores(FMS). Based on text similarity with segments stored in the TM, each sentence to be translated
receives a fuzzy score, ranging from 0% (no similar sentence was found in the TM) to 100%(an exact match was found). In the translation industry, TM matches above 75% are normallyassigned a rate discount, while those segments below this threshold are paid the full rate following the general assumption that they do not yield any productivity increase.1 Since translatorsand other parties involved in a translation project are familiar with this system and accept it asa valid business model, we designed an experiment to verify whether target-side FMS stand up
1 This is an industry general practice.

Plitt and Masselot (2010) also report that in a typical localization scenario
TM matches below 75% are considered no-matches. However, lack of research and the peculiarities of fuzzy score
calculation in each tool may allow for variations of this model.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 131

against traditional methods of MTPE evaluation.
In the experiment, ten professional translators with diverse experience in the industry wereassigned the same file to translate, involving translation from scratch, TM match post-editing,and MTPE. We recorded the time spent by each translator in each segment in order to obtainproductivity values for all TM match bands, including no match segments and exact matches.
We then computed several automated metrics (namely, BLEU and TER) and correlated themwith productivity values. Finally, we compared the performance of these automated metrics
with the performance of the target-side FMS.
The remainder of this paper is structured as follows: Section 2 summarizes traditional MT
evaluation metrics (c.f. 2.1 and 2.2) and presents the metric we use in our experiments, explaining why we deem it appropriate for MT evaluation (c.f. 2.3). The details of the experiment
itself are explained in Section 3. In Section 4 we present the results and analyze them. Finally,in Section 5 we summarize the findings and discuss possible future research.

2 BackgroundA common practice when negotiating MTPE discounts is to either annotate a sample of the MT
output according to any of the methods described below or similar ones, or post-edit it in orderto generate a reference for automatic evaluation, while possibly gathering other data such as
time spent or key strokes. Since rates are normally set before handing off a project, a good MT
evaluation at this stage is critical to avoid underpayments and the mistrust this situation may
cause.
MT evaluation is a research field in itself. As stated by King et al. (2003), Yorick Wilks has
been credited with the famous remark that “more has been written about MT evaluation thanabout MT itself”. While it is not the purpose of this paper to revise all the proposed MT evaluation metrics, we deem it necessary to acknowledge the most commonly used metrics and assess
their usability from the translation industry point of view. Subsection 2.1 summarizes humanevaluation and Subsection 2.2 focuses in automatic evaluation metrics. Subsection 2.3 explains
the shortcomings of traditional evaluation metrics and presents the metric we additionally usedin our experiments: the target-side FMS.
2.1

Human evaluation

As explained by Koehn (2010), two main strategies are used in evaluation campaigns: fluency
and adequacy, and ranking of translations. When human judges are asked to assess the fluency
and adequacy of MT output, the task consists of assigning a score from one to five on thesetwo criteria. Fluency assesses whether the text is fluent in the target language. It refers togrammaticality, correctness and idiomatic word choices. Adequacy, on the other hand, assesses
whether the output sentence conveys the same meaning as the input sentence.
As pointed out by Koehn (2010), “these definitions are very vague, and it is difficult forevaluators to be consistent in their application”. Callison-Burch et al. (2007) also point thisout: “No instructions are given to evaluators in terms of how to quantify meaning, or how
many grammatical errors (or what sort) separates the different levels of fluency. Because ofthis many judges either develop their own rules of thumb, or use the scales as relative rather
than absolute.” To overcome these issues, the “translation ranking” approach may be taken. In
this kind of evaluation, human judges are given the output of different systems and are asked
to choose the best one. This approach is based on the judges’ perception of usefulness in termsof savings and productivity, rather than on usefulness itself. In a translation and productivity
test carried out by Autodesk in 2011,2 a clear mismatch between perception of productivity andactual productivity was found.
2 http://langtech.autodesk.com/productivity.html

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 132

An alternative evaluation methodology is the one proposed by Hurtado Albir (1995). Thismethodology is widely used in translator training courses. It distinguishes different types oferrors such as orthotypographical, grammatical, or semantic errors. More recent proposals are
the TAUS Dynamic Quality Framework (DQF),3 the Multidimensional Quality Metrics (MQM)
Framework proposed by the QTLaunchPad project (Burchardt and Lommel, 2014), and theirharmonized version, recently released (Lommel, 2015). However, since human evaluation (asopposed to automatic evaluation) is costly4 and time consuming, these methods are not always
feasible.
2.2 Automatic evaluation
Throughout the years, several automatic evaluation metrics have been proposed. These can begrouped in lexical, syntactic and semantic evaluation metrics depending on the linguistic levelat which they operate.
Lexical evaluation metrics assess the lexical similarity between the MT output and one orseveral references. These metrics assume that the usefulness of MT is related to its proximity
or similarity to the reference(s). This assumption is most informative when such reference(s)are post-edits of the MT output, rather than previous translations.
They can be further classified into metrics measuring the edit distance, the lexical precision, the lexical recall and the F-Measure. Edit distance is measured by WER (Word Error
Rate) (Nießen et al., 2000) and TER (Translation Edit Rate) (Snover et al., 2006, 2009). Lexicalprecision is measured by BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2001) and
NIST (Doddington, 2002). ROUGE (Lin and Och, 2004) assesses the lexical recall and PER(Position-independent Word Error Rate) (Tillmann et al., 1997) assesses the lexical precisionand recall. Finally, GT Me (Melamed et al., 2003) and METEOR (Lavie and Agarwal, 2005;
Denkowski and Lavie, 2010) are based on the F-Measure.
MT evaluation tools such as Asiya (Giménez and Màrquez, 2010) additionally offer syntactic and semantic similarity evaluation metrics. The syntactic metrics capture similarities
over shallow-syntactic structures, dependency relations and constituent parse trees. The semantic metrics intend to capture similarities over named entities, semantic roles and discourse
representations. Both the syntactic and the semantic metrics require the usage of additional
Language Resources and Tools (LRT), such as parsers, corpora, and specific packages.
All these metrics share the advantage of being fast and cheap to obtain. However, they still
remain obscure for laymen and professional translators not familiarized with MT evaluation
research, and typically are not designed with MTPE in mind.
2.3

Fuzzy match scores for MT evaluation

Although BLEU is a well-established evaluation metric and it is extensibly used in MT engine
development cycles and the evaluation of raw MT output as a final product, it has also received
criticism (Callison-Burch et al., 2006; Koehn, 2010; Bechara, 2013). A general rule of thumb in
MT evaluation is that BLEU scores above 30 reflect understandable translations, while scores
over 50 can be considered good and fluent translations (Lavie, 2010). However, the usefulnessof “understandable” translations is questionable in the case of MTPE, since post-editors do notdepend on MT to understand the meaning of the source text. Also, how should one interpret a
BLEU score improvement from 45 to 50 in terms of productivity? Taking another widely-used
metric, does a TER value of 40 justify any discount? The vast majority of translators (and even
MT researchers) would probably be unable to answer these questions. And yet, they would
probably instantly acknowledge that TM fuzzy matches of 60% are not worth editing, while
3 https://evaluate.taus.net/evaluate/about
4 See, for instance, Section 3, ”Costs”, in Burchardt and Lommel (2014).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 133

they would probably consider it fair to accept discounts for 80% matches. We believe MTPE
tasks would benefit greatly from the familiarity of source-side FMS applied to MTPE evaluation
as a target-side FMS.5
Organizations such as TAUS have already proposed alternative models of MT evaluation
which use FMS, such as the “MT Reversed Analysis”.6 Also, Zhechev (2012) created a metric combining character and word-based FMS and reported it as having the greatest correlation
with productivity gain. In this experiment, bearing in mind that each CAT tool uses a different(and generally unknown) algorithm for their source-side FMS, we use the fuzzy value computed by Okapi Rainbow7 in its Translation Comparison feature. We use this tool because it isfreely available and it natively supports the most common bilingual formats, hence potentially
improving transparency and usability for all parties involved in a translation project. This FMS
is based on the Sørensen-Dice coefficient (Sørensen, 1948; Dice, 1945) using 3-grams. Equation 1 illustrates how the FMS is computed by Okapi Rainbow. An important difference withother automatic metrics is that it does not depend on traditional tokenization: instead of word
n-grams, it computes character n-grams. Segments are split into a list of characters, which isthen used to compute 3-grams and string size.8
2 ∗ (M T output ∩ P E output)∗ 100
size M T output + size P E output)(1)

The MT output (or TM output in the case of TM matches) was compared against the postedited output of each translator. This means that each segment has ten separate scores, one foreach translator. It is also worth noting here that translators faced either MT output or TM output–or no suggestion at all in segments selected for translation from scratch–, so the original output
is unambiguous.

3 Experiment settings
The experiment was based on a pilot experiment (Parra Escartı́n and Arcedillo, 2015) which
seemed to suggest that fuzzy scores might be as good as or even better than BLEU and TER
scores for evaluating MTPE. Similarly to what Federico et al. (2012) did, we replicated a real
production environment.
Ten in-house translators were asked to translate the same file from English into Spanish.
We asked them to translate the text using one of their most common CAT tools, memoQ.9
This tool was chosen because it allows keeping track of the time spent in each segment. We
disregarded using other tools which also record time and other useful segment-level indicators,
such as keystrokes in PET (Aziz et al., 2012) and iOmegaT (Moran et al., 2014), or MTPE
effort in MateCat (Federico et al., 2012) because they are not part of the everyday tools of thetranslators involved. Translators were only allowed to use the TM, the terminology database andthe MT output included in the translation package. We disabled all other memoQ’s productivity
enhancing features such as predictive text, sub-segment leverage and automatic fixing of TM
matches to allow for better comparisons with translation environments which may not offer
similar features.
5 Hereinafter, FMS will refer exclusively to target-side FMS unless otherwise specified.
6 https://www.taus.net/think-tank/best-practices/postedit-best-practices/pricing-machine-translation-post-editing-guidelines
7 http://okapi.opentag.com/
8 For further information, see the open source code available at:okapiframework/.
9 The version used was memoQ 2015 build 3.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

https://bitbucket.org/

Miami, Oct 30 - Nov 3, 2015 | p. 134

3.1

Test set selection

The selection criteria for the text to be translated were the following:
1. Belong to a real translation request.
2. Originate from a client for which our company owned a customized MT engine.
3. Have a word volume capable of engaging translators for several hours.
4. Include significant word counts for each TM match band (i.e., exact matches, fuzzy
matches and no-match segments.
We used the word count analysis already available in our company’s archive to find atranslation project which met our criteria. In this case, the existing word count analysis camefrom SDL Trados Studio.10 The original source text had over 8,000 words and was part of asoftware user guide. To avoid skewing due to the inferior typing and cognitive effort required
to translate the second of two similar segments, repetitions and internal leverage segments were
filtered out.
When transferring the project to memoQ (the CAT tool chosen for the experiment), we
encountered a quite different word count. We expected some discrepancies in this new analysis,
but they turned out to be bigger than we preliminary thought. Table 1 shows the word countdistribution reported by both tools. As can be observed in Table 1, memoQ’s word count forthe 95–99% fuzzy band ended up being significantly smaller despite our efforts to select a text
with a more balanced distribution. This is evidence of a well-known problem in the translationindustry: the different algorithms used by each tool for computing word counts and calculating
fuzzy scores.

TM match100%
95–99%
85–94%
75–84%No MatchTotal

SDL Trados StudioWords Segments
1243
95
1044
55
747
43
608
42
3388
233
7030
468

memoQ
Words Segments
1226
94
231
21
1062
48
696
42
3804
263
7019
468

Table 1: Word count as computed by SDL Trados Studio and memoQ.
The differences in word count unfortunately created a less informative 95–99% fuzzy band.
On the other hand, the increased number of no-match words (from 3388 to 3804) provided
us with a more solid sample for comparing MTPE and translation throughputs. In order tomake this comparison, we randomly divided the no-match band in two halves using the test set
generator included in the m4loc package.11 . One half was translated from scratch, and the otherhalf was machine translated and subsequently post-edited.
3.2

Machine translation system used

To generate the raw MT output we used a customized Systran’s RBMT engine.12 This system was selected because it is the one we normally use for MTPE for this client. It can be
10 The version used was SDL Trados Studio 2014 SP1.
11 m4loc is an open source project initiative to “combine investments to efficiently leverage the potential that Moses

promises for localization” (Ruopp, 2010) For more information and access to the code see: https://code.
google.com/p/m4loc
12 Systran 7 Premium Translator was used.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 135

considered a mature engine since at the time of the experiment it had been subject to ongoingin-house customization for over three years and boasted a consistent record for productivity
enhancement. The customization includes dictionary entries, software settings, and pre- andpost-editing scripts. Although Systran includes a statistical machine translation (SMT) component, this was not used in our experiment. This decision was taken based on our past experience.
In our experiments using this component, it seemed to negate the strengths of RBMT engines(such as predictability and orthotipographic consistency), while not incorporating enough improvements to compensate for them. While we are aware of recent advances in this topic, thiscomponent is not part of our usual MTPE preparation workflow and thus we chose not to use itin our experiment.
3.3

Translators participating in the experiment

Ten translators were engaged in the experiment. Two carried out the task as part of a pilot
experiment (Parra Escartı́n and Arcedillo, 2015) and eight more were engaged with the aimof verifying the initial findings. A preliminary survey was sent to all translators to gather information on their experience. Translators were asked to provide their years of experience intranslation and MTPE, their experience translating and/or MTPE texts from this client, theiropinion on MT (positive or negative), and their opinion on CAT tools (positive or negative). Table 2 summarizes the results of our survey. Translators are sorted in descending order according
to their combined experience in translation and MTPE.

TR1
TR2
TR3
TR4
TR5
TR6
TR7
TR8
TR9
TR10

Trans. exp.
5
5
5.5
5
5
5
2
1.5
2
1

MTPE exp.
3
3
2
1
0
0
1
1.5
0
0

Client exp.
Yes
Yes
Yes
Yes
No
No
Yes
Yes
No
No

MT opinion
Positive
Positive
Positive
Negative
Positive
Positive
Positive
Positive
Negative
Positive

CAT opinion
Positive
Positive
Positive
Positive
Positive
Positive
Positive
Positive
Positive
Positive

Table 2: Overview of translator’s experience (measured in years) and opinion on MT and CAT.
As some translators may be biased against MT, we deemed it necessary to gather theirview on it as positive or negative, to see if this may also have an impact in the results. Only
two translators expressed that they did not like working with MT or MTPE, even though they
acknowledged that high quality MT output generally enhances their productivity.

4 Results and discussion
The ten packages received from the translators were analyzed individually in order to extract
the time spent in each segment and calculate automated evaluation metrics. Segments with exceptionally long times (above 10 minutes) and those with unnaturally high productivity (above
5 words per second) were considered outliers and were discarded for further analysis. The few
exceptionally long times registered can be explained by the translator not “closing” the segment during long pauses or lunch breaks. Since the two segments with highest recorded times(above one hour) belong to the translators who openly hold a grudge against MT, it is tempting
to interpret this as an attempt to pervert the results. However, since the rest of their samples
showed no other data anomalies and the time at which those exceptions were edited matches

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 136

translators’ normal lunch breaks, it can be safely assumed that those two cases were exceptions
due to carelessness rather than deliberate data tampering.
A possible explanation for the segments with unnaturally high productivity might be thatthe translator actually proofread the translation when having the cursor placed in the previous
segment. Some of these segments were 100% matches and thus it seems logical that they only
opened the segment to confirm it and move to the next one. However, we lack a way to confirm
this hypothesis.
Data series of Translators 5 and 8 also showed questionable results. Translator 5’s MT
post-edited output quality showed clear signs of under-editing, which would not have reachedthe minimum quality standards required by the client in a real project. This under-editing canalso be seen when comparing MTPE sample’s automated metrics across translators. As table
3 shows, values for Translator 5 differ significantly from the rest. Closer analysis of the fuzzy
and translation-from-scratch final text also showed significantly more errors than in the rest oftranslators. Although it may prove an interesting starting point for further research on MTPE
and translation errors, we have omitted the analysis of this data series for the purpose of thispaper.

BLEU
TER
FMS

TR-1
64.24
22.14
87.29

TR-2
66.37
20.98
87.74

TR-3
64.23
22.32
87.05

TR-4
63.40
24.52
86.30

TR-5
78.16
12.82
93.06

TR-6
69.57
18.89
88.53

TR-7
66.42
19.74
88.40

TR-8
63.56
21.40
87.62

TR-9
65.88
19.77
88.85

TR-10
68.43
18.84
89.09

Table 3: Automated metrics for the MTPE sample per translator.
Translator 8’s anomaly was detected when studying correlation values between automatedmetrics and productivity. In this case, it was significantly lower than for the rest of translators(see Table 9 in Section 4.3). Upon further enquiries, Translator 8 admitted to have enabled thepredictive text feature which had been originally disabled from the hand-off package. Again,
although extremely interesting for future research, for the purpose of this paper Translator 8’s
data series was discarded.
One last remark about possible anomalies is the generally low productivity of the 95–99%
TM match band when compared to contiguous TM match bands (Table 4). In the vast majorityof these segments there were no textual differences between the match stored in the TM and thecontent to be translated. This is illustrated in Example 2.
(2) Source in TM: If protection has been disabled manually, you can enable it in theapplication settings window.
Target in TM: Si se ha desactivado la protección manualmente, puede activarla en laventana de configuración de la aplicación.
New Source: If protection has been disabled manually, you can <1>enable it in theapplication settings window </1>.
New Target: Si se ha desactivado la protección manualmente, puede <1>activarla en laventana de configuración de la aplicación</1>.
The results obtained show that the time needed to perform these inline tag edits does notmatch what would be expected of an equivalent 95% TM match due to text differences. Even
though it is possible to adjust the fuzzy score assigned by CAT tools to these segments via
formatting penalties (as in SDL Trados Studio) or by using tag weights (as in memoQ), theimpact of inline tag editing in translation throughput has not yet been extensively researched.
For this reason, we have omitted the 95–99% band in all following analysis.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 137

4.1

Productivity report

Although a standard reference of 313–375 words per hour (2500–3000 words per day) is usuallytaken as a baseline for productivity comparisons, we calculated each translator’s translationfrom-scratch throughput individually. As expected, it varied greatly, ranging from 473 to 1701
words per hour (Table 4). It is remarkable that Translator 1 was even able to translate fromscratch faster than when post-editing 75–84% TM matches. In a real project, a rate discountwould have been applied to those matches, which apparently would have been unfair for thistranslator.
Not depending on standard reference values also avoided misleading interpretations of
MTPE throughputs. For example, comparing the average 1329 words per hour with a standard
reference of 375 words per hour would show productivity gains of around 250%. However,based on the translation-from-scratch throughputs obtained, none of our translators reachedsuch numbers, as can be seen in Table 5, which shows a more realistic average gain of “just”
24%. This data confirms the importance of having a translation reference for each sample
instead of relying on standard values (Federico et al., 2012).
Table 4 reports the productivity (words per hour) obtained for each translator in each band.
The words per hour average is provided in the last column.

100%
95–99%
85–94%
75–84%
MTPE
Trans.

Seg.
94
21
48
42
131
132

Words
1226
231
1062
696
1890
1914

TR-1
3277
2642
2960
1592
1804
1701

TR-2
2942
2625
2248
1592
1743
1319

TR-3
1894
1476
1660
1372
1369
993

TR-4
1767
1299
1678
1140
1141
916

TR-6
1579
963
1232
1019
922
933

TR-7
4039
2011
2164
1342
1481
1236

TR-9
2798
1133
2429
1576
1433
1223

TR-10
1395
1543
1012
741
739
473

Avg.
2461
1477
1923
1297
1329
1099

Table 4: Productivity achieved per translator and match band in words per hour.
We additionally computed the productivity gain achieved by each translator in each bandand the average gain across translators. Equation 3 shows the formula used for calculating productivity gains, where P E T hroughput is the productivity achieved by one translator in words
per hour when post-editing MT output or TM matches, and T RA T hroughput is the productivity achieved by that same translator when translating from scratch (taken from productivity
values in Table 4). Table 5 reports the productivity gains obtained.
(

MTPE gains

TR-1
6.06

P E T hroughput− 1) ∗ 100
T RA T hroughput

TR-2
32.15

TR-3
37.78

TR-4
24.62

TR-6
-1.23

TR-7
19.80

(3)

TR-9
17.20

TR-10
56.34

Avg.
24.09

Table 5: Productivity gain percentage due to MTPE for each translator.
All translators except one (Translator 6, who experienced 1.2% productivity loss whenfacing MTPE) were faster in MTPE than translating from scratch, although the productivity
gain varied greatly among them (from 6.06% to 56.34%). A possible explanation for the slight
productivity loss experienced by Translator 6 might be that this translator had experienced an
extensive period of inactivity and had barely used memoQ. The biggest productivity gain wasachieved by the least experienced translator (Translator 10), while the smallest productivity
gain corresponds to the most experienced translator (Translator 1), although this trend cannot
be confirmed by the rest of results. Furthermore, all translators who had MTPE throughput

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 138

higher than the 75–84% band (Translators 1, 2 and 7) had previous experience in MTPE. On
the contrary, both translators (6 and 9) who worked faster with 75–84% TM matches than withMT segments had no MTPE experience. This seems to point out that mature MT engines allow
experienced post-editors to post-edit MT faster than the lowest TM matches.
Also, it is worth noting that the productivity in Table 4 reflects only the time spent by
the translator inside the editor. Other tasks such as reading of instructions, file management,
escalation of queries, self-review, quality assurance and communication with project managers
and/or other parties involved in the project are not directly reflected here, but are all part ofa regular project. Care should also be taken when transferring productivity gains into rate
discounts, as translation rates often include tasks which are not affected by the introductionof MTPE (such as project management, file engineering or review by a second linguist). In
any case, it is interesting to see that MTPE can improve productivity even when the fastest
translators work with content which allow for higher-than-usual translation throughputs.
4.2

MT evaluation metrics

We calculated document-level values for BLEU, TER and FMS13 (Tables 6, 7 and 8 respectively) taking the segments belonging to each band as separate documents. To allow for a more
direct comparison, next to each band average we attach its corresponding average productivity
gain, calculated as the average of the eight values (one for each translator) obtained according
to Equation 3 above.

100%
95–99%
85–94%
75–84%
MTPE

TR-1

TR-2

TR-3

TR-4

TR-6

TR-7

TR-9

TR-10

Avg.

93.17
86.51
82.22
70.42
64.24

92.38
89.35
81.00
71.98
66.37

85.57
78.96
76.15
66.57
64.23

92.33
81.85
80.70
70.34
63.40

91.96
87.91
80.19
73.94
69.57

96.28
85.83
85.62
71.50
66.42

94.25
80.32
84.39
70.52
65.88

95.51
92.10
88.23
70.36
68.43

92.68
85.35
82.31
70.70
66.07

Avg.
gain
127.38%66.20%
76.82%
22.52%
24.09%

Table 6: BLEU scores for each TM match band and average productivity gain (%).

100%
95–99%
85–94%
75–84%
MTPE

TR-1

TR-2

TR-3

TR-4

TR-6

TR-7

TR-9

TR-10

Avg.

4.38
8.24
11.58
21.65
22.14

3.99
6.37
12.69
21.03
20.98

8.43
14.02
16.89
23.74
22.32

4.35
11.50
12.96
21.18
24.52

4.64
6.91
13.02
19.08
18.89

2.00
8.31
10.45
20.07
19.74

3.10
12.83
10.04
20.86
19.77

1.94
5.30
8.33
20.20
18.84

4.10
9.19
11.99
20.98
20.90

Avg.
gain
127.38%66.20%
76.82%
22.52%
24.09%

Table 7: TER scores for each TM match band and average productivity gain (%).
The three metrics agree that the four less experienced translators (6–10) performed lessedits to the MT output and the TM matches than the four more experienced translators (1–5).
The issue reported in the beginning of Section 4 about the 95–99% band not achieving the
expected productivity can also be seen in these metric tables. It would be logical to expect the
productivity of this band being somewhere in between the 100% and the 85–94% bands, and thethree metrics indeed agree with this intuition, but in terms of productivity it was not the case.
13 BLEU and TER were computed using Asiya (Giménez and Màrquez, 2010) and the FMS was computed using the
Okapi framework. For each translator and in each segment, the reference was only the post-edited MT or TM output
delivered by that same translator. In no case multiple references were used.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 139

100%
95–99%
85–94%
75–84%
MTPE

TR-1

TR-2

TR-3

TR-4

TR-6

TR-7

TR-9

TR-10

Avg.

96.91
91.62
92.19
85.07
87.29

97.91
92.76
91.19
85.21
87.74

94.81
90.71
88.42
83.02
87.05

97.34
91.86
90.44
85.62
86.30

97.50
92.43
90.92
87.93
88.53

98.67
92.10
92.19
85.6
88.4

98.01
90.00
92.21
85.48
88.85

98.70
95.48
93.50
86.83
89.09

97.48
92.12
91.38
85.60
87.91

Avg.
gain
127.38%66.20%
76.82%
22.52%
24.09%

Table 8: Fuzzy scores for each TM match band and average productivity gain (%).

As explained earlier, this can be explained by the fact that the vast majority of edits requiredin the 95–99% band involved dealing with inline tags. Although the impact of these operations
has not been researched enough, these results show that they can have a big impact in terms
of productivity, slowing down the translator more than would be expected. Moreover, whencalculating automated metrics, these inline tags are generally deleted to avoid their division indifferent tokens, thus potentially skewing the results. Instead of deleting them, when calculating
the automated metrics reported in this paper we converted each tag into a unique token. This
operation was clearly not enough to compensate all the effort put into tag handling, as hinted by
the productivity values. More research is needed in this area to find out the appropriate weightor penalty that automated metrics should assign to inline tags. As stated earlier, we opted totreat the 95–99% band as an outlier and ignore it from further analysis.
4.3

Productivity vs. MT evaluation metrics correlation

To find out if the proposed FMS stands up against traditional methods of MT evaluation, we
correlated the metrics in Tables 6, 7 and 8 with the productivity values in Table 4. We also
added two alternatives to the FMS calculated by Okapi Rainbow: the FMS provided by SDL
Trados Studio and memoQ, two of the most popular CAT tools worldwide. As can be observedin Table 9, the results indicate that BLEU, TER and the three FMS have a strong correlation
with productivity. Therefore, all of them seem suitable for evaluating MTPE or TM matchproductivity. The anomaly created by Translator 8 using the predictive text feature is also
reflected in Table 9.
rBLEU
rTER
rfuzzyrStudiormemoQTR-1
0.934-0.973
0.973
0.967
0.929

TR-2
0.952
-0.995
0.995
0.987
0.962

TR-3
0.996
-0.987
0.918
0.955
0.903

TR-4
0.934
-0.945
0.902
0.930
0.838

TR-6
0.999
-0.989
0.971
0.942
0.934

TR-7
0.925
-0.968
0.974
0.968
0.998

TR-8
(0.816)(-0.725)(0.548)(0.749)(0.570)TR-9
0.995-0.984
0.911
0.987
0.900

TR-10
0.953
-0.973
0.979
0.969
0.999

Avg.
0.961
-0.977
0.953
0.963
0.933

Table 9: Pearson correlation between productivity and evaluation measures. Translator 8’s
results are omitted from the average.

4.4

Discussion

Analyzing the results further, there seems to be benefits of using FMS over BLEU or TER toevaluate MTPE. Table 10 shows the average BLEU, TER, FMS and productivity gain values
for MTPE and the 75–84% fuzzy band. The average gain due to MTPE was higher than the 75–
84% band. However, according to BLEU scores, the situation should have been the opposite,

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 140

while according to TER both values should have been more or less the same. The FMS, on theother hand, accurately reflects the productivity gains obtained.14

75–84%
MTPE

BLEU

TER

FMS

70.70
66.07

20.98
20.90

85.60
87.91

Prod.
gain
22.52%
24.09%

Table 10: Average BLEU, TER, FMS and productivity gain for MTPE and 75–84% bands.
Another useful application of FMS is the insight they provide when setting the threshold
between TM matching and segments sent to the MT engine. Normally, TM matches below 75%
are assumed to yield no productivity increase. Therefore, when applying MTPE, the generalapproach is to process all segments in the 0–74% range via MT. However, if the MT output
quality is high enough, it may be more productive to post-edit MT output rather than some of
the TM matches, as the results of this experiment actually show. This issue was first brought toour attention by the post-editors themselves, who frequently suggested increasing the thresholdto 85%. After internal testing, it was verified that the post-editors’ intuition was correct and a
new threshold of 85% was established for this client.15 Interestingly enough, that value is also
the average FMS obtained with this client’s projects over a three-year period. In our experience,
this method can also be applied successfully to other language pairs and higher FMS ranges.
For example, the threshold of Spanish-Catalan engines, which generally perform better thanmore distant language pairs, is set at 95%, as suggested by post-editors, confirmed by testing
and reflected in FMS. These experiences suggest that the FMS applied to MTPE evaluation canbe used to adjust the threshold between TM matching and MT segments on a client, domain or
engine basis.
Since FMS seems to perform as well as other widely used metrics and incorporates otherbenefits, it may be reasonable to use FMS as a metric for analyzing both TM leverage and MT
output. Relying on two different scores for these technologies creates contradictory situations,where a client may implicitly acknowledge that TM fuzzy matches below 75% do not yieldproductivity gains, and yet consider MT output with 60& FMS as liable for discounts basedon their own assumptions on BLEU, TER or other metric. A unified framework for combining
evaluation and quality estimation of TM and MT technologies has been recently proposed by
Forcada and Sánchez-Martı́nez (2015). Even though they do not consider MT quality indicators
and TM matching scores comparable, fuzzy scores still play an important role in this unification.
In any case, these efforts may help to provide a better integration of translation technologies andspread best practices for all parties involved in a translation project.

5 Conclusion and future work
In this paper, we reported an experiment where ten professional translators with diverse experience in translation and MTPE complete the same translation and post-editing job within theireveryday work environment using files from a real translation request. The objective was toanalyze the relation between productivity and automated metrics in a commercial setting, andverify if the target-side FMS could be an adequate method of evaluating MT output for PE.
14 This pattern has also been observed in other MTPE projects performed in our company.
15 As explained in Section 3.1, for the purposes of the experiment reported here, this higher threshold was not used,

so all segments below 75% were considered no-match segments. Also, we do not imply that MTPE can always be fasterthan post-editing 75–84% TM matches. With poor output, that threshold should still be set at 75% or even lower, while
better engines may benefit from higher thresholds.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 141

The more than 7,000 words of the file to be translated from English into Spanish wasanalyzed with the usual CAT tools in our company to ensure it contained a significant amountof fuzzy matches, exact matches and no-match segments. Half of the no-match segments was
randomly selected for MTPE, while the other half was translated from scratch. The MT output
for the MTPE sample was generated using one of our customized RBMT engines.
The results show that the FMS applied to MTPE evaluation has a strong correlation withproductivity, comparable to the correlation obtained with more traditional evaluation metrics
such as BLEU and TER. Moreover, the FMS was the only metric able to reflect the higher productivity obtained by the MTPE sample over the 75–84% TM match band. Another advantage
shown is that it can also be used to set the TM matching threshold at which post-editing MT ismore productive than post-editing TM matches.
Another finding is that MT output from a mature engine increases translators’ productivity when compared to translation from scratch and can even surpass the throughput of TM
matches. Only one of the ten translators involved did not experience any productivity gain. The
productivity gain achieved by the rest of translators varied greatly between them, with the mostexperienced translator having the least productivity increase and the least experienced translator
obtaining the biggest gain. However, this trend cannot be confirmed by the performance of therest of translators.
It was also revealed that inline tags have a big impact on productivity, a fact which is notreflected in any of the known metrics and which has not yet received much attention in research.
Other areas for future work include using this data set to perform a study on quality estimation
to avoid depending on reference translations and further analysis of automated metrics in orderto reveal the threshold at which each metric ceases to consider a segment useful in terms of
productivity.
The performance of FMS applied to MTPE evaluation has been shown to be as solid as
any of the traditional metrics. Its correlation with productivity is as strong as BLEU or TER andit can be used to determine thresholds between TM matching and MTPE segments. Moreover,
FMS is easier to calculate for general users and we believe it would be more readily embracedby the industry due to its analogy with the well-established TM match bands.

References
Aziz, W., de Sousa, S. C. M., and Specia, L. (2012). PET; a Tool for Post-Editing and Assessing MachineTranslation. In Eighth International Conference on Language Resources and Evaluation (LREC12),
pages 3982–3987, Istanbul, Turkey. ELRA.
Bechara, H. (2013). Statistical Post-Editing and Quality Estimation for Machine Translation Systems.
Master’s thesis, Dublin City University, School of Computing.
Burchardt, A. and Lommel, A. (2014). Practical Guidelines for the Use of MQM in Scientific Researchon Translation Quality. Technical report, DFKI, Berlin.
Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., and Schroeder, J. (2007). (Meta-) Evaluation of
Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages
136–158.
Callison-Burch, C., Osborne, M., and Koehn, P. (2006). Re-evaluating the role of bleu in Machine translation research. In Proceedings of the 11th Conference of the European Chapter of the Associatio for
Computational Linguistics, pages 249–256, Trento, Italy. EACL.
Denkowski, M. and Lavie, A. (2010). METEOR-NEXT and the METEOR paraphrase tables: improved
evaluation support for five target languages. In Proceedings of the Joint Fifth Workshop on Statistical

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 142

Machine Translation and MetricsMATR, WMT ’10, pages 339–342, Stroudsburg, PA, USA. Association for Computational Linguistics.
Dice, L. R. (1945). Measures of the Amount of Ecologic Association Between Species. Ecological Societyof America, 26(3):297–302.
Doddington, G. (2002). Automatic evaluation of machine translation quality using n-gram co-occurrence
statistics. In Proceedings of the second international conference on Human Language Technology
Research, HLT ’02, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Federico, M., Cattelan, A., and Trombetti, M. (2012). Measuring User Productivity in Machine Translation
Enhanced Computer Assisted Translation. In Proceedings of the Tenth Conference of the Association
for Machine Translation in the Americas (AMTA), San Diego, CA. AMTA.
Forcada, M. L. and Sánchez-Martı́nez, F. (2015). A general framework for minimizing translation effort:
towards a principled combination of translation technologies in computer-aided translation. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pages
27–34, Antalya, Turkey.
Giménez, J. and Màrquez, L. (2010). Asiya: An Open Toolkit for Automatic Machine Translation (Meta)Evaluation. The Prague Bulletin of Mathematical Linguistics, (94):77–86.
Hurtado Albir, A. (1995). La didáctica de la traducción. Evolución y estado actual. In Fernández, P. and
Bravo, J., editors, Perspectivas de la traducción, pages 49–74. Universidad de Valladolid.
King, M., Popescu-Belis, A., and Hovy, E. (2003). FEMTI: Creating and Using a Framework for MT
Evaluation. In Proceedings of the Machine Translation Summit IX, pages 224–231, New Orleans, LA.
Koehn, P. (2010). Statistical Machine Translation. Cambridge University Press.
Lavie, A. (2010). Evaluating the Output of Machine Translation Systems. AMTA, Denver, Colorado,
USA.
Lavie, A. and Agarwal, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved
Correlation with Human Judgments. In Proceedings of ACL Workshop on Intirnsic and Extrinsic Evaluation Measures for MT and/or Summarisation, pages 65–72.
Lin, C.-Y. and Och, F. J. (2004). Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL ’04, Stroudsburg, PA, USA. Association for Computational
Linguistics.
Lommel, A. (2015). Multidimensional Quality Metrics (MQM) Definition. Technical report, DFKI.
Melamed, I. D., Green, R., and Turian, J. P. (2003). Precision and recall of machine translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational
Linguistics on Human Language Technology: companion volume of the Proceedings of HLT-NAACL
2003–short papers, volume 2 of NAACL-Short ’03, pages 61–63, Stroudsburg, PA, USA. Association
for Computational Linguistics.
Moran, J., Saam, C., and Lewis, D. (2014). Towards desktop-based CAT tool instrumentation. In Proceedings of the Third Workshop on Post-Editing Technology and Practice, pages 99–112, Vancouver,
BC. AMTA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 143

Nießen, S., Och, F. J., Leusch, G., and Ney, H. (2000). An Evaluation Tool for Machine Translation: Fast
Evaluation for MT Research. In International Conference on Language Resources and Evaluation,
pages 39–45, Athens, Greece. European Language Resources Association (ELRA).
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2001). Bleu: a Method for Automatic Evaluation of
Machine Translation. IBM Research Report RC22176 (W0109-022), IBM Research Division, Thomas
J. Watson Research Center, P.O. Box 218, Yorktown Heights, NY 10598.
Parra Escartı́n, C. and Arcedillo, M. (2015). A fuzzier approach to machine translation evaluation: A pilot
study on post-editing productivity and automated metrics in commercial settings. In Proceedings of
the ACL 2015 Fourth Workshop on Hybrid Approaches to Translation (HyTra), pages 40–45, Beijing,
China. ACL.
Plitt, M. and Masselot, F. (2010). A Productivity Test of Statistical Machine Translation Post-Editing in a
Typical Localisation Context. The Prague Bulletin of Mathematical Linguistics, 93:7–16.
Ruopp, A. (2010). The Moses for Localisation Open Source Project. In Proceedings of the Ninth Conference of the Association for Machine Translation in the Americas, Denver, Colorado. AMTA.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A Study of Translation Edit
Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for
Machine Translation in the Americas, pages 223–231, Cambridge, Massachusetts, USA.
Snover, M., Madnani, N., Dorr, B. J., and Schwartz, R. (2009). Fluency, adequacy, or HTER? Exploring
different human judgments with a tunable MT metric. In In Proceedings of the Fourth Workshop on
Statistical Machine Translation, pages 259–268.
Sørensen, T. (1948). A method of establishing groups of equal amplitude in plant sociology based on
similarity of species and its application to analyses of the vegetation on Danish commons. Kongelige
Danske Videnskabernes Selskab, 5 (4):1–34.
Tillmann, C., Vogel, S., Ney, H., Zubiaga, A., and Sawaf, H. (1997). Accelerated DP Based Search For
Statistical Translation. In European Conf. on Speech Communication and Technology, pages 2667–
2670.
Zhechev, V. (2012). Machine Translation Infrastructure and Post-Editing Performance at Autodesk. In
AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), pages 87–96, San
Diego, USA. Association for Machine Translation in the Americas (AMTA).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 144

A Distributed Inflection Model for Translating
into Morphologically Rich Languages
Ke TranArianna Bisazza
Christof Monz

m.k.tran@uva.nl
a.bisazza@uva.nl
c.monz@uva.nl

Informatics Institute, University of Amsterdam,
Science Park 904, 1098 XH Amsterdam, The Netherlands

Abstract
Lexical sparsity is a major challenge for machine translation into morphologically rich languages. We address this problem by modeling sequences of fine-grained morphological tags
in a bilingual context. To overcome the issue of ambiguous word analyses, we introduce soft
tags, which are under-specified representations retaining all possible morphological attributesof a word. In order to learn distributed representations for the soft tags and their interactions
we adopt a neural network approach. This approach allows for the combination of source andtarget side information to model a wide range of inflection phenomena. Our re-inflection experiments show a substantial increase in accuracy compared to a model trained on morphologically disambiguated data. Integrated into an SMT decoder and evaluated for English-Italianand English-Russian translation, our model yields improvements of up to 1.0 BLEU over acompetitive baseline.

1

Introduction

In morphologically rich languages (MRLs), words can have many different surface forms depending on the grammatical context. When translating into MRLs, standard statistical machine translation (SMT) models such as phrase translation models and n-gram language models (LMs) often fail to select the right surface form due to the sparsity of observed word sequences (Minkov et al., 2007; Green and DeNero, 2012). While neural LMs (Bengio et al.,
2003; Schwenk, 2007) address lexical sparsity to a certain degree by projecting word sequences
to distributed vector representations, they still suffer from the problem of rare words which isparticularly exacerbated in MRLs (Botha and Blunsom, 2014; Jean et al., 2015; Luong et al.,2015).
A potential solution to overcome data sparsity in MRLs, is to use word representations
that separate the grammatical aspects of a word, i. e. inflection, from the lexical ones. Suchword representations already exist for many languages in the form of morphological analyzers
or lexicons. However, using these resources for statistical language modeling is far from trivial
due to the issue of ambiguous word analyses. Table 1 illustrates this problem in Italian, forwhich a fine-grained morphological lexicon but no sizable disambiguated corpus exists. Thesemorphological analyses1 clearly contain information that is useful to encourage grammaticalagreement and, in this case, detect the highlighted error. Unfortunately, though, the needed
1 In this work we use the terms analysis and tag interchangeably to denote fine-grained word annotations provided
by a morphological analyzer or lexicon.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 145

SMT

idee

ribellichecircola

Gloss

ideas

rebellious

thatcirculatenoun-f:pnoun-f:pnoun-m:p
adj:pos+f+p
adj:pos+m+p...

con
wh-chedet-wh:f+p
det-wh:f+s...

ver:impr+pres+2+s
ver:ind+pres+3+s

Analyses

Table 1: Example of morphological error in Italian SMT output: the verb form should be plural(circolano) and not singular (circola) to agree in number with the subject. Most of the words
have multiple analyses according to our morphological lexicon of reference (Zanchetta andBaroni, 2005). The correct one in context is highlighted.
information is difficult to access because each word can have multiple analyses. Performing
contextual disambiguation during translation is an ill-posed problem because the SMT decoder
produces large numbers of ungrammatical word sequences but gold tagged training data is naturally composed of grammatical sentences.2 Moreover, searching for the optimal tag sequence
introduces spurious ambiguity into the SMT decoder. Finally, training a disambiguator requires
manually disambiguated data, which is not available in many languages and costly to produce.
In this paper, we address this problem with a novel inflection model that is based on two
main ideas: First, morphological ambiguity does not need to be resolved for SMT. Instead, we
map words to a space where all possible morphological attributes of a word are retained. Rather
than enforcing hard tagging decisions, we let the model operate on soft word representations.
The resulting tag set is larger than the original one, but still effective at reducing the lexical
sparsity of purely word-based LM. Second, learning distributed representations for soft morphological tags can help share statistical strength among overlapping tags, i. e. tags that havesome attributes in common. To achieve this, we train a neural network that predicts sequencesof soft tags conditioned on rich contextual features.
We show that: (i) our soft representation model achieves higher accuracies in re-inflecting
translations than a model performing contextual disambiguation, and (ii) our model significantly
improves translation quality on two different target MRLs, including a language for which nosizable disambiguated corpora exist.
The paper is organized as follows: after reviewing the previous work (Section 2), we
present our distributed inflection model based on soft morphological representations (Section 3).
In Section 4 we introduce the general experimental setup, followed by a detailed description of
the re-inflection experiments (Section 5) and the end-to-end SMT experiments (Section 6). We
conclude with a discussion of SMT output examples and an outlook of future work

2

Previous Work

Previous work on inflection modeling for translation into MRLs has mostly relied on the availability of morphologically disambiguated data to choose the most probable analysis of eachword in either a context-independent (Minkov et al., 2007) or context-dependent (Green and
DeNero, 2012; Koehn and Hoang, 2007; Subotin, 2011) way. While the former irrevocably
discards potentially useful attributes of the words, the latter tasks the inflection model with disambiguating the word sequence under construction, which is difficult given the ill-formednessof SMT output and a cause of spurious ambiguity.
2 This issue has also been shown to affect syntactic parsing of SMT output (Post and Gildea, 2008; Carter and Monz,2011).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 146

Considerably less work has focused on MRLs where disambiguated data does not exist,
with few exceptions where ambiguity is solved by randomly selecting one analysis per word
type (Minkov et al., 2007; Toutanova et al., 2008; Jeong et al., 2010).
As for how inflection models are integrated into the STM system, different strategies havebeen proposed. Minkov et al. (2007); Toutanova et al. (2008); Fraser et al. (2012) treat inflection as a post-processing task: the SMT model is trained to produce lemmatized target sentences(possibly enhanced with some form of morphological annotation) and afterwards the best surface form for each lemma is chosen by separate inflection models. Some work has focused on
the generation of new inflected phrases given the input sentence (Chahuneau et al., 2013) or
given the bilingual context during decoding (Koehn and Hoang, 2007; Subotin, 2011). Other
inflection models have been integrated to SMT as additional feature functions: e.g. as an additional lexical translation score (Jeong et al., 2010; Tran et al., 2014) or as an additional target
language model score (Green and DeNero, 2012). We follow this last strategy, rather than generating new inflections, motivated by previous observations that, when translating into MRLs,a large number of reference inflections are already available in the SMT models but are notselected for Viterbi translation (Green and DeNero, 2012; Tran et al., 2014).
More in general, our work is related to class-based language modeling (Brown et al., 1992)
with the major difference that we also condition on source-side context and that we use explicit morphological representations instead of data-driven word clusters (Uszkoreit and Brants,2008), word suffixes (Müller et al., 2012; Bisazza and Monz, 2014) or coarse-grained part-ofspeech tags (Koehn et al., 2008).
Modeling morphology using neural networks has recently shown promising results: in thecontext of monolingual neural language modeling, Luong et al. (2013); Botha and Blunsom(2014) obtain the vectorial representation of a word by composing the representations of its
morphemes. Tran et al. (2014) model translation stem and suffix selection in SMT with abilingual neural network. Soricut and Och (2015) discover morphological transformation rules
from word embeddings learned by a shallow network. We are not aware of work that leveraged
fine-grained morphological tags for neural language or translation modeling.

3

A Distributed Inflection ModelIn MRLs, the surface form of a word is heavily determined by its grammatical features, suchas number, case, tense etc. Choosing the right target word form during translation is a complex
problem since some of these features depend on the source context while others depend on thetarget context (agreement phenomena). We model target language inflection by a Markov process generating a sequence of abstract word representations based on source and target context.
This complements previous work focusing on either the former (Avramidis and Koehn, 2008;
Chahuneau et al., 2013; Tran et al., 2014) or the latter (Green and DeNero, 2012; Fraser et al.,2012; Botha and Blunsom, 2014; Bisazza and Monz, 2014).
3.1

Soft Morphological Representations

As previously stated, it is common for words in MRLs to admit multiple morphological analyses out of context. Rather than trying to disambiguate the analyses in context using for instanceconditional random fields (Green and DeNero, 2012; Fraser et al., 2012), we modify the tagging
scheme so that each word corresponds to only one tag. To also avoid the loss of useful information incurred when arbitrarily selecting one analysis per word type (Minkov et al., 2007; Jeonget al., 2010), we introduce soft morphological representations, or simply soft tags.
Assume that a morphological analysis µ is a set of morphological attributes S(µ) such as
masculine or plural. Given a word w, a morphological analyzer or lexicon LEX returns a list

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 147

of possible analyses of that word Aw = {µ : (w, µ) ∈ LEX}. Then, we can map word w to aunique soft tag rw by simply taking the union of all its possible morphological attributes, that
is:
[
def
rw =
S(µk )µk ∈Aw

For instance, the Italian word “ribelle” has four analyses: adj:pos+f+s, adj:pos+m+s, noun-f:s,and noun-m:s. Its corresponding soft tag is adj:pos|adj:f|adj:s|adj:m|noun-f:s|noun-m:s. Hence,
soft tags maintain all morphological attributes of a word to denote its grammatical dimension
while ignoring the lexical content. This new representation scheme compromises between sparsity and ambiguity, and allows for an efficient integration of our model directly into the decoder
as no additional cost is incurred for the local tagging search.
Soft tags can also be seen as the marginalization of µ when predicting a surface word wi
given a lemma li and its context Ci (i. e. variables that influence wi , such as wi−1 ):Xp(wi |li , Ci ) =p(wi |µk , li , Ci )p(µk |li , Ci )µk ∈Awi=Xp(µk |li , Ci )(1)

µk ∈Awiassuming that any lemma-analysis pair (l, µ) corresponds to at most one inflected form w.
Using soft tags, Equation 1 can be approximated by p(rw |li , Ci ).
3.2

Inflection Neural Network

Our inflection model3 , Inf-NN, is trained on word-aligned bilingual data to predict sequences oftarget soft tags given a fixed-size target history and the input source sentence (see Figure 1). We
adopt a neural LM approach as learning distributed representations for the soft tags can help toshare statistical information among overlapping tags (i. e. tags that share some morphologicalattributes). Moreover, compared to Maximum Entropy models that use lexical features, neuralnetworks can better exploit sparse input features such as lexicalized source context and target
lemma features, as well as their interactions, in high dimensional spaces.
We learn distributed representations for both source words and target soft tags. The source
word representations are initialized from pre-trained embeddings, which has been shown to
encode certain morphological regularities (Soricut and Och, 2015), whereas target tag representations are initialized randomly.
Inf-NN is a feed-forward neural network whose output is a conditional probability distribution over a set of morphological tags given target history and source context. Formally, let hi =(ri−1 , . . . , ri−n+1 ) be the n−1 tag history of the target word wi , and cj = (sj−k , . . . , sj+k ) the
source context centering at the word sj aligned to wi by an automatic aligner. We use simple
heuristics similar to the approach by Devlin et al. (2014) to handle null and multiple alignments
so that each target word wi can be mapped to exactly one source word sj . Let sj ∈ RD andri ∈ RD denote the distributed representations of source sj and target tag ri respectively. Then,the conditional probability pInf-NN (ri |hi , cj ) is computed at the output layer y of the network as
follows:zi = φ(Wc cj + Wh hi + bz )y = softmax(Wm zi + by )3 The implementation is available at https://bitbucket.org/ketran/soft-tags

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 148

wi 2

wi 1

wi
ri

ri 1

sj 2

sj 1

sj

soft-reps

sj+1

sj+2

sourceri 2targetword

Figure 1: Graphical representation of the Inf-NN model: the current target word’s soft tag, ri ,is predicted based on a fixed-size target tag history and a source side context centered aroundsj , the translation of wi . Each target word wi can be deterministically mapped to a soft tag ri .

where Wc , Wh , and Wm are weight matrices, cj and hi are shorthands for [sj−k ; . . . ; sj+k ]and [ri−1 ; . . . ; ri−n+1 ] respectively, [v; v0 ] denotes vector concatenation, and φ is a non-linear
transfer prelu. As φ, we use in all experiments the channel-shared parametric rectified linear
unit (PReLU) introduced by He et al. (2015). PReLU φ(x) is defined as:φ(x)(x
if x > 0φ(x) =
ax otherwisexwhere a is a parameter learned during training. To speed up decoding, we train the Inf-NN
model with a self-normalized objective (Devlin et al., 2014; Andreas and Klein, 2015). Morespecifically, we adopt the objective function proposed by Andreas and Klein (2015):
2`(θ) = − E ln p(ri |hi , cj ) + η kθk2γ 
+ E ln2 Z(hi , cj )|(hi , cj ) ∈ H
pwhere
H is a set of random samples on which self-normalization is performed, θ ={sj }, {ri }, Wc , Wh , Wm , bz , a are the parameters of the networks, and Z(hi , cj ) is thepartition function of the input (hi , cj ). In practice, we obtain H by sampling from a Bernoulli
distribution Bern(p). This is equivalent to applying dropout (Srivastava et al., 2014) on the loss
gradient 1 ∈ Rm of self-normalization term, where m is the size of a mini-batch. We regularize
the networks with `2 norm.

4

Experimental Setup

We evaluate our approach on two related tasks: re-inflecting reference translations and end-toend translation from English into MRLs. With the first task, we test the effectiveness of soft
morphological representations against (i) a model that randomly assigns one tag per word type(among its possible tags) and (ii) a model that admits multiple tags per word and requires a predisambiguated corpus to be trained. With the second task, we measure translation quality when
our inflection model is integrated into a state-of-the-art phrase-based SMT decoder, showing its
applicability to languages where no disambiguated data exists.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 149

4.1

Data

As target languages, we choose two MRLs belonging to different language families and displaying different inflectional patterns: Russian has very rich nominal, adjectival and verbal inflection, while Italian has moderate nominal and adjectival inflection, but extremely rich verbal
inflection. Experiments are performed on the following tasks:• English-Russian WMT (Bojar et al., 2013): translation of news commentaries with largescale training data.
• English-Italian IWSLT (Cettolo et al., 2014): translation of speeches with either smallscale training data (TED talks only) or large-scale training data (TED talks and European
proceedings).
SMT training data statistics are reported in Table 2. The Russian Inf-NN model is trained on a
1M-sentence subset of the bilingual data, while the Italian one is trained on all the data available
in each setting. For each data set, we create automatic word alignments using GIZA++ (Ochand Ney, 2003).

Bilingual

Monoling.

En-Ru
large

small

large

#sentences
src/trg #tokens
src/trg dict.size

2.4M
49.2/47.2M
774K/1100K

180K
3.6/3.4M
55K/80K

2.0M
57.4/57.0M
139K/195K#sentences
trg #tokens
src/trg dict.size

21.0M
390M
2.7M

En-It

2.1M
58.4M
199K

Table 2: Training corpora statistics.
The ambiguous morphological analyses are obtained from the Russian OpenCorpora lexicon4 (Bocharov et al., 2013) and from the Italian Morph-it!5 lexicon (Zanchetta and Baroni,2005). Table 3 shows the number of tags and soft tags occurring in our training data, as wellas the expected counts of analyses per word Ew [t], words per lemma El [w] and analyses perlemma El [t].
Language#tags#soft-tags

Ew [t]El [w]El [t]

RussianItalian

892
450

4431
901

3.8
1.9

7.2
12.7

27.4
24.3

Table 3: Morphological characteristics of the Inf-NN training data: number of tags and soft
tags, expected counts of analyses per word Ew [t], words per lemma El [w] and analyses perlemma El [t].
We find that the Russian tag set and, consequently, the soft tag set are considerably larger
than the Italian ones. The average morphological ambiguity is also larger in Russian (3.8 versus
4 opencorpora.org
5 sslmitdev-online.sslmit.unibo.it/linguistics/morph-it.php

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 150

1.9 tags per word). However, somewhat surprisingly, morphological richness is higher in Italian(12.7 versus 7.2 words per lemma). At a closer inspection, we find that most of this richness isdue to verbal inflection which goes up to 50 forms for frequently observed verbs.
4.2

Neural network training

The Inf-NN models are trained on a history of 4 target tags and source context of 7 words withthe following configuration: Embedding size is set to 200 and the number of hidden units to768. Target word and soft-tag embeddings are initialized randomly from a Gaussian distribution
with mean zero and standard deviation 0.01. Source word embeddings are initialized from pretrained Glove vectors (Pennington et al., 2014) and rescaled by a factor of 0.1. Weight matricesofp linear layers are initialized from a zero-mean Gaussian distribution with standard deviation
2/ni where ni is the number of input units (He et al., 2015). We set self-normalization
strength γ = 0.02, Bernoulli parameter p = 0.1, and regularization parameter η = 10−4 .
All models are trained with a mini-batch size of 128 for 30 epochs. Our stochastic objectivefunctions are optimized using the first-order gradient-based optimizer Adam (Kingma and Ba,2015). We use the default settings suggested by the authors: α = 0.001, β1 = 0.9, β2 = 0.999,
 = 10−8 and λ = 1 − 10−8 .

5Re-inflection Experiments

The purpose of this experiment is to simulate the behavior of the inflection model during SMT
decoding: Given a reference translation and its corresponding source sentence, we re-inflect theformer using a simple beam search and count how many times the model recovers the correct
surface word form on a 10K-sentence held-out data set.
Since we do not assume the availability of a disambiguator, we also have to deal withlemma ambiguity. While this issue does not affect the definition and training of our Inf-NN,
we do need lemmas to determine the set of candidate surface forms Iw for each word w that isbeing re-inflected. As a solution, we define Iw as the union of the surface forms of each possible
lemma of w or, more formally, as:
Iw = {wi | lem(wi ) ∩ lem(w) 6= ∅}where lem(w) denotes the set of lemmas returned by the lexicon for word w. For example, theItalian form baci has two possible lemmas: bacio (noun: kiss) and baciare (verb: to kiss). Its
candidate set Iw will then include all the forms of the noun bacio and all the forms of the verb
baciare: that is, bacio, baci, baciamo, baciate, baciano, etc.
We compare the proposed soft-tag Inf-NN against an Inf-NN trained on randomly assigned
tag per type and to another one trained on tag sequences disambiguated by TreeTagger (Schmid,
1994; Sharoff et al., 2008). The latter model must search through a much larger space of morphological tag sequences. Therefore, to allow for a fair comparison, we set a higher beam
size when re-inflecting with this model. As another difference from the other models, theTreeTagger-based inflection model relies on the lemmatization performed by TreeTagger todefine the candidate set Iw .
To validate the effectiveness of the neural network approach, we also compare Inf-NN to asimpler MaxEnt model trained on a similar configuration. Finally, we evaluate the importanceof source-side context features by experimenting with a series of Inf-NN models that are only
conditioned on the target tag history.
Since no morphological disambiguator is available for Italian, we perform this experiment only for Russian. As shown in Table 4, soft tags perform best in all settings and become
even more effective when moving from MaxEnt to neural network, demonstrating the impor-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 151

Tree-Tagger: all analyses
Random: 1 analysis per word
Soft-Reps: 1 soft tag per word

MaxEntwith src

Inf-NN
w/o src with src

beam

56.33
66.08
66.95

61.19
72.32
75.43

200
5
5

69.68
79.92
81.93

Table 4: Token-level re-inflection accuracy (%) on a 10K-sentence English-Russian held-out
set. The last column indicates the beam size used when searching for the optimal re-inflectedsequence.

tance of learning distributed representations for the soft tags. The notably lower accuracy of
the TreeTagger-based model confirms our intuition that morphological disambiguation is notneeded to model inflection in SMT, but can actually make the task more difficult. This result
can be explained by the fact that, when fixing one tag per word type either by random assignment or with soft tags, the number of tags per lemma becomes substantially smaller (cf. Table 3)
and classification easier. On the other hand, the Tree-Tagger based model operating on all word
analyses has to deal with spurious ambiguity: that is, a correct sequence of inflected words cancorrespond to multiple tag sequences that are competing with one another. Solving this problem
by marginalizing over the ambiguous analyses (cf. Equation 1) can lead to intractable decoding(Sima’an, 1996; Li et al., 2009).
The model using soft-tags, which capture all possible morphological attributes of words,
performs the best. Even without using source context features, our Inf-NN outperforms the
MaxEnt model by 8.5% absolute because of the high dimensional space used to capture complex
morphological regularities. By adding source context, we further increase accuracy by 6.5%,
leading to an overall gain of 15% over the MaxEnt baseline.
Next, we investigate the impact of our most accurate re-inflection model (Soft-Reps InfNN) in an end-to-end SMT setting without relying on any disambiguated data.

6

End-to-end SMT Experiments

We integrated our Inf-NN model into a phrase-based SMT decoder similar to Moses (Koehn
et al., 2007) as an additional log-probability feature function (log pInf-NN ).
When a new target phrase w̃ is produced by the decoder, the Inf-NN model returns aprobability for each word wi that composes it, given the previously translated words’ soft tags
and the source context centered around the source word sj aligned to wi . To detect sj we store
phrase-internal word alignments in the phrase table and use simple heuristics to map each target
index i to exactly one source index j, as done for the Inf-NN training (Section 3.2). Since
every target word corresponds to one soft tag, obtaining the representation of wi is trivial (bylookup in a word-tag map) and so is maintaining the target tag history. This crucially differsfrom previous approaches that distinguish between hypotheses with equal surface forms but
different morphological analyses (Koehn et al., 2007), thereby introducing spurious ambiguity
into what is already a huge search space.6 As a result, the integration of our Inf-NN does notaffect decoding speed.

6 Green and DeNero (2012) also tag each target phrase in context as it is produced. However, they avoid the spurious
ambiguity problem by only preserving the most probable tag sequence for each phrase (incremental greedy decoding).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 152

6.1

Baseline

Our SMT baseline is a competitive phrase-based SMT system including hierarchical lexicalized
reordering models (Galley and Manning, 2008) and a 5-gram target LM trained with modified
Kneser-Ney smoothing (Chen and Goodman, 1999). Since the large English-Italian data comesfrom very different sources (TED talks and European proceedings), we construct phrase table
and reordering models for this experiment using the fillup technique (Bisazza et al., 2011). Notethat our baseline does not include previously proposed inflection models because the main goalof our experiment is to demonstrate the effectiveness of the proposed approach for languages
where no sizable disambiguated data exists, which is indeed the case for Italian.
Feature weights are tuned with pairwise ranking optimization (Hopkins and May, 2011)
on the union of IWSLT’s dev10 and test10 in Italian, and on the first 2000 lines of wmt12
benchmark in Russian (Callison-Burch et al., 2012). During tuning, 14 PRO parameter estimation runs are performed in parallel on different samples of the n-best list after each decoder
iteration. The weights of the individual PRO runs are then averaged and passed on to the next
decoding iteration. Performing weight estimation independently for a number of samples corrects for some of the instability that can be caused by individual samples.
6.2

Results

Translation quality is measured by case-insensitive BLEU (Papineni et al., 2002) on IWSLT’s
test12 and test14 in Italian, and on wmt13 and wmt14 for Russian, all provided with one reference translation. To see whether the differences between the approaches we compared in our
experiments are statistically significant, we apply approximate randomization (Noreen, 1989).7
Data
en→ru

large
small

en→it
large

Test

Baseline

Inf-NN

wmt13

19.0
26.1

19.3N(+0.3)
26.7N(+0.6)

24.6
20.4

25.6N(+1.0)
20.9N(+0.5)

25.0
20.9

25.8N(+0.8)
21.4N(+0.5)

wmt14
iwslt12
iwslt14
iwslt12
iwslt14

Table 5: Impact on translation quality of the Inf-NN model. N marks significance level p < .01.
Results are presented in Table 5. Our Inf-NN model consistently leads to significant improvements over a competitive baseline, for both language pairs and all test sets, without affecting decoding speed. By comparing the two data conditions in English-Italian, we see that mostof the BLEU gain is preserved even after adding a large amount of parallel training data. Thissuggests that morphological phenomena are not sufficiently captured by phrases and stresses
the importance of specifically modeling word inflection. It is possible that adding even moretraining data would reduce the impact of our inflection model, but currently we do not haveaccess to other data sets that would be relevant to our translation tasks.
To put these results into perspective, our improvements are comparable to those achievedby previous work that generated new phrase inflections using a morphological disambiguator(Chahuneau et al., 2013) on the same large-scale English-Russian task.
7 Riezler and Maxwell (2005) have shown that approximate randomization is less sensitive to Type-I errors, i. e. lesslikely to falsely reject the null hypothesis, than bootstrap resampling (Koehn, 2004) in the context of SMT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 153

SRC
REF

(1)

BASE
INFNN

Effect:
SRC
REF

(2)

BASE
INFNN

Effect:
SRC
REF

(3)

BASE
INFNN

Effect:
SRC
REF

(4)

BASE
INFNN

Effect:
SRC
REF

(5)

BASE
INFNN

Effect:and if you’re wondering about those other spikes, those are also fridays
e se vi state chiedendo cosa sono questi altri picchi, sono anche loro dei venerdì
e se vi state chiedendo di queste altre picchi, sono anche il venerdì
e se vi state chiedendo di questi altri picchi, sono anche il venerdì
Correct number agreement between adjectives and noun... a three-hour version of this that’s been viewed four million times... una versione di tre ore che è stata vista 4 milioni di volte... una versione di tre ore di ciò che è stato visto 4 milioni di volte... una versione di tre ore di questo che è stata osservata quattro milioni di volte
Correct gender agreement between subject and present perfect
he died broken by historymorì distrutto dalla storiamorì infranta dalla storiamorì devastato dalla storia
Correct gender agreement between subject and adjective
in one , i was the classic asian student ...
in uno ero la classica studentessa asiatica ...
in uno stato il classico asiatica studente ...
in uno stato il classico asiatico studente ...
Encouraged gender agreement between adjectives and noun, but gender is wrong
in the other , i was enmeshed in lives that were precarious
nell’altro ero invischiata tra esistenze precarie
tra l’altro, sono stato profondamente impegnati in vita che erano più precaria
nell’altro, ero profondamente impegnati in vita che erano più precaria
Failed to encourage gender agreement because surface form is not in the SMT models

Table 6: Examples of SMT output drawn from IWSLT English-Italian test12 showing the effectof our inflection model on lexical selection.

6.3

ExamplesAs previously mentioned, most previous approaches to inflection modeling for SMT may not
be applied to Italian due to the lack of morphological disambiguated data. It is then particularlyinteresting to analyze how our model affects baseline translations. Table 6 presents a number
of English-Italian SMT output examples where the use of our soft-tag Inf-NN either resulted ina better inflection choice (1-3) or not (4-5). Out of the ‘good’ examples, only (1) resulted in
a complete match with the reference translation, while in (2) and (3) the system preferred anequally appropriate lexical choice, showing that automatically evaluating inflection models in
an SMT setting is far from trivial.
The usefulness of source-side features is demonstrated by example (3): here, the translationof broken should agree in gender with the subject he but the baseline system chose instead afeminine form (infranta). Since the subject pronoun can be dropped in Italian, this error cannot
be detected by the target language model and may only be fixed by translating the sequence ‘hedied broken’ as a single phrase, which was never observed in the training data. By contrast, InfNN successfully exploited the source-side context and preferred a masculine form (devastato).
Next are two unsuccessful examples: in (4) Inf-NN encouraged the system to translate
the whole phrase ‘the classic asian student’ as masculine whereas the baseline translation used

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 154

an incoherent mix of masculine and feminine. Unfortunately, though, the student in question,i.e., the speaker, happened to be a woman, but this could not be inferred in any way from thissentence. In (5) Inf-NN failed to fix the agreement between adjective and subject pronoun. By
inspecting the parallel data we found that the word enmeshed always occurred with plural formsof Italian adjectives. This example shows that improving the scoring of the existing translation
options is not always sufficient. While we do not address generation of new inflected forms inthis work, this is an interesting direction for future work.

7

Conclusions

We have proposed a novel morphological representation scheme combined with a neural network for modeling translation into morphologically rich languages (MRLs). Our approachsuccessfully circumvents the problem of ambiguous word analyses and makes it possible toimprove translation into MRLs where morphological lexica but no manually disambiguatedcorpora exist.
Evaluated in a re-inflection task, the proposed soft tags achieve significantly higher accuracy than (i) a model using standard tags and trained on morphologically disambiguated dataand (ii) a Maximum Entropy model that does not learn distributed representations for sourcewords and target tags. When integrated into a state-of-the-art SMT decoder, our inflection
model significantly improves translation quality in two different language pairs, without having to disambiguate during decoding. In particular, our positive English-Italian results underboth small- and large-scale data conditions demonstrate the applicability of our approach tolanguages where no disambiguator exists.
As future work, we will consider learning distributed morphology representation directlyfrom the corpus jointly with the inflection model as well as generating unseen word inflections
during translation.

Acknowledgments
This research was funded in part by the Netherlands Organization for Scientific Research(NWO) under project numbers 639.022.213 and 612.001.218.

References
Andreas, J. and Klein, D. (2015). When and why are log-linear models self-normalizing? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages 244–249, Denver, Colorado. Association for Computational Linguistics.
Avramidis, E. and Koehn, P. (2008). Enriching morphologically poor languages for statistical machinetranslation. In Proceedings of ACL-08: HLT, pages 763–770, Columbus, Ohio. Association for Computational Linguistics.
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model.
Journal of Machine Learning Research, 3:1137–1155.
Bisazza, A. and Monz, C. (2014). Class-based language modeling for translating into morphologically richlanguages. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1918–1927, Dublin, Ireland. Dublin City University and Association
for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 155

Bisazza, A., Ruiz, N., and Federico, M. (2011). Fill-up versus Interpolation Methods for Phrase-based
SMT Adaptation. In International Workshop on Spoken Language Translation (IWSLT), pages 136–
143, San Francisco, CA.
Bocharov, V., Alexeeva, S., Granovsky, D., Protopopova, E., Stepanova, M., and Surikov, A. (2013).
Crowdsourcing morphological annotation. In Proceedings of the International Conference "Dialogue",Bekasovo, Russia.
Bojar, O., Buck, C., Callison-Burch, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M.,
Soricut, R., and Specia, L. (2013). Findings of the 2013 Workshop on Statistical Machine Translation.
In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–44, Sofia, Bulgaria.
Association for Computational Linguistics.
Botha, J. A. and Blunsom, P. (2014). Compositional Morphology for Word Representations and Language
Modelling. In Proceedings of the 31st International Conference on Machine Learning, Beijing, China.
Brown, P. F., deSouza, P. V., Mercer, R. L., Della Pietra, V. J., and Lai, J. C. (1992). Class-based n-grammodels of natural language. Computational Linguistics, 18(4):467–479.
Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2012). Findings of the 2012
workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical
Machine Translation, pages 10–51, Montréal, Canada. Association for Computational Linguistics.
Carter, S. and Monz, C. (2011). Syntactic discriminative language model rerankers for statistical machinetranslation. Machine Translation Journal, 25(4):317–339.
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., and Federico, M. (2014). Report on the 11th IWSLT
evaluation campaign. In International Workshop on Spoken Language Translation (IWSLT), pages 2–
17, Lake Tahoe, California.
Chahuneau, V., Schlinger, E., Smith, N. A., and Dyer, C. (2013). Translating into morphologically richlanguages with synthetic phrases. In Proceedings of the 2013 Conference on Empirical Methods in
Natural Language Processing, pages 1677–1687, Seattle, Washington, USA. Association for Computational Linguistics.
Chen, S. F. and Goodman, J. (1999). An empirical study of smoothing techniques for language modeling.
Computer Speech and Language, 4(13):359–393.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust neuralnetwork joint models for statistical machine translation. In Proceedings of the 52nd Annual Meetingof the Association for Computational Linguistics, pages 1370–1380, Baltimore, Maryland. Association
for Computational Linguistics.
Fraser, A., Weller, M., Cahill, A., and Cap, F. (2012). Modeling inflection and word-formation in smt.
In Proceedings of the 13th Conference of the European Chapter of the Association for Computational
Linguistics, pages 664–674, Avignon, France. Association for Computational Linguistics.
Galley, M. and Manning, C. D. (2008). A simple and effective hierarchical phrase reordering model. In
EMNLP ’08: Proceedings of the Conference on Empirical Methods in Natural Language Processing,pages 848–856, Morristown, NJ, USA. Association for Computational Linguistics.
Green, S. and DeNero, J. (2012). A class-based agreement model for generating accurately inflected translations. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics,
pages 146–155, Jeju Island, Korea. Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 156

He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level
performance on imagenet classification. ArXiv e-prints.
Hopkins, M. and May, J. (2011). Tuning as ranking. In Proceedings of 2011 Conference on Empirical
Methods in Natural Language Processing (EMNLP’11).
Jean, S., Cho, K., Memisevic, R., and Bengio, Y. (2015). On using very large target vocabulary for neuralmachine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:
Long Papers), pages 1–10, Beijing, China. Association for Computational Linguistics.
Jeong, M., Toutanova, K., Suzuki, H., and Quirk, C. (2010). A discriminative lexicon model for complex
morphology. In The Ninth Conference of the Association for Machine Translation in the Americas.
Kingma, D. and Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of ICLR, San
Diego, CA, USA.
Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Proceedings of the2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395.
Koehn, P., Arun, A., and Hoang, H. (2008). Towards better machine translation quality for the GermanEnglish language pairs. In Proceedings of the Third Workshop on Statistical Machine Translation,
pages 139–142, Columbus, Ohio. Association for Computational Linguistics.
Koehn, P. and Hoang, H. (2007). Factored translation models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language
Learning (EMNLP-CoNLL), pages 868–876, Prague, Czech Republic. Association for Computational
Linguistics.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open Source
Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions,pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
Li, Z., Eisner, J., and Khudanpur, S. (2009). Variational decoding for statistical machine translation. In
Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics (ACL), pages
593–601, Singapore.
Luong, T., Socher, R., and Manning, C. (2013). Better word representations with recursive neural networks
for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language
Learning, pages 104–113, Sofia, Bulgaria. Association for Computational Linguistics.
Luong, T., Sutskever, I., Le, Q., Vinyals, O., and Zaremba, W. (2015). Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for
Computational Linguistics and the 7th International Joint Conference on Natural Language Processing(Volume 1: Long Papers), pages 11–19, Beijing, China. Association for Computational Linguistics.
Minkov, E., Toutanova, K., and Suzuki, H. (2007). Generating complex morphology for machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages
128–135.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 157

Müller, T., Schütze, H., and Schmid, H. (2012). A comparative investigation of morphological language
modeling for the languages of the European Union. In Proceedings of the 2012 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies,
pages 386–395, Montréal, Canada. Association for Computational Linguistics.
Noreen, E. W. (1989). Computer Intensive Methods for Testing Hypotheses. An Introduction. WileyInterscience.
Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation ofmachine translation. In Proceedings of the 40th Annual Meeting of the Association of Computational
Linguistics (ACL), pages 311–318, Philadelphia, PA.
Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
Post, M. and Gildea, D. (2008). Parsers as language models for statistical machine translation. In Proceedings of the Eighth Conference of the Association for Machine Translation in the Americas, pages
172–181.
Riezler, S. and Maxwell, J. T. (2005). On some pitfalls in automatic evaluation and significance testing for
MT. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MachineTranslation and/or Summarization, pages 57–64, Ann Arbor, Michigan. Association for Computational
Linguistics.
Schmid, H. (1994). Probabilistic part-of-speech tagging using decision trees. In International Conference
on New Methods in Language Processing, Manchester, UK.
Schwenk, H. (2007). Continuous space language models. Computer Speech and Language, 21:492–518.
Sharoff, S., Kopotev, M., Erjavec, T., Feldman, A., and Divjak, D. (2008). Designing and evaluatinga russian tagset. In Proceedings of the Sixth International Conference on Language Resources and
Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA).
http://www.lrec-conf.org/proceedings/lrec2008/.
Sima’an, K. (1996). Computational complexity of probabilistic disambiguation by means of treegrammars. In Proceedings of the 16th Conference on Computational Linguistics - Volume 2, COLING’96, pages 1175–1180. Association for Computational Linguistics.
Soricut, R. and Och, F. (2015). Unsupervised morphology induction using word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1627–1637, Denver, Colorado. Association for Computational Linguistics.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: A simple
way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–
1958.
Subotin, M. (2011). An exponential translation model for target language morphology. In Proceedingsof the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 230–238, Portland, Oregon, USA. Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 158

Toutanova, K., Suzuki, H., and Ruopp, A. (2008). Applying morphology generation models to machinetranslation. In Proceedings of ACL-08: HLT, pages 514–522, Columbus, Ohio. Association for Computational Linguistics.
Tran, K. M., Bisazza, A., and Monz, C. (2014). Word translation prediction for morphologically rich languages with bilingual neural networks. In Proceedings of the 2014 Conference on Empirical Methods
in Natural Language Processing (EMNLP), pages 1676–1688, Doha, Qatar. Association for Computational Linguistics.
Uszkoreit, J. and Brants, T. (2008). Distributed word clustering for large scale class-based language
modeling in machine translation. In Proceedings of ACL-08: HLT, pages 755–762, Columbus, Ohio.
Association for Computational Linguistics.
Zanchetta, E. and Baroni, M. (2005). Morph-it! a free corpus-based morphological resource for the italian
language. Corpus Linguistics 2005, 1(1).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 159

Bandit Structured Prediction for Learning from
Partial Feedback in Statistical Machine Translation
Artem Sokolov
Stefan Riezler∗

sokolov@cl.uni-heidelberg.de
riezler@cl.uni-heidelberg.de

Computational Linguistics and IWR∗
Heidelberg University, 69120 Heidelberg, Germany

Tanguy Urvoytanguy.urvoy@orange.com

Orange Labs, 2 Avenue Pierre Marzin, 22307 Lannion, France

Abstract
We present an approach to structured prediction from bandit feedback, called Bandit Structured
Prediction, where only the value of a task loss function at a single predicted point, instead ofa correct structure, is observed in learning. We present an application to discriminative reranking in Statistical Machine Translation (SMT) where the learning algorithm only has access toa 1 − BLEU loss evaluation of a predicted translation instead of obtaining a gold standard
reference translation. In our experiment bandit feedback is obtained by evaluating BLEU on
reference translations without revealing them to the algorithm. This can be thought of as asimulation of interactive machine translation where an SMT system is personalized by a user
who provides single point feedback to predicted translations. Our experiments show that ourapproach improves translation quality and is comparable to approaches that employ more informative feedback in learning.

1

Introduction

Learning from bandit1 feedback describes an online learning scenario, where on each of a sequence of rounds, a learning algorithm makes a prediction, and receives partial information interms of feedback to a single predicted point. In difference to the full information supervised
scenario, the learner does not know what the correct prediction looks like, nor what would have
happened if it had predicted differently. This scenario has (financially) important real world applications such as online advertising (Chapelle et al., 2014) that showcases a tradeoff between
exploration (a new ad needs to be displayed in order to learn its click-through rate) and exploitation (displaying the ad with the current best estimate is better in the short term). Crucially, inthis scenario it is unrealistic to expect more detailed feedback than a user click on the displayedad. Similar to the online advertising scenario, there are many potential applications of bandit
learning to NLP situations where feedback is limited for various reasons. For example, online learning has been applied successfully in interactive statistical machine translation (SMT)(Bertoldi et al., 2014; Denkowski et al., 2014; Green et al., 2014). Post-editing feedback clearlyis limited by its high cost and by the required expertise of users, however, current approaches
force the full information supervised scenario onto the problem of learning from user post-edits.
1 The name is inherited from a model where in each round a gambler pulls an arm of a different slot machine(“one-armed bandit”), with the goal of maximizing his reward relative to the maximal possible reward, without apriori
knowledge of the optimal slot machine.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 160

Bandit learning would allow to learn from partial user feedback that is easier and faster to obtain than full information. An example where user feedback is limited by a time constraint issimultaneous translation of a speech input stream (Cho et al., 2013). Clearly, it is unrealistic to
expect user feedback that goes beyond a one-shot user quality estimate of the predicted translation in this scenario. Another example is SMT domain adaptation where the translations ofa large out-of-domain model are re-ranked based on bandit feedback on in-domain data. Thiscan also be seen as a simulation of personalized machine translation where a given large SMT
system is adapted to a user solely by single-point user feedback to predicted structures.
The goal of this paper is to develop algorithms for structured prediction from bandit feedback, tailored to NLP problems. We investigate possibilities to “banditize” objectives such as
expected loss (Och, 2003; Smith and Eisner, 2006; Gimpel and Smith, 2010) that have been
proposed for structured prediction in NLP. Since most current approaches to bandit optimization rely on a multiclass classification scenario, the first challenge of our work is to adapt bandit
learning to structured prediction over exponentially large structured output spaces (Taskar et al.,
2004; Tsochantaridis et al., 2005). Furthermore, most theoretical work on online learning withbandit feedback relies on convexity assumptions about objective functions, both in the nonstochastic adversarial setting (Flaxman et al., 2005; Shalev-Shwartz, 2012) as well as in thestochastic optimization framework (Spall, 2003; Nemirovski et al., 2009; Bach and Moulines,2011). Our case is a non-convex optimization problem, which we analyze in the simple andelegant framework of pseudogradient adaptation that allows us to show convergence of the presented algorithm (Polyak and Tsypkin, 1973; Polyak, 1987).
The central contributions of this paper are:
• An algorithm for minimization of expected loss for structured prediction from bandit feedback, called Bandit Structured Prediction.
• An analysis of convergence of our algorithm in the stochastic optimization framework of
pseudogradient adaptation.
• An experimental evaluation on structured learning in SMT. Our experiment follows a simulation design that is standard in bandit learning, namely by simulating bandit feedbackby evaluating task loss functions against gold standard structures without revealing themto the learning algorithm.
As a disclaimer, we would like to note that improvements over traditional full-information
structured prediction cannot be expected from learning from partial feedback. Instead, thegoal is to investigate learning situations in which full information is not available. Similarly,
a comparison between our approach and dueling bandits (Yue and Joachims, 2009) is skewed
towards the latter approach that has access to two-point feedback instead of one-point feedback
as in our case. While it has been shown that querying the loss function at two points leads toconvergence results that closely resemble bounds for the full information case (Agarwal et al.,2010), such feedback is clearly twice as expensive and, depending on the application, might not
be elicitable from users.

2

Related Work

Stochastic Approximation. Online learning from bandit feedback dates back to Robbins(1952) who formulated the task as a problem of sequential decision making. His analysis, as
ours, is in a stochastic setting, i.e., certain assumptions are made on the probability distributionof feedback and its noisy realization. Stochastic approximation covers bandit feedback as noisy
observations which only allow to compute noisy gradients that equal true gradients in expectation. While the stochastic approximation framework is quite general, most theoretical analyses

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 161

of convergence and convergence rate are based on (strong) convexity assumptions (Polyak andJuditsky, 1992; Spall, 2003; Nemirovski et al., 2009; Bach and Moulines, 2011, 2013) and thus
not applicable to our case.
Non-Stochastic Bandits. Auer et al. (2002) initiated an active area of research on nonstochastic bandit learning, i.e., no statistical assumptions are made on the distribution of feedback, including models of feedback as a malicious choice of an adaptive adversary. The adversarial bandit setting has been extended to take context or side information into account,using models based on general linear classifiers (Auer et al., 2002; Langford and Zhang, 2007;
Chu et al., 2011). However, they formalize a multi-class classification problem that is not easily
scalable to general exponentially large structured output spaces. Furthermore, most theoreticalanalyses rely on online (strongly) convex optimization (Flaxman et al., 2005; Shalev-Shwartz,2012) thus limiting the applicability to our case.
Neurodynamic Programming. Bertsekas and Tsitsiklis (1996) cover optimization for neuralnetworks and reinforcement learning under the name of “neurodynamic programming”. Bothareas are dealing with non-convex objectives that lead to stochastic iterative algorithms. Interestingly, the available analyses of non-convex optimization for neural networks and reinforcement learning in Bertsekas and Tsitsiklis (1996), Sutton et al. (2000), or Bottou (2004) rely
heavily on Polyak and Tsypkin (1973)’s pseudogradient framework. We apply their simple andelegant framework directly to give asymptotic guarantees for our algorithm.
NLP Applications. In the area of NLP, recently algorithms for response-based learning havebeen proposed to alleviate the supervision problem by extracting supervision signals from taskbased feedback to system predictions. For example, Goldwasser and Roth (2013) presented an
online structured learning algorithm that uses positive executability of a semantic parse against
a database to convert a predicted parse into a gold standard structure for learning. Riezler et al.
(2014) apply a similar idea to SMT by using the executability of a semantic parse of a translated
database query as signal to convert a predicted translation into gold standard reference in structured learning. Sokolov et al. (2015) present a coactive learning approach to structured learningin SMT where instead of a gold standard reference a slight improvement over the prediction
is shown to be sufficient for learning. Saluja and Zhang (2014) present an incorporation of binary feedback into an latent structured SVM for discriminative SMT training. NLP applications
based on reinforcement learning have been presented by Branavan et al. (2009) or Chang et al.
(2015). Their model differs from ours in that it is structured as a sequence of states at which
actions and rewards are computed, however, the theoretical foundation of both types of models
can be traced back to Polyak and Tsypkin (1973)’s pseudogradient framework .

3

Expected Loss Minimization under Full Information

The expected loss learning criterion for structured prediction is defined as a minimization of the
expectation of a task loss function with respect to the conditional distribution over structured
outputs (Gimpel and Smith, 2010; Yuille and He, 2012). More formally, let X be a structured
input space, let Y(x) be the set of possible output structures for input x, and let ∆y : Y → [0, 1]
quantify the loss ∆y (y 0 ) suffered for making errors in predicting y 0 instead of y; as a rule,∆y (y 0 ) = 0 iff y = y 0 . Then, for a data distribution p(x, y), the learning criterion is defined asminimization of the expected loss
Ep(x,y)pw (y0 |x) [∆y (y 0 )] =Xx,yProceedings of MT Summit XV, vol.1: MT Researchers' Trackp(x, y)X∆y (y 0 )pw (y 0 |x).

(1)

y 0 ∈Y(x)Miami, Oct 30 - Nov 3, 2015 | p. 162

Assume further that output structures given inputs are distributed according to an underlying
Gibbs distribution (a.k.a. conditional exponential or log-linear model)pw (y|x) = exp(w> φ(x, y))/Zw (x),where φ : X × Y → Rd is a joint feature representation of inputs and outputs, w ∈ Rd is aweight vector, and Zw (w) is a normalization constant.
The natural rule for prediction or inference is according to the minimum Bayes risk principleXŷw (x) = arg min∆y (y 0 )pw (y 0 |x).
(2)y∈Y(x)

y 0 ∈Y(x)This requires an evaluation of ∆y (y 0 ) over the full output space, which is standardly avoided inpractice by performing inference according to a maximum a posteriori (MAP) criterion (whichequals criterion (2) for the special case of ∆y (y 0 ) = 1[y 6= y 0 ] where 1[s] evaluates to 1 ifstatement s is true, 0 otherwise)ŷw (x) = arg max pw (y|x)(3)

y∈Y(x)= arg max w> φ(x, y).
y∈Y(x)

Furthermore, since it is unfeasible to take expectations over the full space X × Y to perform minimization of objective (1), in the full information case the data distribution p(x, y) is
PTapproximated by the empirical distribution p̃(x, y) = T1 t=0 1[x = xt ]1[y = yt ] for i.i.d.
training data {(xt , yt )}Tt=0 . This yields the objectiveTEp̃(x,y)pw (y0 |x) [∆y (y 0 )] =1X X
T t=0 0∆yt (y 0 )pw (y 0 |xt ).

(4)y ∈Y(xt )While being continuous and differentiable, the expected loss criterion is typically nonconvex. For example, in SMT, expected loss training for the standard task loss BLEU leads tohighly non-convex optimization problems. Despite of this, most approaches rely on gradientdescent techniques for optimization (see Och (2003), Smith and Eisner (2006), He and Deng(2012), Auli et al. (2014), Wuebker et al. (2015), inter alia) by following the opposite directionof the gradient of (4):∇Ep̃(x,y)pw (y0 |x) [∆y (y 0 )]hi= Ep̃(x,y) Epw (y0 |x) [∆y (y 0 )φ(x, y 0 )] − Epw (y0 |x) [∆y (y 0 )] Epw (y0 |x) [φ(x, y 0 )]hi= Ep̃(x,y)pw (y0 |x) ∆y (y 0 )(φ(x, y 0 ) − Epw (y0 |x) [φ(x, y 0 )]) .

4

Bandit Structured Prediction

Bandit feedback in structured prediction means that the gold standard output structure y, withrespect to which the objective function is evaluated, is not revealed to the learner. Thus we canneither calculate the gradient of the objective function (4) nor evaluate the task loss ∆ as in thefull information case. A solution to this problem is to pass the evaluation of the loss function
to the user, i.e, we access the loss directly through user feedback without assuming existence ofa fixed reference y. We indicate this by dropping the subscript y in ∆(y 0 ). Assuming a fixed,

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 163

Algorithm 1 Bandit Structured Prediction
1: Input: sequence of learning rates γt
2: Initialize w0
3: for t = 0, . . . , T do
4:
5:
6:
7:
8:

Observe xt
Calculate Epwt (y0 |xt ) [φ(xt , y 0 )]Sample ỹt ∼ pwt (y 0 |xt )Obtain feedback ∆(ỹt )Update wt+1 = wt − γt ∆(ỹt )(φ(xt , ỹt ) − Epwt (y0 |xt ) [φ(xt , y 0 )])unknown distribution p(x) over input structures, we can formalize the following new objective
for expected loss minimization in a bandit setup
J(w) = Ep(x)pw (y0 |x) [∆(y 0 )]
X
X
=
p(x)
∆(y 0 )pw (y 0 |x).
x(5)y 0 ∈Y(x)Optimization of this objective is then as follows:
1. We assume a sequence of input structures xt , t = 1, . . . , T that are generated by a fixed,
unknown distribution p(x).
2. We use a Gibbs distribution estimate as a sampling distribution to perform simultaneous
exploration / exploitation on output structures (Abernethy and Rakhlin, 2009).
3. We use feedback to the sampled output structures to construct a parameter update rule thatis an unbiased estimate of the true gradient of objective (5).
4.1

Algorithm

Algorithm 1 implements these ideas as follows: We assume as input a given learning rate schedule (line 1) and a deterministic initialization w0 of the weight vector (line 2). For each random
i.i.d. input structure xt , we calculate the expected feature count (line 5). This can be done exactly, provided the underlying graphical model permits a tractable calculation, or for intractablemodels, with MCMC sampling. We then sample an output structure ỹt from the Gibbs model(line 6). If the number of output options is small, this is done by sampling from a multinomial
distribution. Otherwise, we use a Perturb-and-MAP approach (Papandreou and Yuille, 2011),
restricted to unary potentials, to obtain an approximate Gibbs sample without waiting for the
MC chain to mix. Finally, an update in the negative direction of the instantaneous gradient,evaluated at the input structure xt (line 8), is performed.
Intuitively, the algorithm compares the sampled feature vector to the average feature vector,
and performs a step into the opposite direction of this difference, the more so the higher the lossof the sampled structure is. In the extreme case, if the sampled structure is correct (∆(ỹt ) = 0),
no update is performed.
4.2

Stochastic Approximation Analysis

The construction of the update in Algorithm 1 as a stochastic realization of the true gradient
allows us to analyze the algorithm as a stochastic approximation algorithm. We show how
our case can be fit in the pseudogradient adaptation framework of Polyak and Tsypkin (1973)
which gives asymptotic guarantees for non-convex and convex objectives. They characterize an

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 164

iterative process
wt+1 = wt − γt st(6)where γt ≥ 0 is a learning rate, wt and st are vectors in Rd with fixed w0 , and the distributionof st depends on w0 , . . . , wt . For a given lower bounded and differentiable function J(w) with
Lipschitz continuous gradient ∇J(w), that is, for all w, w0 , for all L ≥ 0,k∇J(w + w0 ) − ∇J(w)k ≤ Lkw0 k,(7)

the vector st in process (6) is said to be a pseudogradient of J(w) if∇J(wt )> E[st ] ≥ 0,

(8)

where the expectation is taken over all sources of randomness. Intuitively, the pseudogradient st
is on average at an acute angle with the true gradient, meaning that −st is on average a directionof decrease of the functional J(w).
In order to show convergence of the iterative process (6), besides conditions (7) and (8),only mild conditions on boundedness of the pseudogradientE[kst k2 ] < ∞,(9)and on the use of a decreasing learning rate satisfying
γt ≥ 0,∞X

γt = ∞,

t=0∞X

γt2 < ∞,

(10)

t=0

are necessary. Under the exclusion of trivial solutions such as st = 0, the following convergenceassertion can be made:Theorem 1 (Polyak and Tsypkin (1973), Thm. 1) Under conditions (7)–(10), for any w0 inprocess (6):J(wt ) → J ∗ a.s., and lim ∇J(wt )> E(st ) = 0.
t→∞

The significance of the theorem is that its conditions can be checked easily, and it applies to awide range of cases, including non-convex functions, in which case the convergence point J ∗is a critical point of J(w).
The convergence analysis of Theorem 1 can be applied to Algorithm 1 as follows: First
note that we can define our functional J(w) with respect to expectations over the full space of X
as J(w) = Ep(x)pw (y0 |x) [∆(y 0 )]. This means, convergence of the algorithm can be understood
directly as a generalization result that extends to unseen data. In order to show this result, wehave to verify conditions (7)–(10). It is easy to show that condition (7) holds for our functional
J(w). Next we match the update in Algorithm 1 to a vector
st = ∆(ỹt )(φ(xt , ỹt ) − Epwt (y0 |xt ) [φ(xt , y 0 )]).
Taking the expectation of st yields Ep(x)pwt (y0 |x) [st ] = ∇J(wt ) such that condition (8) issatisfied by∇J(wt )> Ep(x)pwt (y0 |x) [st ] = k∇J(wt )k2 ≥ 0.
Assuming kφ(x, y 0 )k ≤ R and ∆(y 0 ) ∈ [0, 1] for all x, y 0 , condition (9) is satisfied byEp(x)pwt (y0 |x) [kst k2 ] ≤ 4R2 .
For a decreasing learning rate, e.g., γt = 1/t, condition (10) holds, such that convergence to a
critical point of the expected risk follows according to Theorem 1.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 165

Algorithm Structured Dueling Bandits
1: Input: γ, δ, w0
2: for t = 0, . . . , T do
3:
4:
5:
6:
7:
8:
9:
10:

5

Observe xt
Sample unit vector ut uniformly
Set wt0 = wt + δut
Compare ∆(ŷwt (xt )) to ∆(ŷwt0 (xt ))if wt0 wins thenwt+1 = wt + γut
else
wt+1 = wt

Structured Dueling Bandits

For purposes of comparison, we present an extension of Yue and Joachims (2009)’s duelingbandits algorithm to structured prediction problems. The original algorithm is not specificallydesigned for structured prediction problems, but it is generic enough to be applicable to suchproblems when the quality of a parameter vector can be proxied through loss evaluation of aninferred structure.
The Structured Dueling Bandits algorithm compares a current weight vector wt with aneighboring point wt0 along a direction ut , performing exploration (controlled by δ, line 5) byprobing random directions, and exploitation (controlled by γ, line 8) by taking a step into thewinning direction. The comparison step in line 6 is adapted to structured prediction from theoriginal algorithm of Yue and Joachims (2009) by comparing the quality of wt and wt0 via anevaluation of the losses ∆(ŷwt (xt )) and ∆(ŷwt0 (xt )) of the structured arms corresponding to
MAP prediction (3) under wt and wt0 , respectively.
Further, note that the Structured Dueling Bandit algorithm requires access to a two-point
feedback instead of a one-point feedback as in case of Bandit Structured Prediction (Algorithm 1). It has been shown that two-point feedback leads to convergence results that are close
to those for learning from full information Agarwal et al. (2010). However, two-point feedback
is twice as expensive as one-point feedback, and most importantly, such feedback might not beelicitable from users in real-world situations where feedback is limited by time- and resourceconstraints. This limits the range of applications of Dueling Bandits to real-world interactive
scenarios.

6

Experiments

Our experimental design follows the standard of simulating bandit feedback by evaluating task
loss functions against gold standard structures without revealing them to the learner. We compare the proposed Structured Bandit Prediction algorithm to Structured Dueling Bandits, andreport results by test set evaluations of the respective loss functions under MAP inference. Furthermore, we evaluate models at different iterations according to their loss on the test set inorder to visualize the empirical convergence behavior of the algorithms.
All experiments with bandit algorithms perform online learning for parameter estimation,
and apply early stopping to choose the last model in a learning sequence for online-to-batch
conversion at test time. Final results for bandit algorithms are averaged over 5 independentruns.
In this experiment, we present bandit learning for the structured 1 − BLEU loss used in
SMT. The setup is a reranking approach to SMT domain adaptation where the k-best list ofan out-of-domain model is re-ranked (without re-decoding) based on bandit feedback from in-Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 166

full information
in-domain SMT out-domain SMT
0.28540.2579bandit information
DuelingBandit BanditStruct
0.2731±0.001

0.2705±0.001

Table 1: Corpus BLEU (under MAP decoding) on test set for SMT domain adaptation from
Europarl to NewsCommentary by k-best reranking.

0.275

corpus-BLEU

0.270
Dueling
BanditStructout-domain SMT

0.265

0.260

0.255

50

100

150

200
250
300
iteration×5000

350

400

450

500

Figure 1: Corpus-BLEU on test set for early stopping at different iterations for the SMT task.
domain data. This can also be seen as a simulation of personalized machine translation wherea given large SMT system is adapted to a user solely by single-point user feedback to predictedstructures.
We use the data from the WMT 2007 shared task for domain adaptation experiments ina popular benchmark setup from Europarl to NewsCommentary for French-to-English (Koehn
and Schroeder, 2007; Daumé and Jagarlamudi, 2011). We tokenized and lowercased our datausing the moses toolkit, and prepared word alignments by fast align (Dyer et al., 2013).
The SMT setup is phrase-based translation using non-unique 5,000-best lists from moses(Koehn et al., 2007) and a 4-gram language model (Heafield et al., 2013).
The out-of-domain baseline SMT model is trained on 1.6 million parallel Europarl dataand includes the English side of Europarl and in-domain NewsCommentary in the language
model. The model uses 15 dense features (6 lexicalized reordering features, 1 distortion, 1 outof-domain and 1 in-domain language model, 1 word penalty, 5 translation model features) thatare tuned with MERT (Och, 2003) on a dev set of Europarl data (dev2006, 2,000 sentences).
The full-information in-domain SMT model gives an upper bound by MERT tuning the out-ofdomain model on in-domain development data (nc-dev2007, 1,057 sentences). MERT runs
for both baseline models were repeated 7 times and median results are reported.
Learning under bandit feedback started at the learned weights of the out-of-domain medianmodel. It uses the parallel NewsCommentary data (news-commentary, 43,194 sentences)to simulate bandit feedback, by evaluating the sampled translation against the gold standard
reference using as loss function ∆ a smoothed per-sentence 1 − BLEU (by flooring zero n-gramProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 167

counts to 0.01). The meta-parameters of Dueling Bandits and Bandit Structured Prediction were
adjusted by online optimization of cumulative per-sentence 1−BLEU on a small in-domain dev
set (nc-devtest2007, 1,064 parallel sentences). The final results are obtained by online-tobatch conversion where the model trained for 100 epochs on 43,194 in-domain training data isevaluated on a separate in-domain test set (nc-test2007, 2,007 sentences).
Table 1 shows that the results for Bandit Structured Prediction and Dueling Bandits arevery close, however, both are significant improvements over the out-of-domain SMT model thateven includes an in-domain language model. We show the standard evaluation of the corpusBLEU metric evaluated under MAP inference. The range of possible improvements is given by
the difference of the BLEU score of the in-domain model and the BLEU score of the out-ofdomain model – nearly 3 BLEU points. Bandit learning can improve the out-of-domain baseline
by about 1.26 BLEU points (Bandit Structured Prediction) and by about 1.52 BLEU points (Dueling Bandits). All result differences are statistically significant at a p-value of 0.0001, using
an Approximate Randomization test (Riezler and Maxwell, 2005; Clark et al., 2011). Figure
1 shows that per-sentence BLEU is a difficult metric to provide single-point feedback, yielding a non-smooth progression of loss values against iterations for Bandit Structured Prediction.
The progression of loss values is smoother and empirical convergence speed is faster for Dueling Bandits since it can exploit preference judgements instead of having to trust real-valued
feedback.

7

Discussion

We presented an approach to Bandit Structured Prediction that is able to learn from feedback inform of an evaluation of a task loss function for single predicted structures. Our experimental
evaluation showed promising results, both compared to Structured Dueling Bandits that employ
two-point feedback, and compared to full information scenarios where the correct structure isrevealed.
Our approach shows its strength where correct structures are unavailable and two-point
feedback is infeasible. In future work we would like to apply bandit learning to scenarioswith limited human feedback such as the interactive SMT applications discussed above. In
such scenarios, per-sentence BLEU might not be the best metric to quantify feedback. We
will instead investigate feedback based on HTER (Snover et al., 2006), or based on judgements
according to Likert scales (Likert, 1932).

Acknowledgements
This research was supported in part by DFG grant RI-2221/2-1 “Grounding Statistical MachineTranslation in Perception and Action”.

References√
Abernethy, J. and Rakhlin, A. (2009). An efficient bandit algorithm for T regret in online
multiclass prediction? In Conference on Learning Theory (COLT), Montreal, Canada.
Agarwal, A., Dekel, O., and Xiao, L. (2010). Optimal algorithms for online convex optimization
with multi-point bandit feedback. In Conference on Learning Theory (COLT), Haifa, Israel.
Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. (2002). The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48–77.
Auli, M., Galley, M., and Gao, J. (2014). Large-scale expected BLEU training of phrase-based
reordering models. In Conference on Empirical Methods in Natural Language Processing(EMNLP).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 168

Bach, F. and Moulines, E. (2011). Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems (NIPS),
Granada, Spain.
Bach, F. and Moulines, E. (2013). Non-strongly-convex smooth stochastic approximation withconvergence rate O(1/n). In Advances in Neural Information Processing Systems (NIPS),
Lake Tahoe, CA, USA.
Bertoldi, N., Simianer, P., Cettolo, M., Wäschle, K., Federico, M., and Riezler, S. (2014).
Online adaptation to post-edits for phrase-based statistical machine translation. MachineTranslation, 29:309–339.
Bertsekas, D. P. and Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific.
Bottou, L. (2004). Stochastic learning. In Bousquet, O. and von Luxburg, U., editors, Advanced
Lectures on Machine Learning, pages 146–168.
Branavan, S., Chen, H., Zettlemoyer, L. S., and Barzilay, R. (2009). Reinforcement learning for
mapping instructions to actions. In Proceedings of the 47th Annual Meeting of the ACL andthe 4th IJCNLP of the AFNLP, Suntec, Singapore.
Chang, K.-W., Krishnamurthy, A., Agarwal, A., Daume, H., and Langford, J. (2015). Learning
to search better than your teacher. In International Conference on Machine Learning (ICML),
Lille, France.
Chapelle, O., Manavaglu, E., and Rosales, R. (2014). Simple and scalable response prediction
for display advertising. ACM Transactions on Intelligent Systems and Technology, 5(4).
Cho, E., Fügen, C., Hermann, T., Kilgour, K., Mediani, M., Mohr, C., Niehues, J., Rottman, K.,
Saam, C., Stüker, S., and Waibel, A. (2013). A real-world system for simultaneous translationof German lectures. In Interspeech, Lyon, France.
Chu, W., Li, L., Reyzin, L., and Schapire, R. E. (2011). Contextual bandits with linear payoff
functions. In International Conference on Artificial Intelligence and Statistics (AISTATS),
Fort Lauderdale, FL, USA.
Clark, J., Dyer, C., Lavie, A., and Smith, N. (2011). Better hypothesis testing for statisticalmachine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics (ACL’11), Portland, OR.
Daumé, H. and Jagarlamudi, J. (2011). Domain adaptation for machine translation by mining unseen words. In Meeting of the Association for Computational Linguistics: Human
Language Technologies (ACL-HLT), Portland, OR, USA.
Denkowski, M., Dyer, C., and Lavie, A. (2014). Learning from post-editing: Online model
adaptation for statistical machine translation. In Conference of the European Chapter of the
Association for Computational Linguistics (EACL), Gothenburg, Sweden.
Dyer, C., Chahuneau, V., and Smith, N. A. (2013). A simple, fast, and effective reparameterization of IBM Model 2. In Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA,
USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 169

Flaxman, A. D., Kalai, A. T., and McMahan, H. B. (2005). Online convex optimization in thebandit setting: gradient descent without a gradient. In ACM-SIAM Symposium on Discrete
Algorithms (SODA), Philadelphia, PA.
Gimpel, K. and Smith, N. A. (2010). Softmax-margin training for structured log-linear models.
Technical Report CMU-LTI-10-008, Carnegie Mellon University, Pittsburgh, PA, USA.
Goldwasser, D. and Roth, D. (2013). Learning from natural instructions. Machine Learning,
94(2):205–232.
Green, S., Wang, S. I., Chuang, J., Heer, J., Schuster, S., and Manning, C. D. (2014). Humaneffort and machine learnability in computer aided translation. In Conference on Empirical
Methods in Natural Language Processing (EMNLP), Doha, Qatar.
He, X. and Deng, L. (2012). Maximum expected BLEU training of phrase and lexicon translation models. In Meeting of the Association for Computational Linguistics (ACL), Jeju Island,
Korea.
Heafield, K., Pouzyrevsky, I., Clark, J. H., and Koehn, P. (2013). Scalable modified KneserNey language model estimation. In Meeting of the Association for Computational Linguistics(ACL), Sofia, Bulgaria.
Koehn, P., Hoang, H., Birch, A., Callison-Birch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007).
Moses: Open source toolkit for statistical machine translation. In ACL Demo and Poster
Sessions, Prague, Czech Republic.
Koehn, P. and Schroeder, J. (2007). Experiments in domain adaptation for statistical machinetranslation. In Workshop on Statistical Machine Translation, Prague, Czech Republic.
Langford, J. and Zhang, T. (2007). The epoch-greedy algorithm for multi-armed bandits withside information. In Advances in Neural Information Processing Systems (NIPS). Vancouver,
Canada.
Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology,
140:5–55.
Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. (2009). Robust stochastic approximation
approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574–1609.
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Conferenceof the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies (HLT-NAACL), Edmonton, Canada.
Papandreou, G. and Yuille, A. (2011). Perturb-and-map random fields: Using discrete optimization to learn and sample from energy models. In IEEE International Conference on
Computer Vision (ICCV), Barcelona, Spain.
Polyak, B. T. (1987). Introduction to Optimization. Optimization Software, Inc., New York.
Polyak, B. T. and Juditsky, A. B. (1992). Acceleration of stochastic approximation by averaging.
SIAM Journal on Control and Optimization, 30(4):838–855.
Polyak, B. T. and Tsypkin, Y. Z. (1973). Pseudogradient adaptation and training algorithms.
Automation and remote control: a translation of Avtomatika i Telemekhanika, 34(3):377–397.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 170

Riezler, S. and Maxwell, J. (2005). On some pitfalls in automatic evaluation and significancetesting for MT. In Proceedings of the ACL-05 Workshop on Intrinsic and Extrinsic Evaluation
Measures for MT and/or Summarization, Ann Arbor, MI.
Riezler, S., Simianer, P., and Haas, C. (2014). Response-based learning for grounded machinetranslation. In Meeting of the Association for Computational Linguistics (ACL), Baltimore,
MD, USA.
Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of theAmerican Statistical Society, 55:527–535.
Saluja, A. and Zhang, Y. (2014). Online discriminative learning for machine translation withbinary-valued feedback. Machine Translation, 28:69–90.
Shalev-Shwartz, S. (2012). Online learning and online convex optimization. Foundations andTrends in Machine Learning, 4(2):107–194.
Smith, D. A. and Eisner, J. (2006). Minimum risk annealing for training log-linear models.
In International Committee on Computational Linguistics and the Association for Computational Linguistics (COLING-ACL), Sydney, Australia.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation
edit rate with targeted human annotation. In Conference of the Association for MachineTranslation in the Americas (AMTA), Cambridge, MA, USA.
Sokolov, A., Riezler, S., and Cohen, S. B. (2015). A coactive learning view of online structured
prediction in statistical machine translation. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), Beijing, China.
Spall, J. C. (2003). Introduction to Stochastic Search and Optimization: Estimation, Simulation,
and Control. Wiley.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (2000). Policy gradient methods
for reinforcement learning with function approximation. In Advances in Neural Information
Processings Systems (NIPS), Vancouver, Canada.
Taskar, B., Klein, D., Collins, M., Koller, D., and Manning, C. (2004). Max-margin parsing.
In Conference on Empirical Methods in Natural Language Processing (EMNLP), Barcelona,
Spain.
Tsochantaridis, I., Joachims, T., Hofmann, T., and Altun, Y. (2005). Large margin methods
for structured and interdependent output variables. Journal of Machine Learning Research,
5:1453–1484.
Wuebker, J., Muehr, S., Lehnen, P., Peitz, S., and Ney, H. (2015). A comparison of update strategies for large-scale maximum expected bleu training. In Conference of the North American
Chapter of the Association for Computational Linguistics Human Language Technologies(NAACL-HLT), Denver, CO, USA.
Yue, Y. and Joachims, T. (2009). Interactively optimizing information retrieval systems as a dueling bandits problem. In International Conference on Machine Learning (ICML), Montreal,
Canada.
Yuille, A. and He, X. (2012). Probabilistic models of vision and max-margin methods. Frontiersof Electrical and Electronic Engineering, 7(1).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 171

An Empirical Study of Segment Prioritization for
Incrementally Retrained Post-Editing-Based SMT
Jinhua Du
Ankit Srivastava
Andy Way

jdu@computing.dcu.ie
asrivastava@computing.dcu.ie
away@computing.dcu.ie
ADAPT Centre, School of Computing, Dublin City University, Ireland

Alfredo Maldonado-Guerra
David Lewis

maldonaa@tcd.ie
dave.lewis@cs.tcd.ie

ADAPT Centre, Trinity College Dublin, Ireland

Abstract
Post-editing the output of a statistical machine translation (SMT) system to obtain high-qualitytranslation has become an increasingly common application of SMT, which henceforth we refer to as post-editing-based SMT (PE-SMT). PE-SMT is often deployed as an incrementally
retrained system that can learn knowledge from human post-editing outputs as early as possible
to augment the SMT models to reduce PE time. In this scenario, the order of input segments
plays a very important role in reducing the overall PE time. Under the active learning-based(AL) framework, this paper provides an empirical study of several typical segment prioritization methods, namely the cross entropy difference (CED), n-grams, perplexity (PPL) andtranslation confidence, and verifies their performance on different data sets and language pairs.
Experiments in a simulated setting show that the confidence of translations performs best withdecreases of 1.72-4.55 points TER absolute on average compared to the sequential PE-based
incrementally retrained SMT.

1 Introduction
In recent years, SMT systems have been widely deployed into the translator’s workflow in thelocalization and translation industry to improve productivity, refereed to as post-editing-based
SMT. However, in most cases, current SMT systems cannot generate high-quality translations,so human effort is usually required. With the help of incrementally improved SMT systems, theproductivity of translators/post-editors can be significantly increased due to the early learningof knowledge from the previously post-edited segments (Guerberof, 2009; Plitt and Masselot,
2010; Carl et al., 2011; OBrien, 2011; Zhechev, 2012; Guerberof, 2013). Furthermore, the orderof input segments has been found to have a significant impact on the overall PE-time, i.e., an
optimized sequence of input segments can reduce the overall PE-time compared to the typicalchronological sequence (Dara et al., 2014).
Regarding the PE-SMT, the incremental retraining can be roughly categorized into two different scenarios, namely the segment-level online incremental retraining (segment mode) (Levenberg et al., 2010; Denkowski et al., 2014) and batch-level incremental retraining (batch mode) (Hardt and Elming, 2010; Henrı́quez Q. et al., 2011; Mathur et al., 2013; Simard and Foster,
2013; Dara et al., 2014; Bertoldi et al., 2014). The former takes one post-edited segment per retraining cycle to immediately update the models, which requires rapid incremental processing ofProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 172

the word alignment, phrase/rule generation, language model and parameters tuning etc., while
the latter firstly accumulates a batch of segments, and then performs the incremental retraining
process to update the system. The batch-level mode can perform the incremental retraining
process in the background while the translators/post-editors continue to work on the next batchof segments. From the point of view of parameter estimation, the former can promptly adapt its
feature weights to the newly post-edited segment and learn the translator’s knowledge, but thefrequent change of weights might make the system unstable; the latter adapts the parameters
on an average level of segments in a batch, which can relatively keep the system more robust,
however, it cannot learn the knowledge as early as possible and cannot demonstrate a quick response to translator’s practice and preference. In our task, in order to better show the impact of
the order of the input segments on the PE time, we select the batch-level incremental retraining
SMT as our experimental platform.
The order in which post-editors review and correct machine-translated segments has an
impact on the evaluation score (PE time in our case) of the incrementally retrained PE-SMT
systems. That is, assuming that post-editors work on batches, and after post-editing each batch
the SMT system is dynamically retrained, the order of segments in these batches will have an
impact on how quickly the overall translation performance grows. The expectation is that if
post-editors work first on the segments that are most informative or most difficult to translate
for SMT, the SMT system will learn most from the corrections, and as a consequence, translation quality will increase more steeply in the following retraining iterations. In doing so, it
is possible to devise a process in which the most experienced and potentially more expensive
post-editors/translators tackle the first few batches of segments, leaving the rest of the segments
to either be worked upon by less experienced and potentially cheaper post-editors/translators,
or to be left completely unedited, depending on the quality vs. cost requirements of the actual
translation project at hand. Therefore, in this paper, we carry out an empirical study on several different mainstream segment prioritization strategies, and then investigate the factors thatclosely correlate to the effectiveness of the methods.
The main contributions of this paper include:• Confidence of translation and perplexity methods are proposed to reorder the input segments in the AL-based dynamically retrained SMT.
• A deep comparison and investigation of different segment prioritization methods for PESMT using different data sets and language pairs.
• A detailed data and results analysis of the correlation between the reordering score and thefactors.
• Our experiments show that the unnormalized confidence of translations performs best in
all tasks and gains around 1.72 to 4.55 TER (Snover et al., 2006) absolute on average.

2 Related Work
The purpose of the input segment prioritization is to reduce the overall PE time to improve
productivity and to reduce the cost. In this scenario, the involvement of human effort implies
that the segment prioritization process can be regarded as AL framework-based PE-SMT. In thisframework, the input segments are ranked based on the information or uncertainty contained
therein. In this section, we will introduce the related work in terms of two aspects: AL-based
framework for PE-SMT, and the incrementally retrained PE-SMT.
The practical active learning framework for SMT was firstly proposed in Haffari et al.
(2009) where a number of high-quality parallel data are acquired from large-scale monolingual

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 173

data. Relatively inexpensive human costs are iteratively used to translate information-rich sentences. Experimental results show that generally the translation unit-based selection strategies,
namely phrases and n-grams, performed best compared to other methods such as random selection, translation confidence, inverse model etc. However, in their work, the AL frameworkis used for low-resource SMT rather than the PE-SMT scenario. Furthermore, it is a static retraining process in which the test set is constant per iteration, and the retraining procedure is
not incremental.
Gonzalez-Rubio et al. (2012) apply AL to the interactive MT in which AL techniques areused to select the most informative sentences to reduce human effort for a given translation
quality. Experimental results show that applying AL techniques in an interactive MT setting
can prove a better tradeoff between required human effort and final translation quality.
To the best of our knowledge, the most relevant previous work is that of Dara et al. (2014),
which proposes a Cross Entropy Difference (CED) criterion to prioritize input segments in an
AL framework for PE-based incremental MT update applications. The fundamental goal is to
reduce the overall PE time rather than aiming at reducing human effort. The proposed CED
method calculates the rank score by the entropy difference of a sentence s in the untranslated
corpus (or the incremental data) U and the current training corpus L. The higher the score,
the more informative the sentence is and the greater the possibility of the sentence being more
highly ranked. Experimental results on the industrial data in a simulated setting show that the
proposed method significantly reduces the TER score compared to the random and sequential
order. In their work, Dara et al. (2014) used batch mode for the incrementally retrained PESMT with the CED only considering the information of the source side of the data in orderto keep the costs to a minimum for the commercial PE MT applications. However, in thepractical scenario, we can take the information of the target side (e.g. translations) into accountin batch mode without a significant increase in extra time and human costs by pre-translating
the remaining batches in the background while post-editing the current batch. In doing so, we
propose to use the confidence of MT translations to rank the segments.
Regarding the incrementally retrained SMT, the most challenging and time-consuming
steps are the word alignment and the phrase/rule generation. Ortiz-Martinez et al. (2010) incrementally update the feature values of the phrase table by extracting new phrases from the new
sentence pairs based on the pre-stored statistics related to the feature scores. Hardt and Elming(2010) propose a sentence-level retraining scheme in which a local phrase table is created andincrementally updated as a file is translated and post-edited. In their work, a modified revisionof GIZA++ (Och and Ney, 2003) is used to approximate word alignments of a newly translated
sentence to reduce the incremental training time, and then an additional phrase table is produced
from the newly aligned sentences with higher priority. The experiments show the efficiency of
the incremental retraining system.
In the incrementally retrained PE-SMT system, suffix arrays (Callison-Burch et al., 2005;
Zhang and Vogel, 2005) are a very efficient technique for the incremental retraining process.
Levenberg et al. (2010) introduce a dynamic suffix array to incorporate new training text to thecurrent training data. Denkowski et al. (2014) propose an online model adaptation for PE-SMT
in which three methods are used for incremental model adaptation: adding new data to a suffix
array-indexed bitext from which grammars are extracted, updating a Bayesian language modelwith incremental data, and using an online MIRA (Crammer and Singer, 2003) to update theparameters. The simulated experiments show that significant improvement in MT quality isachieved when these methods are used individually and in tandem. Germann (2014) proposesa dynamic phrase table strategy for an interactive PE-SMT that computes phrase table entries
on demand by sampling a suffix array-indexed bitext. Experiments show that without loss oftranslation quality, the sampling phrase table achieves good performance in terms of speed. In

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 174

our task, we use this dynamic phrase table for incremental retraining in Moses (Koehn et al.,2007).

3 The Incrementally Retrained PE-SMT Paradigm
In the post-editing scenario, humans are involved to continuously edit MT outputs into highquality translations. As discussed in Dara et al. (2014), the fundamental goal of input segment
prioritization for PE-SMT is to reduce the overall PE time taken to complete a translation job.
The crucial step is to first select the most uncertain sentences or most informative sentences
for post-editing in order to learn as much knowledge as possible from these sentences. The
workflow of an AL-based incrementally retrained PE-SMT system is as shown in Figure 1.

Figure 1: The workflow of active learning-based incrementally retrained PE-SMTIn Figure 1, the translations of the input segments are post-edited and the corrected translations are used for incremental update of the models. The process is repeated until the incremental data (or translation job) is finished. In a typical PE scenario, post-editors are presented
with SMT outputs in chronological order (i.e. sequentially) of the input segments. However, an
optimized order of the input segments in a translation job can significantly reduce the overall
PE time.
In our scenario, the PE time is simulated using TER score between the MT output and thereference translations for the sentences in each batch. The overall performance of the segment
prioritization method is evaluated by the average TER score for all the batches.

4 Methodology
To prioritize the input segments, an importance score or uncertainty score for the sentence s
must be calculated under some metric, which can be formalized as follows:
Given an initial parallel training corpus L := {(fi , ei )} and a monolingual corpus (translation job) U := {fj }, the goal of the segment prioritization is to rank a sentence s with thescore ϕ(s) under a scoring metric F . This process can be defined as a triple in (1):ϕ(s) = F (s, U, L)(1)

Clearly, we can see that the scoring metric F is most important in a prioritization algorithm.
We use the sequential order of input segments as our baseline.1 In the following sections,
1 The random and sequential methods have similar performance in Dara et al. (2014), so we only use sequential as
the baseline.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 175

we carry out an empirical study on different information-driven prioritization methods.
4.1

Confidence of Translations (Confidence)In the decoding process, a translation output ê is produced with the probability p(e|f ) thatis calculated by different features, such as bidirectional lexical probabilities, language modeletc. It can be treated as a confidence score for the translation because it reflects the translationdifficulty or uncertainty of the source segment in some sense.
Generally, the probability p(e|f ) is influenced by two aspects, namely the out-ofvocabulary (OOV) words and the sentence length (c.f. Section 6). For human translators, these
two aspects are often more time-consuming. That is, a long sentence with many OOVs willtake much more time to post-edit. Therefore, intuitively, the unnormalized confidence score oftranslations can better measure the uncertainty of a sentence.
Based on the confidence of translations, we rank the input segments in an inverted order,
i.e. those segments with the lowest MT confidence scores are at the top and those with higher
confidence scores are at the bottom.
4.2 Geometry n-gram (Geom n-gram)n-grams are often used as an information unit to measure the importance score of a sentence.
Dara et al. (2014) used an “n-gram Overlap method” that computes the unseen score of a sentence s in U by the ratio of n-grams not seen in the training data. Particularly, n-grams thatare seen fewer than V times in the training data are defined as ‘unseen’. However, the “ngram Overlap” method does not consider the information in the incremental data U . In our
experiments, we utilize the “Geometry n-gram” method in Haffari et al. (2009) to calculate thesentence score as in (2):ϕ(s) =N∑ωn ∑
P (x|U, n)
log
n
|Xs |
P (x|L, n)
n
n=1

(2)

x∈Xs

where Xsn {n = 1, . . . , N } denotes n-grams in the sentence s, and P (x|U, n) and P (x|L, n)
are the probability of x occurring in the set of n-grams in U and L, respectively, which can becomputed via maximum likelihood estimation. ωn is the weight that adjusts the importance ofthe scores of n-grams with different lengths. The weights for ωn are same as in (Haffari et al.,2009).
From the equation, we can see that “Geom n-gram” takes into account the training corpus
L and the untranslated corpus U at the same time.
4.3

Perplexity of Sentences (PPL)In NLP tasks, the perplexity (PPL) is closely related to the concept of entropy, which reflects
the degree of uncertainty of the information in a sentence: the larger the entropy, the greater the
perplexity, and the more informative the sentence. Thus, we use PPL to calculate the importancescore of a sentence s in U as in (3):log p(s)ϕ(s) = 10− N −OOV s(3)

where N is the number of words in the sentence s. In our experiments, the language model istrained by SRILM (Stolcke, 2002) using the source side of the parallel data with trigrams.
4.4

Cross Entropy Difference (CED)

This metric is proposed in Dara et al. (2014) for the sentence reranking in the incrementally
retrained PE-SMT scenario. In this scenario, given the training corpus L and an incremental

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 176

corpus U , language models (3-grams) are built from both, and each sentence s in U is scoredaccording to the entropy H difference as in (4):ϕ(s) = HU (s) − HL (s)(4)

where HU (s) is the entropy of the sentence s in U and the HL (s) is the entropy of s in L.
The higher the score given to a sentence, the more useful it is to L. That is, CED selectssentences from U that are different from L and similar to the overall corpus U.

5 Experiments
5.1

Data Settings

In order to have a full and fair study of the prioritization methods, we run our incremental
retraining experiments on two open data sets, namely the Europarl2 and DGT3 corpora. For
DGT data, we use four language pairs, namely English–German (En-De), English–Spanish(En-Es), English–French (En-Fr) and English–Polish (En-Pl), in one direction, i.e. the source
language is English. For Europarl data, we use two language pairs bidirectionally, namely
English–German and English–Spanish.
For each language pair, we extract 50k pairs of sentences as the parallel training data L
for the initial SMT systems, and 10k pairs of sentences as the incremental data U that willbe translated, (quasi-) post-edited4 and added into the parallel training data iteratively in theretraining cycle. For the Europarl data, we use Newswire 2012 set as the development set(devset) to tune the initial SMT systems. For the DGT data, we extract 2,000 pairs of sentences
as the devset to tune the initial SMT systems.
5.2 PE-SMT System Settings
The work flow of our incrementally retrained PE-SMT is shown in Figure 2.

Figure 2: The work flow of the incrementally retrained PE-SMT in our experiments
2 http://www.statmt.org/wmt15/translation-task.html
3 http://open-data.europa.eu/data/dataset/dgt-translation-memory
4 As mentioned before, we use the references of the translated segments as the PEs.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 177

Figure 2 shows a simulated post-editing workflow in which we use the references of thetranslated segments instead of post-edited MT output per se. Si indicates the batch of sentences
for translation, which is determined by the segment prioritization algorithm. M odeli represents
the initial SMT system and the incrementally updated SMT system by the newly post-edited
data. Ti indicates the MT outputs of the input segment batch Si , and T ∗i the post-edited MTtranslations that return to the SMT system for dynamic retraining. This process is repeateduntil the incremental data set or the translation job is finished. At each retraining cycle, theincremental batch contains 500 segments that are sorted and selected from the incremental data
set according to the prioritization method.
A practical incremental retrained PE-SMT system requires a quick update for its related
components at each retraining cycle, such as the translation model, language model and parameter weights. In our experiments, we use Moses to build up an incrementally retrained PE-SMT
system as:• the word alignment is performed using incremental GIZA++;5• the translation model is implemented by the dynamic phrase tables based on sampling
word-aligned bitexts (Germann, 2014);• the language mode is updated by appending the newly post-edited data to the training data;• in our experiments, the weights for the features are kept unchanged. The update of parameter weights is time-consuming and is not suitable for real-time incremental retraining.
From the viewpoint of system stability, the parameters can perform robustly in a limited range when the data changes. Experiments are conducted on DGT data sets to verify
our assumption. The results in terms of BLEU (Papineni et al., 2002) score are shown in
Table 1.
Pair
En–De
En–Es
En–Fr
En–Pl

Static (%)
32.72
47.29
44.94
36.15

Incremental-Seq. (%)
32.85
47.18
44.68
35.91

Incremental-Confidence (%)
32.88
47.14
44.76
36.04

Table 1: Robustness test of parameter weights for PE-SMT (BLEU score)
In Table 1, the numbers are BLEU scores evaluated on a constant test set (or progressset) that contains 2,000 sentence pairs. “Static” indicates that the system is built by adding all
incremental data into the initial training data, tuned on the devset and tested on the progress
set. “Incremental-Seq.” and “Incremental-Confidence” indicate that the parameter weights are
tuned by the initial training data, and kept unchanged during the whole retraining process. The
BLEU scores for these two systems are obtained at the last iteration.
We can see that the differences between the incremental systems and the static system are
not significant in terms of BLEU score, which show that for the same domain data, the weights
are robust in a limited data scale so that it is not necessary for them to be updated per iteration.
5.3 Statistics of Experimental Data
The statistics of entries in two data sets are shown in Table 2 and Table 3. It can be seen that wehave similar and consistent distributions of entries for all language pairs and the data sets.
5 http://code.google.com/p/inc-giza-pp/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 178

Pair
En-De
En-Es
En-Fr
En-Pl

Training Data
43,475/85,403
43,037/50,178
42,852/46,553
43,517/75,764Devset7,307/10,540
7,251/8,619
7,263/8,372
7,354/12,001

Incremental Data
17,992/31,131
17,724/21,360
17,737/20,094
17,963/31,587

Table 2: Entries of the DGT data sets in our experiments
Pair
En-De
En-Es

Training Data
50,893/99,206
53,765/75,117Devset
9,532/14,078
100,98/12,165

Incremental Data
22,383/35,017
21,863/29,100

Table 3: Entries of the Europarl data sets in our experiments
5.4

Prioritization Experiments

The prioritization experiments are mainly to simulate PE time by the TER score per iteration.
The test set at each retraining cycle is dynamic, and contains 500 segments selected from the incremental data according to the prioritisation criteria. The average TER score of the incremental
test sets for different language pairs and data sets are shown in Tables 4 and 5.
Pair
En–De
En–Es
En–Fr
En–Pl

Sequential
55.94
41.65
44.86
51.38

Geom n-gram
56.50
42.40
46.92
51.53

PPL
56.58
42.48
46.75
51.63

CED
55.23
41.58
44.62
51.16

Confidence
51.39
38.69
41.42
48.09

Gains
4.55
2.96
3.44
3.29

Table 4: Incremental results on DGT data set (TER Score)
Pair
De–En
En–De
Es–En
En–Es

Sequential
73.73
80.12
84.14
64.10

Geom n-gram
73.87
80.00
84.20
64.10

PPL
72.60
79.38
83.75
63.37

CED
72.48
79.02
83.25
63.11

Confidence
70.56
77.83
81.82
62.38

Gains
3.17
2.29
2.32
1.72

Table 5: Incremental results on Europarl data set (TER Score)
In Tables 4 and 5, the “Gains” are computed by the best result and the baseline (Sequential).
We can see that the best result is obtained by the “Confidence” criterion for all tasks. The
decrease in TER score for the “Confidence” criterion range from 1.72 to 4.55 absolute points(2.68∼8.13 relative points) compared to the baseline.
It can also be seen that 1) the “CED” criterion beats the baseline in all tasks, and it performs better than other prioritization methods except “Confidence”; 2) the “Geom n-gram”
method performs worst in all experiments; 3) the “PPL” method performs slightly better thanthe baseline only in the Europarl “De–En” and “En–De” tasks.
Figures 3 and 4 show the TER scores of the En–De language pair per iteration for each of
the criteria in terms of the DGT and Europarl data sets.6
From the figures we can see that there is no obvious decrease (i.e. improvement) for thebaseline in terms of TER score. However, the other four prioritization criteria have a trend of
6 The trends are similar for the other langauge pairs.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 179

Results of En−De pair on DGT data
0.8
0.7
0.6

TER

0.5
0.4
Sequential
Geom N−gram
PPL
CED
Confidence

0.3
0.2
0.1

2

4

6

8
10
12
14
Number of Iterations
Added Sentences (multiple of 500)

16

18

20

Figure 3: Results of En–De pair on DGT data
Results of En−De pair on Europarl data
1.1
Sequential
Geom N−gram
PPL
CED
Confidence

1

TER

0.9

0.8

0.7

0.6

2

4

6

8
10
12
14
Number of Iterations
Added Sentences (multiple of 500)

16

18

20

Figure 4: Results of En–De pair on Europarl data

decreasing the TER score, i.e. starting from a higher score and arriving at a lower score for thelast iterations. The decreasing trends show the effectiveness of these methods to prioritize theinput segments. As in Dara et al. (2014), the improvements over the baseline are shown after
the initial 8-9 iterations. In our scenarios, the “Confidence” results in a noticeable decrease of
the overall TER score.
In Figure 3, the “CED” and “Confidence” methods have a fluctuation at Iteration 2 andIteration 8, respectively, but the overall trend decreases in the TER score. In Figure 4, we cansee that the TER score at Iteration 1 for the “Confidence” method is over 1 which indicatesthe MT translation is quite poor and needs too many edits to transform it into a good output
sentence.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 180

6 Analysis
The “Confidence” method performs best in our incremental retraining experiments, which motivates us to investigate the hidden reasons by examining: the sentence length distribution and
OOVs as well as the correlation between them.
6.1

Sentence Length Distribution

As in Section 5.4, we take “En–De” as an illustration of the sentence length distribution shownin Figures 5 and 6. From both figures, we find that the sentence length distribution of the“Confidence” criterion strongly fluctuates per iteration, i.e. starting from very long sentence
length and arriving at very short sentence length. Furthermore, the fluctuations of the length
distribution of “Confidence” consistently correspond to the TER score trend in Figures 3 and 4,
i.e., when the length is short, the TER score is low and vice versa.
Sentence Length Dist. of En−De pair on DGT data
60
Sequential
Geom N−gram
PPL
CED
Confidence

55
50
45
Words

40
35
30
25
20
15
10

2

4

6

8
10
12
14
Number of Iterations
Added Sentences (multiple of 500)

16

18

20

Figure 5: Sentence length distribution of En–De pair on DGT data
The length distributions of the “Geom n-gram”, “PPL” and “CED” methods are moresmooth than that of “Confidence”. The “Geom n-gram” always starts from shorter sentences
and then the length increases that indicates this method prefers to select short sentences as mostinformative candidates at the beginning. The distributions of the “PPL” and “CED” methods
are quite similar as they both correlate with the entropy.
From the sentence length distributions, we hypothesize that the prioritization score ϕ(s)of “Confidence” may correlate to the sentence length of the input segment. We then calculate
correlations between the score ϕ(s) and the sentence length by the Pearson Correlation,7 andthe results for the En-De language pair are shown in Table 6.

DGT
EuroparlGeom n-gram
0.21
0.14

PPL
0.009
0.06

CED
0.02
0.10

Confidence
0.29
0.48

Table 6: Correlation between the prioritization score and the sentence length of input segments
In Table 6, we can see that the “Confidence” method is more correlated to the sentence
7 http://en.wikipedia.org/wiki/Pearsoncorrelationcoefficient

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 181

Sentence Length Dist. of En−De pair on Europarl data
40
Sequential
Geom N−gram
PPL
CED
Confidence

35

Words

30
25
20
15
10
5

2

4

6

8
10
12
14
Number of Iterations
Added Sentences (multiple of 500)

16

18

20

Figure 6: Sentence length distribution of En–De pair on Europarl data
length of the input segment than the other methods, which shows that the longer the sentenceis, the more difficult it is to be translated.
The relationship between the score ϕ(s) and sentence length poses a question: should wenormalize the score by the sentence length or not in the segment prioritization task? In order toanswer this question and verify that the unnormalized “Confidence” method is more effective
to the incremental retrained PE-SMT, we perform a further experiment using the normalized“Confidence”. The results for En–DE on DGT data between these two “Confidence” methods
are shown in Figure 7. In Figure 7, we can see that the trends of these two “Confidence” criteria
are similar, but the normalized “Confidence” curve is more smooth. The average TER score
for the normalized “Confidence” is 56.09 which is much higher (i.e. worse) than the baseline.
Based on these results, we can say that the unnormalized “Confidence” method is more effective
to reduce the PE time. Some other language pairs in our experiments have similar results.
6.2

OOVs

In SMT, it is known that OOVs are a big problem and significantly influence translation quality.
In the Moses decoding process, when an OOV occurs, the probability p(e|f ) will be significantly decreased, i.e. the confidence of the translation becomes lower. Thus, the MT output
score ϕ(s) is not only correlated with the sentence length, but is more closely correlated with
the number of OOVs in the sentence.
We then calculate the correlation between the score ϕ(s) of “Confidence” and the OOVs
by the Pearson Correlation. For En–De scenario, ρ = 0.9993 for the DGT data and ρ = 0.9997
for the Europarl data. It can be seen that as expected the more OOVs a sentence contains, thelower the confidence score, and the greater the possibility that it is ranked at the top.
6.3 Pros and Cons
The “Confidence” criterion achieved the best performance in our segment prioritization experiments for the incrementally retrained PE-SMT. However, it has some potential disadvantages
that should be considered from a practical point of view:• at each iteration, all the incremental segments need to be translated beforehand, whichmight be a problem for sentence-level incremental PE-SMT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 182

Comparison of unnomalized and normalized Confidence method of En−De pair on DGT data
0.8
0.7

TER

0.6
0.5
0.4
0.3
Sequential
Unnormalized Confidence
Normalized Confidence

0.2
0.1

2

4

6

8
10
12
14
Number of Iterations
Added Sentences (multiple of 500)

16

18

20

Figure 7: Comparison of unnormalized and normalized Confidence method of En–De pair on
DGT data• the sentence length at the first several iterations is much longer than that of the last several
iterations, which might significantly increase the amount of post-editing which in turn may
adversely affect the perception of translators/post-editors as to the utility of this approach.
However, based on the analysis in the sections above, we know that the performance of
the “Confidence” is strongly correlated with the sentence length and OOVs, so we can design
a new practical segment prioritization algorithm that only takes into account the training datarather than the translations according to these two crucial factors.

7 Conclusions and Future WorkIn this paper, we conducted an empirical study on four different segment prioritization algorithms, namely the Sequential, Geom n-gram, PPL, CED and Confidence methods for incrementally retrained PE-SMT. Experiments conducted on two data sets and several language pairs
show that the “Confidence” method achieved the best results in all tasks that reduced the TER
score of 1.72-4.55 absolute points. An investigation was carried out to examine the crucial
factors that make the “Confidence” effective. Finally, some suggestions are proposed for thedesign of new algorithms going forward.
In future work, we intend to carry out further studies on incrementally retrained PE-SMT
regarding 1) the context problem: the sorted input segments lose the sequential context that is
helpful to the translators; 2) developing a new algorithm which fully considers the influence ofsentence length and OOVs; 3) carrying out actual PE experiments using our different segment
prioritization algorithms.

Acknowledgments
We thank the reviewers for their insightful comments. We also thank our ex-colleague Dr.
Sandipan Dandapat for his preliminary work on this topic. This research is supported by Science
Foundation Ireland through the ADAPT Centre (Grant 13/RC/2106) (www.adaptcentre.ie) at
Dublin City University and Trinity College Dublin, and by Grant 610879 for the Falcon project
funded by the European Commission.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 183

References
Bertoldi, N., Simianer, P., Cettolo, M., Wschle, K., Federico, M., and Riezler, S. (2014). Online adaptation
to post-edits for phrase-based statistical machine translation. Machine Translation, 28:309–339.
Callison-Burch, C., Bannard, C., and Schroeder, J. (2005). A compact data structure for searchable translation memories. In Proceedings of the 10th Conference of the European Association for MachineTranslation (EAMT), Budapest, Hungary.
Carl, M., Dragsted, B., Elming, J., Hardt, D., and Jakobsen, A. L. (2011). The process of post-editing: A
pilot study. Copenhagen Studies in Language, 41:131–142.
Crammer, K. and Singer, Y. (2003). Ultraconservative online algorithms for multiclass problems. Journalof Machine Learning Research, 3:951–991.
Dara, A., van Genabith, J., Liu, Q., Judge, J., and Toral, A. (2014). Active Learning for Post-Editing
Based Incrementally Retrained MT. In Proceedings of the 14th conference of the European Chapter ofthe Association for Computational Linguistics, pages 185–189, Gothenburg, Sweden.
Denkowski, M., Dyer, C., and Lavie, A. (2014). Learning from Post-Editing: Online Model Adaptationfor Statistical Machine Translation. In Proceedings of the 14th conference of the European Chapter ofthe Association for Computational Linguistics, pages 395–404, Gothenburg, Sweden.
Germann, U. (2014). Dynamic phrase tables for machine translation in an interactive post-editing scenario. In Proceedings of the workshop on interactive and adaptive machine translation, pages 20–31,
Vancouver, Canada.
Gonzalez-Rubio, J., Ortiz-Martnez, D., and Casacuberta, F. (2012). Active learning for interactive machinetranslation. In Proceedings of the 13th Conference of the European Chapter of the Association for
Computational Linguistics, pages 245–254, Avignon, France.
Guerberof, A. (2009). Productivity and quality in mt post-editing. In Proceedings of MT Summit XII Workshop: Beyond Translation Memories: New Tools for Translators MT, Ottawa, Canada.
Guerberof, A. (2013). What do professional translators think about post-editing? Journal of Specialised
Translation, 19:75–95.
Haffari, G., Roy, M., and Sarkar, A. (2009). Active learning for statistical phrase-based machine translation. In Proceedings of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 415–423, Colorado, USA.
Hardt, D. and Elming, J. (2010). Incremental Re-training for Post-editing SMT. In Proceedings of theNinth Conference of the Association for Machine Translation in the Americas, Denver, USA.
Henrı́quez Q., C. A., Mariño, J. B., and Banchs, R. E. (2011). Deriving translation units using small
additional corpora. In Proceedings of the 15th International Conference of the European Association
for Machine Translation (EAMT), pages 121–128, Leuven, Belgium.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., abd Wade Shen, B. C.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: open source
toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on
Interactive Poster and Demonstration Sessions, ACL 2007, pages 177–180, Prague, Czech Republic.
Levenberg, A., Callison-Burch, C., and Osborne, M. (2010). Stream-based translation models for statistical machine translation. In Proceedings of the 2010 Annual Conference of the North American Chapterof the Association for Computational Linguistics, pages 394–402, Los Angeles, USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 184

Mathur, P., Cettolo, M., and Federico, M. (2013). Online learning approaches in computer assisted translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 301–308, Sofia,
Bulgaria.
OBrien, S. (2011). Towards predicting post-editing productivity. Machine Translation, 25(3):197–215.
Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.
Ortiz-Martinez, D., Garcia-Varea, I., and Casacuberta, F. (2010). Online learning for interactive statisticalmachine translation. In Proceedings of the 2010 Annual Conference of the North American Chapter ofthe Association for Computational Linguistics, pages 546–554, Los Angeles, USA.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation ofmachine translation. In Proceedings of the 40th annual meeting of the association for computational
linguistics, pages 311–318, Philadelphia, USA.
Plitt, M. and Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a
typical localisation context. Prague Bulletin of Mathematical Linguistics, 93:7–16.
Simard, M. and Foster, G. (2013). Pepr: Postedit propagation using phrase-based statistical machinetranslation. In Proceedings of the XIV Machine Translation Summit, pages 191–198, Nice, France.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation edit
rate with targeted human annotation. In Proceedings of the Association for Machine Translation in theAmericas, pages 223–231, Cambridge, USA.
Stolcke, A. (2002). SRILM – An extensible language modeling toolkit. In Proceedings of the 7th International Conference on Spoken Language Processing, Denver, USA.
Zhang, Y. and Vogel, S. (2005). An efficient phrase-to-phrase alignment model for arbitrarily long phrase
and large corpora. In Proceedings of the 10th Conference of the European Association for MachineTranslation (EAMT), Budapest.
Zhechev, V. (2012). Machine Translation Infrastructure and Post-editing Performance at Autodesk. In
Proceedings of AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP2012), pages
87–96, San Diego, USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 185

Effects of Word Alignment Visualization on
Post-Editing Quality & Speed†
Lane Schwartz

lanes@illinois.edu
Dept. of Linguistics, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA

Isabel Lacruz
Tatyana Bystrova

ilacruz@kent.edu
tbystro2@kent.edu
Dept. of Modern & Classical Language Studies, Kent State University, Kent, OH 44242, USA

Abstract
Phrase-based machine translation can be configured to produce alignment data that indicates
which machine translated target language words correspond to which original source language
words. In most prior work that examined the efficacy of post-editing machine translation, posteditors were presented with machine translations (and in most cases the original source language
sentences) without also being presented with source-to-target alignment links. We select four
news articles, and ask six Russian-English bilinguals and eleven Spanish-English bilinguals topost-edit English machine translation results, in some cases using alignments and in other cases
without. We obtain human adequacy judgements of the post-edited sentences, and demonstratethat when machine translation quality is low, post-editing quality is consistently higher, by astatistically significant amount, when bilingual post-editors are presented with alignment data.

1

Introduction

Post-editing is the process whereby a human user corrects the output of a machine translation
system. The use of basic post-editing tools by bilingual human translators has been shown toyield substantial increases in terms of productivity (Plitt and Masselot, 2010) as well as improvements in translation quality (Green et al., 2013) when compared to bilingual human translators
working without assistance from machine translation and post-editing tools. More sophisticatedinteractive interfaces (Langlais et al., 2000; Barrachina et al., 2009; Koehn, 2009b; Denkowski
and Lavie, 2012) may also provide benefit (Koehn, 2009a).
The question of how a post-editing interface should be configured and presented to users
is a fundamentally interdisciplinary and empirical one. Issues of user interface design, humanfactors, translation studies, and machine translation quality are all likely relevant. Phrase-based
machine translation can be configured to produce alignment data that indicates which machinetranslated target language words correspond to which original source language words. In mostprior work that examined the efficacy of post-editing machine translation, post-editors were
presented with machine translations (and in most cases the original source language sentences)
without also being presented with source-to-target alignment links.
This work begins an attempt to answer two novel questions regarding post-editing interface
design: To what extent, if at all, does the presentation of source-to-target word-level alignment
links affect the quality or speed of post-editing? Is any such effect, if it exists, dependent on
certain aspects of machine translation quality, or on the language pair?
† All code, scripts, data & analysis files for this paper are at https://github.com/dowobeha/MT_Summit_2015

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 186

(a) Russian-English adequacy evaluation guidelines

12
10
8
6
4
2

The post-edited translation is superior to the reference translation
The meaning of the Russian sentence is fully conveyed in the English translation
Most of the meaning of the Russian sentence is conveyed in the English translation
The English translation misunderstands the Russian sentence in a major way, or has
many small mistakes
Very little information from the Russian sentence is conveyed in the English translation
The English translation makes no sense at all(b) Spanish-English adequacy evaluation guidelines

10
8
6
4
2

The meaning of the Spanish sentence is fully conveyed in the English translation
Most of the meaning of the Spanish sentence is conveyed in the English translation
The English translation misunderstands the Spanish sentence in a major way, or has
many small mistakes
Very little information from the Spanish sentence is conveyed in the English translation
The English translation makes no sense at all

Table 1: Adequacy evaluation guidelines for bilingual Russian-English human judges (Schwartz
et al., 2014), and for bilingual Spanish-English human judges (Albrecht et al., 2009). Because
no reference translation was available for Spanish-English, the 12 category is omitted.
To address these questions, we conduct a bilingual post-editing experiment (§2) wherebilingual post-editors are presented with machine translation output of varying quality, withand without word-level alignment link visualization. In the first condition, we ask six RussianEnglish bilingual translation students to post-edit two Russian language news articles startingwith relatively low quality English machine translation. In the second condition, we ask eleven
Spanish-English bilingual translation students to post-edit two Spanish language news articles
starting with relatively high quality English machine translation. We find (§3) that when machine translation quality is low, post-editing quality is consistently higher, by a statisticallysignificant amount, when bilingual post-editors are presented with alignment data. We find nostatistically significant effect when machine translation quality is high. We also found that for
both Russian-English and Spanish-English the mean post-editing times were shorter for texts
with alignment than for texts without alignment. These differences were not significant, but thedifference for the Russian-English texts approached significance. Finally, in §4 we briefly survey the current state of post-editing research and situate this work within the context of related
work in post-editing.

2

Methodology

We hypothesize that the presentation of word-level alignment links to human post-editors may
affect the quality or speed of the resulting output, and that such effects may be dependent on
the quality of the underlying machine translations. To test this hypothesis, we conduct a bilingual post-editing experiment where bilingual post-editors are presented with machine translation
output of varying quality, with and without word-level alignment link visualization.
2.1 Bilingual Participants
2.1.1 Russian-English Bilingual Participants
There were six participants who served as post-editors in the Russian-English portion of thisstudy, all of whom were paid for their time at the rate of $25 for each hour or part of an hour.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 187

These participants were all English-Russian bilinguals. We designate these participants as PE1–
PE6. Four of the six bilingual participants (PE2, PE3, PE4, & PE6) had Russian as their first
language (L1) and were highly proficient in English as their second language (L2). The othertwo bilingual participants (PE1 & PE5) had English as their first language and were highly proficient in Russian as their second language. Three of the six bilingual participants were graduate
students and three were undergraduate students; all were enrolled in a university Russian Translation program.
2.1.2

Spanish-English Bilingual Participants

There were eleven participants who served as post-editors in the Spanish-English portion of thisstudy, all of whom were paid for their time at the rate of $25 for each hour or part of an hour.
They were all Spanish-English bilinguals. We designate these participants as PE7–PE17. The
first language (L1) of all eleven participants was English, and all eleven were highly proficient in
Spanish as their second language (L2). These participants were students in a university Masterof Spanish Translation program.
2.2
2.2.1

Source Language Data
Russian Data

For the Russian-English portion of this study, we selected as source texts a subset of the texts
from the 2014 Workshop on Statistical Machine Translation (WMT-14) shared translation task
(Bojar et al., 2014). Source texts were news articles covering world news events in late 2013.
The first text was originally a Russian-language BBC news article covering Syrian chemical
weapons. The second text was originally an English-language news article covering U.S. spying
policy. We designate the former as Doc A and the latter as Doc B.
These two texts were each divided into segments (32 and 33 segments, respectively) that
corresponded to sentences or stand-alone phrases (typically corresponding to news headlines,
captions, or cutlines). Segments in Doc A varied in length from 3 to 35 words (mean length 17
words); segments in Doc B varied in length from 9 to 55 words (mean length 23 words).
Professional translations of Doc A into English and Doc B into Russian were commissioned as part of the WMT-14 shared translation task (Bojar et al., 2014). The Russian versionof each text was translated automatically using Moses (Koehn et al., 2007) by Schwartz et al.
(2014) as part of their WMT14 shared task submission. As a side effect of the phrase-based MTprocess, Moses can be configured to produce alignment links, indicating which target language
words were produced from which source language words. To enable maximal comparability
with the post-editing results of Schwartz et al. (2014), we make use of Russian-English machinetranslation results and alignments from that work here.
2.2.2

Spanish Data

Two Spanish source texts were selected. Both were extracts from a news article from a Spanish
newspaper covering current world news events. The two texts were divided up into segmentsthat corresponded to sentences or stand-alone phrases. The first text had 26 segments that variedin length from 4 to 24 words (mean length 15 words) and the second text had 25 segments thatvaried in length from 4 to 28 words (mean length 16 words). We designate the former as Doc C
and the latter as Doc D. No reference translations exist for either Spanish text.
The Spanish source texts were translated automatically using Microsoft Bing Translator
through its online developer API. Bing Translator, when accessed via the developer API, can be
configured to return character-level alignment links from source characters to target characters,in addition to translated target language sentences. Our scripts derive word alignments from thecharacter alignments returned by the Bing Translator API.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 188

Figure 1: Post-editing interface, with alignment links displayed. Sentence shown is from
Spanish-English Doc C.

2.3

Translation Quality

We hypothesize that any effects of word alignment visualization on post-editing may be dependent on the quality of the underlying machine translations displayed to the post-editors. Because
we care about the adequacy of post-edited translations, we consider actual human judgements tobe preferable to automated metrics such as BLEU (Papineni et al., 2002), which at best serve asa flawed proxy for human judgements. Instead, following Albrecht et al. (2009) and Schwartz
et al. (2014), we therefore obtained human judgements of translation adequacy for the RussianEnglish and Spanish-English machine translations used in this study.
The Russian language news articles used in this study have corresponding reference translations. It is therefore possible (although given current machine translation quality, highly unlikely) that machine translation quality for any given segment could conceivably surpass thequality of the corresponding reference translation (if for example, the reference translator makes
a mistake). For assessing the quality of the Russian-English machine translations, then, we follow the 12-point adequacy scale of Schwartz et al. (2014). This adequacy scale is shown in
Table 1a on page 2; this scale ranges from a low of 2 (the English translation makes no sense atall) to a high of 12 (the translation is superior to the reference).
The Spanish language news articles used in this study lack corresponding reference translations. Thus, unlike the case of our Russian data, no matter how high the quality of machinetranslations, no Spanish-English machine translation segment could possibly receive a score of
12. For Spanish-English, we therefore follow the 10-point adequacy scale of Albrecht et al.
(2009). This adequacy scale is shown in Table 1b on page 2; this scale is very similar to theformer, but has a high of 10 (the meaning of the source sentence is fully conveyed in the English
translation) instead of 12.
2.4

Post-Editing Interface

For this study, we developed a novel post-editing interface, based on the open source software
used and released by Schwartz et al. (2014). Our software is written using Scala (Odersky,2014), and is released as open source (see the software supplement that accompanies this work).
This code constitutes a ground-up rewrite of the Java-based post-editing interface of Schwartz
et al. (2014), written using a strict model-view-controller software design pattern to be easy forother researchers to use and extend.
Our post-editing interface can be seen in Figure 1 above. Each text was presented to posteditors in one of two variant modalities — word-level alignment links could either be visualized
or left absent. In both variants, each source language segment was presented along with thecorresponding machine translated English segment; a text field (initially populated with themachine translated segment) where the post-editor could make changes was also presented. In
the first variant, the word-level alignment links produced by the machine translation decoder(Moses for Russian-English, Bing Translator for Spanish-English) were graphically displayed,
linking source words to their corresponding machine translated target words. In the second
variant, the word-level alignment links were omitted from the visualization interface.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 189

Monolingual PE with Alignments
Bilingual PE with Alignments
Bilingual PE no Alignments
Machine Translation
50%
40%
30%
20%
10%
0%
2

4

6

8

10

12

(a) Russian-English

Bilingual PE with Alignments
Bilingual PE no Alignments
Machine Translation
50%
40%
30%
20%
10%
0%
2

4

6

8

10

(b) Spanish-English

Figure 2: Percentage of segments judged to be in each adequacy category. For each language
pair, we report percentages for raw (unedited) machine translation output, as well as output postedited by a bilingual post-editor with access to alignments and without access to alignments. For
Russian-English, we additionally report percentages for output post-edited by a monolingual
post-editor (Schwartz et al., 2014) with access to alignments.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 190

2.5

Procedure

Participants performed the test individually in an office setting and were instructed to minimally
post-edit. Specifically, participants were asked to disregard issues of style and to focus on a) how
well the translation conveyed the meaning of the source text, and b) the grammatical correctnessof the translated segments. Participants sat in front of a computer that displayed the source texts
divided up into segments (see Figure 1 on page 4). Directly below each source text segment, its
machine translation was displayed. Below that was an active response area, where participants
were asked to carry out the post-editing.
During initial data collection (the Russian-English condition), the only data collected was
the final post-edited output and the overall time taken per text. Subsequently, we enhanced thepost-editing software with additional logging functionality, enabling the software to record keylogging and mouse-logging data. For the subsequent Spanish-English condition, this enhanced
software was utilized; for this condition millisecond-precision keyboard-event and mouse-event
logs were recorded in addition to collecting final post-edited output and overall time taken per
text.
We believe that scientific inquiry is at its strongest when experiments can be easily replicated, and when the raw and processed data from such experiments can be directly verified byreviewers, readers, and other experimenters. In that spirit, all data and code produced or usedin this work are provided in the attached dataset and software supplements. This includes alllogs, along with raw machine translation output, alignment data, post-edited output, adequacy
judgements, post-editing software, and supplementary scripts.
2.5.1 Russian-English Participant Assignment
Each participant was instructed to edit one of the two texts using the interface where alignment
links were shown, and to edit the other text using the interface where alignment links were omitted. Participants post-edited the two texts in one session lasting less than two hours, although
there were no time limits set for the task. The experimenter was present in the room and manually recorded the times taken. Post-Editors 2, 4, and 6 were assigned to post-edit Doc A using
the variant 1 interface that displayed alignments, and Doc B using the variant 2 interface thatomitted alignments. Post-Editors 1, 3, and 5 were assigned Doc A using variant 2 and Doc B
using variant 1.
2.5.2 Spanish-English Participant Assignment
Participants post-edited one text using the interface where alignment links were shown and theother text using the interface where alignment links were omitted. Participants post-edited thetwo texts in one session lasting less than one hour. Timings were recorded by a keylogger. Posteditors 7, 9, 11, 13, 15, and 17 edited Doc C using the interface that omitted the alignments
and Doc D using the interface that displayed the alignments. Post-editors 8, 10, 12, 14, and 16
edited Doc C using the interface that displayed the alignments and Doc D using the interface
that omitted the alignments.

3

Results

3.1

Rating Translation Adequacy

Following the methodology outlined in §2.3, all post-edited output as well as all machine translations were evaluated by bilingual human judges using the adequacy scales shown in Table 1.
3.1.1 Rating Adequacy of Russian-English
Following the adequacy guidelines from §2.3, an experienced English-Russian translator andgrader rated all English output translations of the Russian-English post-edited segments. In
addition, all English machine translations of the Russian documents were manually rated forProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 191

(a) Russian-English

Monolingual PE with Alignments
Bilingual PE with Alignments
Bilingual PE no Alignments

Mean Adequacy Score

10
9
8
7
6
2

4
6
8
Adequacy of unedited MT

10(b) Spanish-English

Bilingual PE with Alignments
Bilingual PE no Alignments
10
9
8
7
6

2

4
6
8
Adequacy of unedited MT

10

Figure 3: Mean adequacy score, categorized by the adequacy score of the unedited MT. The red
horizontal line indicates the mean adequacy score (Russian-English: 6.1; Spanish-English: 7.1)of the unedited MT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 192

adequacy.
For each segment, the human rater was presented with a vertically-arranged list showing
all variants of that segment. The first entry in each list was the segment in the source language(Russian). The source segment was followed by the reference translation in English. The subsequent eight entries were English translations of the source segment, presented in a randomizedorder. The English translations included the unedited machine translation output, as produced
by Moses, a post-edited translation produced by a monolingual post-editor from Schwartz et al.
(2014), and the six post-edited translations produced by the Russian-English bilingual posteditors in this study.
All Russian-English translations were rated using the translation adequacy scale in Table 1a
on page 2, with possible ratings ranging from 2 (translation makes no sense) to 12 (translation
is superior to the reference translation).
3.1.2

Rating Adequacy of Spanish-English

Following the adequacy guidelines from §2.3, an experienced Spanish-English translator andgrader worked in cooperation with a second Spanish-English bilingual to rate all English output translations of the Spanish-English post-edited segments. In addition, all English machinetranslations of the Spanish documents were manually rated for adequacy. For each segment, thehuman raters were presented with a vertically-arranged list showing all variants of that segment.
The first entry in each list was the segment in the source language (Spanish). The subsequenttwelve entries were English translations of the source segment, presented in a randomized order.
The English translations included the unedited machine translation output, as produced by Bing
Translator, and the eleven post-edited translations produced by the Spanish-English bilingual
post-editors in this study.
Unlike the Russian documents, no reference translation was available for the Spanish documents; for this reason (as described in §2.3), the top category (12) used in evaluating RussianEnglish segments was omitted from the Spanish-English evaluation. All Spanish-English translations were rated using the translation adequacy scale in Table 1b on page 2, with possible
ratings ranging from 2 (translation makes no sense) to 10 (the meaning of the Spanish sentence
is fully conveyed in the English translation). Unlike all other participants, participant PE17
consistently produced post-edited segments of lower adequacy than the corresponding raw MT
output. This participant was therefore dropped from all analyses of the Spanish-English data.
3.2

Adequacy Results

Figure 2 on page 5 presents the percentage of segments judged to be in each adequacy category.
Mean adequacy scores for each experimental condition are presented in Figure 3 on the previous
page. By subtracting the adequacy score of each machine translated segment, we obtain theadequacy gain obtained by post-editing; these values are presented in Figure 4 on the following
page. Finally, Figure 5 on page 11 presents mean adequacy score by post-editor. We now analyze
these results by experimental condition.
3.2.1

Machine Translation Adequacy

In Figure 2 on page 5, we observe that the Russian-English machine translation segments tend
to be of lower quality (as measured by adequacy), while the Spanish-English machine translation segments tend to be of higher quality. Two-thirds of Russian-English machine translation
segments are judged to have major errors (ratings 2-6), while one-third are rated as mostly orcompletely correct (8-12). Contrast this with the Spanish-English machine translations segments; a minority (about two-fifths) are judged to have major errors (ratings 2-6), while themajority (about three-fifths) are rated as mostly or completely correct (8-10).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 193

(a) Russian-English

Mean Gain in Adequacy Score

Monolingual PE with Alignments
Bilingual PE with Alignments
Bilingual PE no Alignments

8
6
4
2
0

2

4
6
8
Adequacy of unedited MT

10(b) Spanish-English

Bilingual PE with Alignments
Bilingual PE no Alignments

8
6
4
2
0
−2
2

4
6
8
Adequacy of unedited MT

10

Figure 4: Mean gain in adequacy score over unedited MT, categorized by the adequacy score of
the unedited MT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 194

3.2.2 Russian-English Adequacy
The mean adequacy score when bilingual participants were presented with alignments was 8.35.
When alignments were omitted from the post-editing tool, the mean adequacy score was 7.85.
A Wilcoxon signed-rank test (Wilcoxon, 1945) showed that when participants were presented
with alignments the ratings of their translations were significantly higher than when participants
post-edited without access to alignments (N = 6, Z = -2.207, p = 0.027).
3.2.3 Spanish-English Adequacy
The mean adequacy score when Spanish bilingual participants were presented with alignments
was 8.22. When alignments were omitted from the post-editing tool, the mean adequacy score
was 8.02. These means are not significantly different.
3.3

Timing Results

3.3.1 Russian-English Timing
The times taken by the Russian-English post-editors to post-edit each text were recorded manually to the nearest minute. Participants post-edited each text without a break. They took a breakof approximately five minutes between texts. The mean times were 33 minutes for texts withalignment and 40 minutes for texts without alignment. This difference approached significance(p = .082).
3.3.2 Spanish-English Timing
The times taken by the Spanish-English post-editors to post-edit each text were recorded by the
keylogger to nearest millisecond. Participants post-edited each text without a break. They tooka break of approximately five minutes between texts. The mean times were 21 minutes and5 seconds for texts with alignment and 22 minutes and 5 seconds for texts without alignment.
An independent samples t-test showed that these times are not significantly different from eachother (t(18) = .295, p = .77.) Participants were not given time limitations, so timing data must
be interpreted with caution. Note, however, that the mean time with alignment is numericallyshorter than the mean time without alignment. The shorter editing times for Spanish-English
may in part be explained by the shorter length of these documents.

4

Analysis and Related Work

Our results suggest that when machine translation quality is poor (2–4), bilingual post-editors
may produce higher quality translations when presented with bilingual alignment links between
source words and machine-translated target words. Alternatively, when machine translation
quality is high (8–12), no effect is seen by presenting alignment visualizations. We explain thisby hypothesizing that word alignment visualization may enable post-editors to better recoverfrom certain types of translation errors produced by MT systems; when MT quality is highenough that such errors are absent, word alignment visualization may no longer play a restorativerole.
We examine the effect that alignment link visualization has on each bilingual post-editor in
Figure 5 on the next page. In the Russian-English condition, where overall MT quality is poor,
we observe that post-editing quality varies widely between post-editors (with PE2 and PE3 performing best). For all six bilingual post-editors, we observe higher mean adequacy scores whenalignment links were presented than when they were omitted from the post-editing tool. We
also note that when alignment links were absent, one bilingual post-editor (PE5) performed
worse than the monolingual post-editor (PE0) from Schwartz et al. (2014). On the other hand,in the Spanish-English condition, where overall MT quality is good, we observe relatively little
variation in quality between the ten post-editors. When compared to the unedited machine trans-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 195

Mean Adequacy Score by ParticipantMonolingual PE with Alignments
PE with Alignments
PE no Alignments
10
9
8
7
6
PE0 PE1 PE2 PE3 PE4 PE5 PE6
Post-Editor(a) Russian-English

PE with Alignments
PE no Alignments

10
9
8
7
6
PE7

PE8

PE9

PE10 PE11 PE12 PE13 PE14 PE15 PE16
Post-Editor(b) Spanish-English

Figure 5: Mean adequacy score per post-editor. The red horizontal line indicates the meanadequacy score (Russian-English: 6.1; Spanish-English: 7.1) of the unedited MT.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 196

GER

PE with Alignments
PE no Alignments
2.53
1.47

Figure 6: Mean adequacy gain to effort ratio (GER) values for segments post-edited with alignment and without alignment. Effort is measured by pause to word ratio (PWR).

lations, post-editing resulted in improved mean adequacy for all post-editors, both bilingual andmonolingual.
Our results also suggest that post-editing time tends to be reduced for texts with alignment.
These numerical reductions were consistent across all the Russian-English participants with the
exception of PE3, but they were not consistent for the Spanish-English participants.
We hypothesize that texts with alignment are less cognitively demanding to process, andso less effortful to post-edit than texts without alignment. If this is the case, shorter post-editingtimes for texts with alignment are consistent with previous findings by Koponen et al. (2012),
who found that per word post-editing times were shorter for segments that were less cognitively
demanding because of the linguistic structure. Related work on cognitive effort in post-editing(Lacruz et al., 2014; Lacruz and Shreve, 2014) has also shown decreased densities of short pauses
when less cognitively demanding segments are post-edited.
The keystroke logging data gathered for Spanish-English post-editors allowed the computation of Pause to Word Ratio (PWR). For each segment, PWR is the ratio of the number of pauses
exceeding 300ms to the number of words in the MT segment; it is a measure of cognitive effort
in post-editing (Lacruz and Shreve, 2014). Higher PWR corresponds to higher cognitive effort.
Contrary to expectation, the mean PWR for Spanish-English post-editors was slightly higher
for the segments with alignment (0.70) than for those without alignment (0.63). However, thenumerical difference was not significant.
It is possible that the effect of alignment on PWR was masked by the fact that the adequacyof the Spanish-English MT segments was generally high. Since our prediction is that alignment
should both increase post-editing adequacy and reduce post-editing effort, the Gain to Effort Ratio, GER = (PE Rating – MT Rating)/PWR is a promising metric to investigate. We hypothesizethat GER is higher for segments with alignment than for segments without alignment.
Figure 6 above shows GER for Spanish-English. GER for segments with alignment was
2.53, and GER for segments without alignment was 1.47. Our prediction was confirmed: apaired samples t-test showed that GER was higher for segments with alignment (t(9) = 2.49,
p =.034.). This result suggests that GER may be a robust metric for measuring the effects ofalignment on post-editing. It would be interesting to conduct further studies involving language
pairs different from Spanish-English where the adequacy of machine translations may be lower.

5

Conclusion

In this work, we observe that when machine translation quality is poor, bilingual post-editors
may produce higher quality translations when presented with bilingual alignment links between
source words and machine-translated target words. We explain this by hypothesizing that wordalignment visualization may enable post-editors to better recover from certain types of translation errors produced by MT systems; when MT quality is high enough that such errors areabsent, word alignment visualization may no longer play a restorative role. The timing results
we observe, while not statistically significant, appear to be consistent with prior work that found
per word post-editing times to be shorter for segments that were less cognitively demanding.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 197

References
Albrecht, J. S., Hwa, R., and Marai, G. E. (2009). Correcting automatic translations throughcollaborations between MT and monolingual target-language users. In Proceedings of the
12th Conference of the European Chapter of the Association for Computational Linguistics,
pages 60–68, Athens, Greece.
Barrachina, S., Bender, O., Casacuberta, F., Civera, J., Cubel, E., Khadivi, S., Lagarda, A., Ney,
H., Tomás, J., Vidal, E., and Vilar, J.-M. (2009). Statistical approaches to computer-assistedtranslation. Computational Linguistics, 35(1):3–28.
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P.,
Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. (2014). Findings of the2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on
Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA.
Denkowski, M. and Lavie, A. (2012). TransCenter: Web-based translation research suite. In
Proceedings of the AMTA 2012 Workshop on Post-Editing Technology and Practice Demo
Session.
Green, S., Heer, J., and Manning, C. D. (2013). The efficacy of human post-editing for languagetranslation. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing
Systems (CHI ’13), pages 439–448, Paris, France.
Koehn, P. (2009a). A process study of computer aided translation. Machine Translation,
23(4):241–263.
Koehn, P. (2009b). A web-based interactive computer aided translation tool. In Proceedings of
the ACL-IJCNLP 2009 Software Demonstrations, pages 17–20, Suntec, Singapore.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007).
Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th
Annual Meeting of the Association for Computational Linguistics (ACL ’07) Demo and Poster
Sessions, pages 177–180, Prague, Czech Republic.
Koponen, M., Aziz, W., Ramos, L., and Specia, L. (2012). Post-editing time as a measure ofcognitive effort. In AMTA 2012 Workshop on Post-Editing Technology and Practice, WPTP,
pages 11–20, San Diego, USA.
Lacruz, I., Denkowski, M., and Lavie, A. (2014). Cognitive demand and cognitive effort inpost-editing. In Proceedings of the AMTA 2014 Workshop on Post-Editing Technology andPractice, pages 73–84.
Lacruz, I. and Shreve, G. (2014). Pauses and cognitive effort in post-editing. In O’Brien,S., Balling, L. W., Carl, M., Simard, M., and Specia, L., editors, Post-Editing: Processes,
technology and applications, pages 246–272.
Langlais, P., Foster, G., and Lapalme, G. (2000). TransType: A computer-aided translation
typing system. In Proceedings of the ANLP/NAACL 2000 Workshop on Embedded MachineTranslation Systems, pages 46–51, Seattle, Washington.
Odersky, M. (2014). The Scala language specification, version 2.9. Technical report, EPFL
Programming Methods Laboratory, Lausanne, Switzerland.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 198

Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association
for Computational Linguistics (ACL ’02), pages 311–318, Philadelphia, Pennsylvania.
Plitt, M. and Masselot, F. (2010). A productivity test of statistical machine translation postediting in a typical localisation context. The Prague Bulletin of Mathematical Linguistics,
93:7–16.
Schwartz, L., Anderson, T., Gwinnup, J., and Young, K. M. (2014). Machine translation andmonolingual postediting: The AFRL WMT-14 system. In Proceedings of the Ninth Workshop
on Statistical Machine Translation, pages 186–194, Baltimore, Maryland. Association for
Computational Linguistics.
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–
83.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 199

A Systematic Evaluation of MBOT
in Statistical Machine Translation
Nina SeemannFabienne Braune
Andreas Malettiseemanna@ims.uni-stuttgart.debraunefe@ims.uni-stuttgart.demaletti@ims.uni-stuttgart.deInstitute for Natural Language Processing, University of Stuttgart,
Pfaffenwaldring 5b, 70569 Stuttgart, Germany

Abstract
Shallow local multi-bottom up tree transducers (MBOTs) have been successfully used as translation models in several settings because of their ability to model discontinuities. In this contribution, several additional settings are explored and evaluated. The first rule extractions fortree-to-tree MBOT with non-minimal rules and for string-to-string MBOT are developed. Allexisting MBOT systems are systematically evaluated and compared to corresponding baseline systems in three large-scale translation tasks: English-to-German, English-to-Chinese, andEnglish-to-Arabic. Particular emphasis is placed on the use of discontinuous rules. The developed rule extractions and analysis tools will be made publicly available.

1

Introduction

The area of statistical machine translation (SMT) (Koehn, 2009) concerns itself mostly with thedevelopment of automatic methods of deriving probabilistic translation models from a parallel
corpus. Such a corpus contains corresponding sentences in two languages. The rule extraction mechanism automatically extracts rules for the translation model from such a corpus andassigns them a probability. Several different translation approaches are currently discussed in
SMT. Among the best-performing systems one often finds phrase-based (Koehn et al., 2003)
and syntax-based translation systems. Phrase-based systems use no linguistic information andcan be obtained directly from the corpus. The same observation is true for hierarchical phrasebased systems (Chiang, 2005) as those are syntactic in a formal sense only. Syntax-based
systems require some form of syntactic annotation, which is often represented as a (parse)
tree. Correspondingly, we obtain several models, namely tree-to-tree, string-to-tree, and treeto-string systems, which use trees on the source or the target language side. A multitude of formalisms has been proposed as syntax-based translation models. The most prominent might be
the synchronous tree substitution grammars (STSGs) of Eisner (2003) and the non-contiguoussynchronous tree sequence substituition grammars (STSSGs) of Sun et al. (2009). Recently,
Maletti (2011) proposed local multi bottom-up tree transducers (MBOT) as a translation model
for syntax-based SMT. An MBOT is an extension of an STSG that allows sequences of tree
fragments on the target side of its rules. In this manner, it can model discontinuities. It canalso be understood as a restricted form of a STSSG, in which the rules consist of sequences ofsource and target tree fragments.
Recently, MBOTs have been implemented as a translation model inside the Moses framework (Koehn et al., 2007) by Braune et al. (2013). Initially, only the rule extraction for minimal

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 200

tree-to-tree rules of Maletti (2011) was available. In combination with synchronous context-freegrammar (SCFG) rules, already this system led to significant improvements over a corresponding system using SCFG rules only. Later, Seemann et al. (2015) proposed a rule extraction
procedure for string-to-tree MBOTs that is able to extract non-minimal rules. Since the numberof such rules usually explodes, certain (parametric) restrictions are imposed on the extracted
rules in the same spirit as in (Chiang, 2005). Also in this setting, significant improvements overa corresponding SCFG-based system are obtained. However, systems imposing even further
syntactic restriction still outperform those systems in some cases. To understand the benefits of
the MBOT model and its dependence on syntax, we develop additional rule extractions for the
missing scenarios and provide a systematic evaluation of MBOT-based systems. In particular,
we develop a rule extraction for non-minimal tree-to-tree rules. It is known that minimal rules
are rather restrictive (Galley et al., 2004), so we expect sizable improvements from the obtained
tree-to-tree system (when compared to the tree-to-tree system using only minimal rules). In addition, we explore whether the discontinuous rules of MBOTs remain useful without any syntax.
Consequently, we also develop a string-to-string rule extraction for MBOT. All the mentioned
MBOT models are systematically evaluated and compared to popular translation models.
We evaluate our models on three large-scale translation tasks: English-to-German,
English-to-Arabic, and English-to-Chinese. We demonstrate that the expected improvementsfrom the non-minimal rules is indeed realized. However, the tree-to-tree SCFG baseline system, which also utilizes non-minimal rules, still outperforms the MBOT model. In the string-totree setting, MBOT significantly outperform SCFG as demonstrated by Seemann et al. (2015).
Finally, in the string-to-string setting, discontinuous rules seem to be hardly useful at all. The
obtained evaluation scores for such string-to-string (hierarchical) systems are comparable. Onlya detailed analysis of the number of the used rules that are (potentially) discontinuous indeed
confirms that hardly any discontinuous rules are used when decoding the test set. Overall, theseresults seem to suggest that discontinuous rules are most successful in the string-to-tree setting,where the strict syntactic structure of the output tree makes discontinuities valuable, while theflat structure of the input (string) allows sufficient freedom. Chiang (2010) arrives at a similar
conclusion for general syntax-based systems with the argument that the target-side syntax mightenable more grammatical translations. String-to-tree MBOTs offer an even better performance
than string-to-tree SCFGs in our translation tasks, so discontinuities seem to be very relevant.

2

Statistical Machine Translation with MBOTs

Syntax-based statistical machine translation models use the syntactic structure of the input
and/or output sentences during training and decoding. The syntactic structure of the sentences
is typically automatically obtained with the help of a (constituent) parser, so we use ‘parse’ torefer to the syntactic structure. Several different settings can be distinguished depending on theuse of parses:• tree-to-tree: In this setting, parses are used for both the source and target sentences during
training and for the input sentences during decoding.
• tree-to-string: Parses are used only for the source language; i.e., for the source sentences
during training and for the input sentences during decoding.
• string-to-tree: In this setup, parses are only used for the target language sentences.
• string-to-string: No parses are used and the system is not syntax-based.
Shallow local multi bottom-up tree tranducers (MBOTs) have already been successfully applied
as translation model in the tree-to-tree setting (Braune et al., 2013) as well as in the string-to-tree
setting (Seemann et al., 2015). It is generally accepted that tree-to-string systems yield worse
performance than string-to-tree systems (Chiang, 2010). We complete the picture by adding
non-minimal rules to the tree-to-tree system, establishing a string-to-string variant (i.e., similar

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 201

S

S

NP
JJ
Official

VP
NNS

VBD

forecasts

predicted

nur

NP
JJ

NP
QP

NN

RB

CD

just

3

Official

VP
NNS

VBD

forecasts

predicted

%

3

nur

von ADV CARD %Offizielle Prognosen APPR

AP

ADJA

PP

NN
NP

sind
VAFIN

NP
QP

NN

RB

CD

just

3

%

3

von ADV CARD %NN ausgegangen
VVPP
VP

Offizielle Prognosen APPR

AP

ADJA

PP

NN
NP

S

sind
VAFIN

NN ausgegangen
VVPP
VP

S

Figure 1: Word-aligned, bi-parsed sentence pair before (left) and after (right) excision of a rule.
to a hierarchical phrase-based model) that requires no parses at all, and providing an extensive
evaluation of all those variants.
Let us start by illustrating the different variants. To this end, we first show the generalrule shape of MBOTs, and then recall the existing rule extraction algorithms together with theparticularities of the obtained rules. Our presentation is necessarily illustrative only. For theformal details we refer the reader to (Maletti, 2011; Braune et al., 2013; Seemann et al., 2015).
Roughly speaking, each MBOT has terminal symbols (e.g., lexical items) and nonterminal
symbols (e.g., part-of-speech tags and syntactic categories). In essence, an MBOT is simply afinite set of rules. Each rule ` → r consists of a left-hand side ` and a right-hand side r. The
left-hand side ` = t consists of an object t (string or tree) for the source side, and the right-handside r = (t1 , . . . , tm ) similarly consists of a sequence t1 , . . . , tm of objects for the target side.
The objects t, t1 , . . . , tm are either strings or trees (with the restriction that t1 , . . . , tm have thesame type) formed from the terminal and nonterminal symbols with the additional restrictionthat each exposed occurrence of a nonterminal in an object in the right-hand side is linked
to an exposed occurrence of a nonterminal in the left-hand side. More precisely, in a string,
the lexical items are the terminal symbols and each nonterminal occurrence is exposed. In atree, the lexical items only occur as leaves and additionally nonterminals are allowed as leaves.
However, an occurrence of a nonterminal is exposed if and only if it is a leaf. The 4 different
settings naturally correspond to the choice of strings or trees for the source and target side
objects t and t1 , . . . , tm , respectively.
2.1

Minimal tree-to-tree rule extraction

Let us start with the tree-to-tree setting, for which a rule extraction for minimal rules1 was
proposed in (Maletti, 2011). In this case, the rule extraction requires a word-aligned, bi-parsed(parsed on both sides) parallel corpus. A sample entry of such a corpus is shown left in Figure 1.
The rule extraction is applied to each sentence pair of the corpus and essentially performs thefollowing steps:(1) Select a minimal number of alignment edges E such that• the maximal (non-leaf nonterminal) source node v containing (as leaves) all sourcesof the selected edges and no sources of non-selected edges exists and
1 A rule is minimal if it cannot be obtained by means of substitution from others.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 202

JJ→

 ADJA 

NNSOffizielle

forecasts

Official
CD→ CARD 

NN

3

%

3

→
NN

NN





JJ

%
VP

NN

AP

VBD

von

 VAFIN→predicted

NP NN 


APPR

VBD



Prognosen

PP

NP
QP

→

→

→

NP



ADJA

NNS

→



NP

QP
→
RB

→

NP

nur

AP



ADV

→
VP

 ADV 

just


CD
S



RB





VVPP

PP

VVPP
ausgegangen

NN

VP

VAFIN ,

,

sind

CARD
S


NP

VAFIN


VP

Figure 2: All minimal MBOT rules extractable from Figure 1.
• the maximal (non-leaf nonterminal) target nodes w1 , . . . , wm containing all targets ofthe selected edges and no targets of non-selected edges exists.
In other words, the selected set E of edges admits maximal subtrees (below nodesv and w1 , . . . , wm ) that are consistent with the word alignment (Chiang, 2005).
(2) Excise the subtree t below v for the source side and the subtrees t1 , . . . , tm below
w1 , . . . , wm , respectively, for the target side. In this way, we obtain the tree-to-tree rulet → (t1 , . . . , tm ). After the excision, the nonterminals at v and w1 , . . . , wm remain as leaf
nonterminals and are linked. The result of excising the rule containing ‘predicted’ from theleft entry of Figure 1 is shown right in Figure 1.
(3) Repeat the process with the linked pair of trees obtained after the excision.
Figure 2 shows all minimal MBOT rules that can be extracted from the word-aligned, bi-parsed
sentence pair of Figure 1. The obtained rules are made shallow (Braune et al., 2013) by removing the internal (i.e., non-root and non-leaf) nodes. Figure 3 shows this process on the only ruleof Figure 2 that is not yet shallow.
2.2

Non-minimal string-to-tree rule extraction

The parametrized non-minimal rule extraction of Seemann et al. (2015) for string-to-tree MBOT
rules is an extension of the rule extraction of Chiang (2005) for hierarchical rules. As expectedfrom the string-to-tree setting, it extracts rules from a word-aligned parallel corpus with parses
for the target language side. Figure 4 shows an entry in such a corpus. Recall that a stringto-tree rule has the shape w → (t1 , . . . , tm ) for a string w and trees t1 , . . . , tm . In the source
string w, we only allow lexical items and the nonterminal X. The rule extraction proceeds
similarly as described in Section 2.1 with the exception that a phrase (or span) is selected in thesource side (instead of a node of the tree) and that the minimality and maximality conditions
are dropped (non-minimality). When excising from the source side, we leave the nonterminal X
instead of the excised material. In essence, we obtain all string-to-tree rules that are consistently
word-aligned in this manner. However, there are way too many such rules, and Seemann et al.
(2015) establish the following conditions on w that need to be fulfilled for a rule to be extracted.
• It should have (i) a lexical item that is the source of a word alignment or (ii) an occurrenceof X (i.e., selecting the empty set E of edges is excluded).
• It should correspond to a span of length at most 10 and contain at most 5 occurrences of
PP

NP
→
QP

NN




APPR

AP

NN

von

NP
→
QP

NN

PP


von

AP


NN

Figure 3: Non-shallow MBOT rule (left) and its shallow counterpart (right).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 203

Official forecasts predicted just 3 %nurOfficial forecasts

3

AP

ADJA

PP

NN
NP

sind

just 3 %nur

von ADV CARD %Offizielle Prognosen APPRX

von ADV CARD %NN ausgegangen
VVPP

VAFIN

3

Offizielle Prognosen APPR

AP

ADJA

PP

VP

NN
NP

sind

NN ausgegangen
VVPP

VAFIN

S

VP

S

Figure 4: Sentence pair with target side parse before (left) and after (right) excision of a rule.
Official → ADJA 

forecasts →predicted →sindpredicted just 3 % →

,

VVPP



ausgegangen VAFINsindjust 3 % →

von

von

nur

3

nur


%

ausgegangen

NP




3

Official

X →

NP



%

predicted

Prognosen

Offizielle
PP



VP

,

Official forecasts →Prognosen

Offizielle
 VAFIN

NN

Offizielle
X → VAFIN
sind


NN

VP

,
PP



ausgegangen

Figure 5: String-to-tree rules extracted from the sentence pair of Figure 4.
lexical items or X.
• It cannot start with X and consecutive ‘X’ are forbidden.
All extracted rules are made shallow as in Section 2.1. In Figure 5 we illustrate some string-totree rules that are extracted from the sentence pair left in Figure 4. The excision of the rule with
left-hand side ‘predicted’ from the sentence pair is illustrated right in Figure 4.

3

Non-minimal tree-to-tree rule extraction

To provide a complete picture, we also want to consider a non-minimal tree-to-tree rule extraction. To this end, we modify the (non-minimal) string-to-tree rule extraction of Section 2.2 to extract tree-to-tree rules as follows. We first extract string-to-tree rules, so let
r = w → (t1 , . . . , tm ) be such an extracted rule. Let w = w1 · · · wn be the decompositionof the string w into tokens. Since we want to extract tree-to-tree rules, our sentence pairs are
now bi-parsed (as in Figure 1), so a parse of the source sentence is available. Based on this parse,
we determine whether the left-hand side w corresponds to a constituent in it. If it corresponds toa constituent labeled N , then we construct the tree-to-tree rule N (w10 , . . . , wn0 ) → (t1 , . . . , tm ),where wi0 is simply wi for all lexical items wi and the corresponding constituent for wi = X.
Otherwise we ignore the string-to-tree rule r and proceed with the next one. For example, thestring-to-tree rule
predicted just → VAFIN(sind), ADV(nur), VVPP(ausgegangen)is extracted in the string-to-tree setting for the sentence pair of Figure 4 (left), but ‘predicted
just’ is not a constituent in the source sentence parse of Figure 1 (left). Consequently, thisstring-to-tree rule is ignored.
Since the left hand side w has to match a constituent, the number of extractable rules
is drastically lower than in the string-to-tree setting. Hence, we can remove the last of theconditions described above. Note that rules like those in Figure 2 can be extracted using theProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 204

NP
Official→

Offizielle

forecasts
VP→

predicted

just

3

%

NP



 VAFIN

→
just

nur

3

von

nur

3

NP

AP 



VP

,

sind

QP



Prognosen

→
just

3

%

3

%

VP



→

ausgegangenpredicted

PP


von

 VAFIN

NP

sind

nur


3

%

VP

,
PP



ausgegangen

Figure 6: Tree-to-tree rules extracted by non-minimal rule extraction.
non-minimal tree-to-tree rule extraction. Some additional tree-to-tree rules that can be extracted
from the word-aligned, bi-parsed sentence pair left in Figure 1 are displayed in Figure 6.

4String-to-string rule extraction

Secondly, we want to completely abandon the syntactic annotation and derive a rule extraction
for string-to-string MBOT rules. We again achieve this by a simple modification of the existing
string-to-tree rule extraction of Seemann et al. (2015). Overall, string-to-string MBOT rules are
similar to hierachical rules (Chiang, 2005), but with a sequence of strings in the right-hand side.
We now have no parses at all, so our training data is a simple word-aligned parallel corpus. An
entry of such a corpus is displayed in Figure 7. To accommodate this situation, we performthe same changes mentioned in the step from the tree-to-tree rule extraction to the string-to-tree
rule extraction also for the target side. Thus, we no longer need to identify a node in the target
sentence parse, but rather identify a sequence of phrases (or spans) that are consistent with theword alignment. The additional restrictions on the left-hand side w in the string-to-tree rule
extraction are also imposed onto the left-hand sides of the extracted string-to-string rules, but
we do not impose them onto the right-hand side. These restrictions were imposed to reasonably
limit the number of rules, but it shows that restricting the right-hand side is (generally) notnecessary.
Official

Offizielle

forecasts

Prognosensindpredicted

just

3

%

von

3

%

ausgegangennur

Figure 7: Word-aligned sentence pair.
From the sentence pair in Figure 7 we can, for example, extract the following rules. We
indicate linked nonterminals by using the same subscript on them.
• Official → (Offizielle)• forecasts → (Prognosen)• predicted just → (sind , nur , ausgegangen)• predicted just → (sind von nur , ausgegangen)• just 3 % → (nur 3 %)• just 3 % → (von nur 3 %)• predicted just 3 % → (sind , nur 3 % ausgegangen)• predicted just 3 % → (sind von nur 3 % ausgegangen)• Official X1 → (Offizielle X1 )• predicted X1 → (sind , nur , X1 ausgegangen)5

Experimental Evaluation

Our main contribution is the experimental evaluation of MBOTs in the various settings (treeto-tree, string-to-tree, and string-to-string). We also compare to standard models (SCFG andProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 205

training data
training data size
language modeladd. LM data
LM data size
tuning data
tuning size
test data
test size

English to German
English to Arabic
English to Chinese
7th EuroParl (Koehn, 2005)
MultiUN (Eisele and Chen, 2010)≈ 1.8M sentence pairs≈ 5.7M sentence pairs≈ 1.9M sentence pairs
5-gram SRILM (Stolcke, 2002)
WMT 2013
Arabic in MultiUN
Chinese in MultiUN≈ 57M sentences≈ 9.7M sentences≈ 9.5M sentences
WMT 2013
cut from MultiUN
NIST 2002, 2003, 2005
3,000 sentences
2,000 sentences
2,879 sentences
WMT 2013 (Bojar et al., 2013)
cut from MultiUN
NIST 2008 (NIST, 2010)
3,000 sentences
1,000 sentences
1,859 sentences

Table 1: Summary of the used resources.
hierarchical models) in order to evaluate the effect of the discontinuities offered by the MBOT
model. We try to be comprehensive, but naturally we can only report results for a limited
number of experiments. We chose to perform experiments for the translation directions Englishto-German, English-to-Arabic, and English-to-Chinese. The languages were selected such that
constituency parsers and large parallel corpora are readily available. In addition, we selected
target languages, in which the discontinuity offered by the MBOT model might be useful.
5.1

Resources

For better comparisons, we use exactly the same resources as Seemann et al. (2015) for theevaluation. We summarize the experimental setup in Table 1. We applied length-ratio filtering
to all data sets. Furthermore, all training sets have been word aligned using GIZA++ (Och andNey, 2003) using the grow-diag-final-and heuristic (Koehn et al., 2005).
The tasks required various forms of preprocessing of the data. The English (source) sideof the training data was true-cased and parsed with the provided grammar of the Berkeley
parser (Petrov et al., 2006). Next, we comment on the preprocessing tasks that are specificfor each translation task.
• English-to-German: The German text was also true-cased and parsed with the provided
grammar of BitPar (Schmid, 2004). German is morphologically rich and in order to avoid
sparseness issues, we removed the functional and morphological annotation from the tags
used in the parses.
• English-to-Arabic: The Arabic text was tokenized with MADA (Habash et al., 2009) andtransliterated according to Buckwalter (2002). Since the Berkeley parser (Petrov et al.,2006) also provides a grammar for Arabic, we parsed the Arabic training data with it.
• English-to-Chinese: The Chinese sentences were word-segmented using the Stanford
Word Segmenter (Chang et al., 2008). Again, the Berkeley parser (Petrov et al., 2006)
with its provided grammar delivers the parse trees for the Chinese training data.
After the preprocessing steps, we obtained a word-aligned, bi-parsed parallel corpus, to which
we applied the described rule extractions together with the baseline rule extractions provided by
Moses (Koehn et al., 2007; Hoang et al., 2009). To give a quick overview, we report the numberof extracted rules for all translation tasks and rule extractions in Table 2. We can immediately
confirm that relaxing the conditions during rule extraction (e.g., from tree-to-tree to string-totree) greatly increases the number of extracted rules for each translation task. For example,
the string-to-tree rule extraction for MBOTs (Seemann et al., 2015) extracts 12–17 times morerules than the minimal tree-to-tree rule extraction for MBOTs (Maletti, 2011). In addition,
the availability of several target-side objects (and thus discontinuities) also leads to additional
freedom during rule extraction, which is evidenced by the larger numbers of rules extracted for
MBOTs compared to those extracted for the SCFG baseline. For our experiments we heavily

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 206

Systemtree-to-tree SCFG (baseline)
minimal tree-to-tree MBOT
tree-to-tree MBOT
string-to-tree SCFG (baseline)
string-to-tree MBOT
hierarchical SCFG (baseline)
string-to-string MBOT

Number of extracted rules
English-To-German English-To-Arabic English-To-Chinese
6,630,590
24,358,001
8,161,362
12,478,160
28,725,229
10,162,325
40,736,687
151,322,970
84,220,528
14,092,729
55,169,043
17,047,570
143,661,376
491,307,787
162,240,663
406,433,344
842,290,537
209,482,192
1,084,007,782
2,208,445,501
682,505,767

Table 2: Number of extracted rules for the different rule extractions.
parallelized the processes using different architectures. Unfortunately, this precludes a sensible
analysis of decoding times.
5.2

Translation features

As usual, the task of the decoder is to find the best translation fˆ of the input object e (string ortree) licensed by the translation model and the language model.
fˆ = arg maxf p(f | e) = arg maxf p(e | f ) · p(f )The probability p(f ) is provided by a (string) 5-gram language model for the target language.
Thus, if target syntax is used, then the yield (the sentence written on the frontier) of the tree f
is used for language model scoring. The data used to train the language model and the used
toolkit are reported in Table 1. The translation model provides the probability p(e | f ) and uses
either the MBOT model or the baseline SCFG model as implemented in Moses. More precisely,
the translation model2 uses a log-linear model (Och, 2003) of weighted features hk (·)λk overderivations D for the pair (e, f ).
Yp(e | f ) =maxp(D) =maxhi (D)λi ,D derivation for (e, f )D derivation for (e, f )iwhere hi (·) are features on derivations. The features of the derivation are usually derived as a
product of the rule features of those rules that constitute the derivation. We used the following(mostly standard) rule features (Koehn, 2009):• the forward and backward translation probabilities,• the forward and backward lexical translation probabilities,• the phrase and word penalty, and• the gap penalty, which is specific for MBOTs.
The forward and backward translation probabilities are obtained as normalized relative frequencies. We applied Good-Turing smoothing (Good, 1953) to all rules that were extracted at most
10 times. Both lexical translation probabilities are obtained as usual, and the MBOT-specific
gap penalty is defined as 1001−c , where c is the number of target objects used in all rules that
contributed to D. This feature is intended to allow the model to tune the amount of discontinuity
to the specific target language. Indeed in all experiments the feature weights λi of the log-linear
model were trained using minimum error rate training (Och, 2003).
The task of the decoder is the identification of the best-scoring target object f and derivation D in the above definition of p(e | f ). In all our experiments a CYK+ chart parser is usedas decoder. The decoder for the SCFG model is provided by the syntax component (Hoang
et al., 2009) of the Moses framework, and the decoder for the MBOT model is provided by
MbotMoses branch (Braune et al., 2013) of Moses.
2 Indeed the language model is also scaled with a feature weight λ.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 207

Setting
tree-to-tree

string-to-treehierarchical

SystemMoses (baseline)
minimal MBOT
non-minimal MBOT
Moses (baseline)
MBOT
GHKM (Galley et al., 2004, 2006)
Moses (baseline)
MBOT
phrase-based Moses

En-De
14.50
14.09
14.41
14.96
∗
15.49
17.10
17.00
16.57
16.80

BLEU
En-Ar
43.49
32.88
41.37
48.23
∗
49.10
46.66
51.71
—
51.90

En-Zh
17.63
12.01
16.77
17.69
∗
18.35
18.33
18.74
18.60
18.09

Table 3: BLEU evaluation results for all 3 translation tasks. Starred results indicate statisticallysignificant improvements over the baseline (at confidence p < 1%).

5.3

Quantitative evaluation

In this section, we first compare all systems to each other using the score BLEU (Papineni et al.,2002). We also present the results obtained by systems that were high-ranked on public sharedtasks (Bojar et al., 2014) such as phrase-based systems (Koehn et al., 2003) or string-to-treesystems obtained following Galley et al. (2004, 2006). All systems were tuned for BLEU on thetuning data, and we report the BLEU scores obtained by the tuned systems on the test sets. The
MBOT-based systems were evaluated against their corresponding syntax component (Hoang
et al., 2009) of the Moses toolkit, which implements tree-to-tree, string-to-tree, and stringto-string (hierarchical) rule extractions. All of them follow essentially the procedure outlined
in Chiang (2005), which was also the basis for the rule extraction of Seemann et al. (2015) andour string-to-string rule extraction. Our implementation of the non-minimal tree-to-tree MBOT
rule extraction is also an extension of the corresponding procedure of the syntax component ofMoses. We also checked statistical significance for the MBOT results using the implementationof Gimpel (2011).
We performed large scale experiments on three major translation tasks, namely English-toGerman (En-De), English-to-Arabic (En-Ar), and English-to-Chinese (En-Zh). The goal wasto evaluate the following MBOT systems: (i) the minimal tree-to-tree system (Section 2.1),(ii) the non-minimal tree-to-tree system (Section 3), (iii) the non-minimal string-to-tree system(Section 2.2), and (iv) the string-to-string system (Section 4). The obtained results are reported
in Table 3. Unfortunately, the rule table of the string-to-string MBOT for the English-to-Arabictranslation task — although already filtered on the given input — was too large to load intomain memory (available: 500GB RAM).
Let us now discuss the results for the various settings. Overall, we observe that the tree-totree systems perform worst. For the baseline system using SCFG rules (i.e., MBOT rules witha single tree on both the left- and right-hand side), this result is not surprising. Already, Ambati
and Lavie (2008) have shown that tree-to-tree rules are too restrictive to achieve good lexical
coverage. However, our results show that making rules more flexible by allowing several target
trees hurts the performance instead of yielding improvements. This effect is particularly visible
when only using minimal rules. On the English-to-Arabic and English-to-Chinese translation
tasks, the minimal MBOT system loses 10.61 and 5.62 BLEU points, respectively, over thebaseline. Interestingly, on the English-to-German translation task the loss is only 0.41 BLEU
points. Adding non-minimal MBOT rules yields the expected large improvements, but is overall
still not good enough to beat the tree-to-tree baseline. This result is interesting insofar as it

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 208

English-to-German
MBOT variant
minimal t-to-t
non-minimal t-to-t
non-minimal s-to-t
non-minimal s-to-s

Type
cont.
discont.
cont.
discont.
cont.
discont.
cont.
discont.

Lex
55,9102,167
44,951
4,149
27,351
9,336
29,972259

Struct
4,492
7,386
2,850
2,348
635
1,110
3,600
131

Total
60,402
9,553
47,801
6,497
27,986
10,44633,572390

2Target tree fragments
3
4
5

≥6

6,458

2,389

471

34

1

5,601

821

62

13

–

5,565

3,441

1,076

312

52

387

3

–

–

–

Table 4: Number of rules per type used when decoding test (Lex = lexical rules; Struct =structural rules; [dis]cont. = [dis]contiguous).
does not confirm the results of Sun et al. (2009) on large scale experiments with target side
discontinuities.3
The results for the string-to-tree setting are much better than those for the tree-to-treesystems (see Table 3). The BLEU score improvements are not very pronounced for Englishto-German and English-to-Chinese, but on the English-to-Arabic translation task the stringto-tree systems (both baseline and MBOT) achieve huge improvements. Those systems comein at 4.74 and 5.61 BLEU points, respectively, ahead of the tree-to-tree baseline. As alreadydemonstrated by Seemann et al. (2015), the MBOT system yields significant improvements
over the baseline on all those language pairs. The GHKM systems achieve mixed results. They
outperform the MBOT system on English-to-German, achieve the same performance as the
MBOT system on English-to-Chinese, and lose even against the baseline on the English-toArabic translation task.
Finally, the string-to-string systems generally yield the best translation quality (as measured by BLEU). The experiments for the English-to-German and English-to-Chinese translation task show that our string-to-string MBOT system does not improve performance in thesecases. Indeed, the analysis presented in Section 6 suggests that the string-to-string rules are
flexible enough to achieve high coverage even without the need for multiple phrases in the righthand side. This is slightly disappointing as Galley and Manning (2010) incorporated discontinuous phrases into a phrase-based system, and their evaluation on Chinese-to-English showedsignificant improvements over a standard phrase-based baseline as well as over a hierarchicalbaseline. However, the differences are generally not large, and even the phrase-based systemachieves similar performance.

6

Analysis of Discontinuity

Another goal was to identify whether discontinuous rules are useful and to what extent these areuseful. We try to estimate their impact on the translation quality by inspecting the statistics on
the rules used in the derivations. Consequently, only rules that produce part of the final output ineach of the translation tasks count. The current tools of MbotMoses (Braune et al., 2013) only
allow the counting of rules used during decoding. At present, it is infeasible to track discontinuous objects through the derivation to decide whether they are actually assembled continuouslyor discontinuously. Thus, discontinuous rules only indicate a potential discontinuity.
3 The experiments of Sun et al. (2009) report scores for the translation task Chinese-to-English for systems trained on
240,000 sentences only. Their model allows discontinuities on the source language side, which should be comparable
to target-side discontinuities for the opposite translation direction English-to-Chinese.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 209

English-to-Arabic
MBOT variant
minimal t-to-t
non-minimal t-to-t
non-minimal s-to-t

Type
cont.
discont.
cont.
discont.
cont.
discont.

Lex
18,389
1,138
9,826
1,605
1,839
3,670

Struct
2,855
1,920
1,581
746
651
1,324

Total
21,244
3,085
11,407
2,315
2,490
4,994

2

Target tree fragments
3
4
5 ≥6

2,525

455

67

8

3

1,577

565

158

38

13

3,008

1,269

528

153

36

Table 5: Number of rules per type used when decoding test (Lex = lexical rules; Struct =structural rules; [dis]cont. = [dis]contiguous).

Tables 4, 5, and 6 show the statistics on the rules used during decoding. Continuous rules(i.e., rules with a single object in the right-hand side) are abbreviated by cont, and (potentially)
discontinuous rules are abbreviated by discont. To provide a deeper analysis, we also distinguish between lexical and structural rules. Lexical rules, abbreviated Lex, are rules that containno exposed nonterminal symbols. Similarly structural rules, abbreviated Struct, are rules containing at least one such nonterminal symbol. Finally, we abbreviate the settings tree-to-tree,
string-to-tree, and string-to-string by ‘t-to-t’, ‘s-to-t’, and ‘s-to-s’, respectively.
We first discuss the results for the tree-to-tree systems presented across Tables 4, 5, and 6.
If we only use minimal MBOT rules, then 11% of the rules used during decoding are discontiguous in the English-to-German and English-to-Chinese translation tasks. The rate is slightly
higher in the English-to-Arabic translation task (14.5%). For all the translation tasks, the majority of the discontinuous rules are structural. This fact is not very surprising since the leavesof the minimal tree-to-tree rules are either lexical items or exposed non-terminal occurrences.
The minimality constraint encourages word-by-word translation, and once the lexical rules are
excised, only structural rules remain. Based on the observed high BLEU score losses, it seems
that minimal tree-to-tree rules are, at present, unable to correctly assemble discontinuous parts.
If we additionally use non-minimal tree-to-tree rules, then the rates of discontinuous ruleschange. For English-to-German the rate remains almost the same at 13%, whereas for the
English-to-Arabic translation task it increases to 20%. Finally, for English-to-Chinese suddenly only 4% of the rules applied during decoding are discontinuous. The non-minimality
encourages large rules, which are more likely to contain only lexical items. As expected, thenumber of discontinuous lexical rules is always larger than the number of discontinuous structural rules in this setting. This is particularly true for English-to-German and English-to-Arabic,
where two third of the discontinuous rules are lexical, whereas the distribution is almost even forEnglish-to-Chinese. We believe that these lexical discontinuous rules capture relevant idiomatic
expressions or encode agreement or correspondences, which yield the large improvements intranslation quality over minimal rules only.
For string-to-tree systems we only present the statistics since they have been discussed
already by Seemann et al. (2015). It is evident that they generally use the largest amounts of
discontinuous rules, which is rewarded with significant improvements over the baseline system
without discontinuieties.
Finally, for the string-to-string systems, the opposite situation presents itself. Here, thenumber of discontinuous rules is indeed marginal. On the English-to-German translation task
only 1.1% of 33,962 rules are discontinuous. The English-to-Chinese system also only uses
2.3% discontinuous rules (out of 25,575 rules). We believe that the low use of discontinuous
string-to-string rules can be explained by the absence of linguistic annotations. Without them,Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 210

English-to-Chinese
MBOT variant
minimal t-to-t
non-minimal t-to-t
non-minimal s-to-t
non-minimal s-to-s

Type
cont.
discont.
cont.
discont.
cont.
discont.
cont.
discont.

Lex
34,275516
35,031771
17,135
4,822
15,769108

Struct
8,820
4,292
2,045
744
1,585
3,341
9,208
490

Total
43,095
4,808
37,076
1,515
18,7208,163
24,977598

Target tree fragments
2
3
4
5 ≥6
3,816

900

82

6

4

1,222

248

36

4

5

6,411

1,448

247

55

2

591

7

–

–

–

Table 6: Number of rules per type used when decoding test (Lex = lexical rules; Struct =structural rules; [dis]cont. = [dis]contiguous).

the rules become very flexible, thus removing the need for discontinuous MBOT rules in thissetting. Since the number of used discontinuous rules is so low, it can be assumed that essentially the same rules were used during decoding when comparing the MBOT system to thebaseline. This would also explain their comparable BLEU scores.

7

Conclusion

We have extended the existing rule extraction techniques for shallow local multi bottom-up treetransducers to the two main missing settings. First, we designed a non-minimal tree-to-tree
rule extraction for MBOT, which extends the corresponding rule extraction for minimal rules.
Secondly, we developed a rule extraction for the string-to-string setting, which does not rely on
syntactical information. Naturally, we also evaluated these new rule extractions together withseveral other systems in 3 large scale translation tasks (English-to-German, English-to-Arabic,
and English-to-Chinese).
As expected, the non-minimal tree-to-tree system performs much better than the corresponding system using only minimal rules, but even the system with non-minimal rules does
not beat the SCFG baseline (using non-minimal rules). It seems that discontinuity remains achallenge for tree-to-tree rules. Overall, tree-to-tree systems report the worst scores. For thestring-to-tree systems already Seemann et al. (2015) report significant improvements in translation quality when using discontinuous rules. Finally, in the string-to-string (hierarchical) setting,discontinuous rules are hardly ever used, so when compared to the SCFG baseline essentiallythe same performance is obtained. Most likely, hierarchical rules are flexible enough to handle
most common forms of discontinuity without the need to explicitly represent it in its rules. In
summary, since MBOT offers certain consistent advantages across the different language pairs
it may be useful to exploit a hybrid approach in the future.
To support further experimentation by the community, we publicly release our developed
software and analysis tools (http://www.ims.uni-stuttgart.de/forschung/ressourcen/werkzeuge/mbotmoses.en.html).

Acknowledgement
The authors would like to thank the reviewers for their helpful comments and suggestions.
All authors were financially supported by the German Research Foundation (DFG) grant
MA 4959 / 1-1, which we gratefully acknowledge.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 211

References
Ambati, V. and Lavie, A. (2008). Improving syntax driven translation models by re-structuring
divergent and non-isomorphic parse tree structures. In Proc. AMTA 2008, pages 235–244.
Bojar, O., Buck, C., Callison-Burch, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post,
M., Soricut, R., and Specia, L. (2013). Findings of the 2013 Workshop on Statistical MachineTranslation. In Proc. 8th WMT, pages 1–44. Association for Computational Linguistics.
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P.,
Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. (2014). Findings of the2014 workshop on statistical machine translation. In Ninth Workshop on Statistical MachineTranslation, WMT, pages 12–58, Baltimore, Maryland.
Braune, F., Seemann, N., Quernheim, D., and Maletti, A. (2013). Shallow local multi bottomup tree transducers in statistical machine translation. In Proc. 51st Annual Meeting of the
Association for Computational Linguistics, pages 811–821.
Buckwalter, T. (2002).
Arabic transliteration.
transliteration.htm.

http://www.qamus.org/

Chang, P.-C., Galley, M., and Manning, C. D. (2008). Optimizing Chinese word segmentation
for machine translation performance. In Proc. 3rd WMT, pages 224–232. Association for
Computational Linguistics.
Chiang, D. (2005). Hierarchical phrase-based translation. In Proceedings of the 43th Annual
Meeting of the Association of Computational Linguistics, page 263270. Association for Computational Linguistics.
Chiang, D. (2010). Learning to translate with source and target syntax. In Proc. 48th ACL,
pages 1443–1452. Association for Computational Linguistics.
Eisele, A. and Chen, Y. (2010). MultiUN: A multilingual corpus from United Nation documents. In Proc. 7th LREC, pages 2868–2872. European Language Resources Association.
Eisner, J. (2003). Learning non-isomorphic tree mappings for machine translation. In Proceedings of the 41th Annual Meeting of the Association of Computational Linguistics, pages
205–208.
Galley, M., Graehl, J., Knight, K., Marcu, D., Deneefe, S., Wang, W., and Thayer, I. (2006).
Scalable inference and training of context-rich syntactic translation models. In In ACL, pages
961–968.
Galley, M., Hopkins, M., Knight, K., and Marcu, D. (2004). What’s in a translation rule? In
HLT-NAACL.
Galley, M. and Manning, C. D. (2010). Accurate non-hierarchical phrase-based translation. In
HLT-NAACL, pages 966–974. The Association for Computational Linguistics.
Gimpel, K. (2011). Code for statistical significance testing for MT evaluation metrics. http://www.ark.cs.cmu.edu/MT/.
Good, I. J. (1953). The population frequencies of species and the estimation of population
parameters. Biometrika, 40(3–4):237–264.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 212

Habash, N., Rambow, O., and Roth, R. (2009). MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. In Proc. 2nd MEDAR, pages 102–109. Association for Computational Linguistics.
Hoang, H., Koehn, P., and Lopez, A. (2009). A unified framework for phrase-based, hierarchical, and syntax-based statistical machine translation. In In Proceedings of the International
Workshop on Spoken Language Translation (IWSLT, pages 152–159.
Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In Proc. 10th
MT Summit, pages 79–86. Association for Machine Translation in the Americas.
Koehn, P. (2009). Statistical Machine Translation. Cambridge University Press.
Koehn, P., Axelrod, A., Mayne, A. B., Callison-Burch, C., Osborne, M., and Talbot, D. (2005).
Edinburgh system description for the 2005 IWSLT Speech Translation Evaluation. In Proc.
2nd IWSLT, pages 68–75.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007).
Moses: Open Source Toolkit for Statistical Machine Translation. In Proc. of ACL: Demo andPoster Sessions, pages 177–180, Prague, Czech Republic. ACL.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedingsof the 2003 Conference of the North American Chapter of the Association for Computational
Linguistics, pages 48–54. Association for Computational Linguistics.
Maletti, A. (2011). How to train your multi bottom-up tree transducer. In Proceedings of the49th Annual Meeting of the Association of Computational Linguistics, pages 825–834.
NIST (2010). NIST 2002 [2003, 2005, 2008] open machine translation evaluation. Linguistic
Data Consortium. LDC2010T10 [T11, T14, T21].
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proc. 41st
ACL, pages 160–167. Association for Computational Linguistics.
Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models.
Computational Linguistics, 29(1):19–51.
Papineni, K., Roukos, S., Ward, T., and jing Zhu, W. (2002). B LEU: a method for automatic
evaluation of machine translation. In Proc. 40th ACL, pages 311–318. Association for Computational Linguistics.
Petrov, S., Barrett, L., Thibaux, R., and Klein, D. (2006). Learning accurate, compact, and interpretable tree annotation. In Proc. 44th ACL, pages 433–440. Association for Computational
Linguistics.
Schmid, H. (2004). Efficient parsing of highly ambiguous context-free grammars with bit vectors. In Proc. 20th COLING, pages 162–168. Association for Computational Linguistics.
Seemann, N., Braune, F., and Maletti, A. (2015). String-to-tree multi bottom-up tree transducers. In Proc. 53rd ACL, pages 815–824. Association for Computational Linguistics.
Stolcke, A. (2002). SRILM — an extensible language modeling toolkit. In Proc. 7th INTERSPEECH, pages 257–286.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 213

Sun, J., Zhang, M., and Tan, C. L. (2009). A non-contiguous tree sequence alignment-basedmodel for statistical machine translation. In Proceedings of the Joint Conference of the 47th
Annual Meeting of the ACL, pages 914–922.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 214

Register-Based Machine Translation Evaluation
with Text Classification Techniques
Mihaela Velam.vela@mx.uni-saarland.de

Saarland University, Saarbrücken, Germany

Ekaterina Lapshinova-Koltunski

e.lapshinova@mx.uni-saarland.de

Saarland University, Saarbrücken, Germany

Abstract
This paper presents a novel approach to machine translation evaluation by combining registerfeatures – characterised by particular distributions of lexico-grammatical features – with textclassification techniques. The goal of this method is to compare machine translation output
with comparable originals in the same language, as well as with human reference translations.
The degree of similarity – in terms of register features – between machine translations andoriginals, and machine translations and reference translations is measured by applying two
text classification methods trained on 1) originals and 2) reference translations, and tested onmachine translations. The results from the experiments prove our assumption that machinetranslations share register features rather with human translations than with non-translated texts
produced by humans. This confirms that registers are one of the most important factors thatshould be integrated into register-based machine translation evaluation.

1

Introduction: Motivation and Goals

The state-of-the-art in evaluating machine translation (MT) nowadays is to measure lexicaland eventually syntactic and semantic overlap between a machine translation (called hypothesis translation) and a human-produced reference translation. In this paper, we present a new
approach to evaluation, integrating the knowledge on register, i.e. language variation according to context, as defined by Halliday and Hasan (1989) and Quirk et al. (1985). The difference in terms of register between original and translated texts has been shown by several
studies (Hansen-Schirra et al., 2012; Kruger and van Rooy, 2012; Neumann, 2013), provingthat translations tend to share a set of lexical, syntactic and/or textual features. More recent
investigations by Baroni and Bernardini (2006), Kurokawa et al. (2009) and Lembersky et al.
(2012) applied text classification methods to automatically identify these differences.
The aim of the research presented here is twofold: 1) to show that machine translations andthe corresponding reference translations are related to each other in terms of register-specificfeatures and as a consequence of this 2) to show that hypothesis translations and human translations share more than the lexical surface. The novel idea introduced here is the notion ofregister-specific features which relate reference and hypothesis translations, and therefore have
implications for MT (evaluation).
We measure the “closeness” between comparable non-translated originals and machinetranslations, as well as between human and machine translations by applying two different classification methods. The classification is performed on the basis of extracted register-specificfeatures for two data sets. First, we use original non-translated texts as training data and ma-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 215

chine translations as test data. In a second step, human translations are used as training data
and machine translations as test data. Our assumption is that in terms of register specificity
quantified in the corresponding features, MT output is closer to the corresponding referencetranslations than to the comparable non-translated originals. We base our hypothesis on the factthat b) translations tend to normalise towards target language conventions and that a) machinetranslations will adapt more to these conventions than to source texts (Diwersy et al., 2014).
The remainder of the paper is structured as follows. In Section 2, we present related workfrom the areas of machine translation evaluation and register theory. In Section 3 we present
our research questions, describe the selected features and resources used for the experiments, aswell as the applied methods. Section 4 demonstrates the results of our analysis, and in Section
5, we discuss the outcome and give an outlook on further analyses.

2

Related Work

2.1

Machine Translation Evaluation

State-of-the-art MT evaluation applies automatic language-independent metrics such as
BLEU (Papineni et al., 2002) and NIST (Doddington, 2002) in order to compare MT output(hypothesis translation) with one or more human translations (reference translations). Several
studies (Callison-Burch et al., 2006; Vela et al., 2014a,b) have confirmed the fact that BLEU
scores should be treated carefully, thus advancing the development of new metrics. New evaluation metrics such as METEOR (Denkowski and Lavie, 2014), Asiya (Gonzàlez et al., 2014)
and VERTa (Comelles and Atserias, 2014), are incorporating lexical, syntactic and semantic information into their scores, whereas metrics like BEER (Stanojević and Sima’an, 2014),
ReVal (Gupta et al., 2015) and COMET (Vela and Tan, 2015) use machine learning approaches
for MT evaluation. The accuracy of the newly introduced evaluation methods is usually provenby human evaluation inputs, more specifically by measuring the correlation of the automatically provided scores with human judgements. Human evaluation is realised by ranking MT
outputs (Bojar et al., 2013, 2014; Vela and van Genabith, 2015). In addition, post-editing, which
is mainly used for measuring productivity (Guerberof, 2009; Zampieri and Vela, 2014), is also
a valid human evaluation method.
2.2

Main Notions within Register Theory

Studies related to register theory (Quirk et al., 1985; Halliday and Hasan, 1989; Biber, 1995)
are concerned with contextual variation of languages, and state that languages vary with respect
to usage context within and across languages. For example, languages may vary according to
the activity of the participants involved or the relationship between speaker and addressee(s).
These parameters correspond to the variables of (1) field, (2) tenor and (3) mode defined in theframework of systemic functional linguistics (SFL), which describes language variation according to situational contexts; see, for instance, Halliday and Hasan (1989) and Halliday (2004).
These variables are associated with the corresponding lexico-grammatical features, e.g. field ofdiscourse is realised in term patterns or functional verb classes, e.g. activity (approach, supply,etc.), communication (answer, inform, suggest, etc.) and others; tenor is realised in modality
expressed e.g. by modal verbs (can, may, must, etc.) or stance expressions (used by speakers to convey personal attitude to the given information, e.g. adverbs like actually, certainly,
amazingly, importantly, etc.); and mode is realised in information structure and textual cohesion, e.g. coreference via personal (she, he, it) and demonstrative (this, that) pronouns. Thus,
differences between registers can be identified through the analysis of occurrence of lexicogrammatical features in these registers; see Biber’s studies on linguistic variation, e.g. Biber(1988), Biber (1995) or Biber et al. (1999). The field of discourse also includes experiential
domain realised in the lexis. This corresponds to the notion of domain used in the machine

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 216

translation community. However, it also includes colligation (morpho-syntactic preferences ofwords), in which grammatical categories are involved. Thus, domain is just one of the parameterfeatures a register can have.
2.3

Register in Translation

Several studies in systemic functional linguistics are concerned with register settings in humantranslation (Steiner, 2004; Hansen-Schirra et al., 2012; De Sutter et al., 2012; Neumann, 2013;
House, 2014) and their application into translation practice (Vela and Hansen-Schirra, 2006;
Vela et al., 2007). To our knowledge, the machine translation (including its evaluation) community has not yet taken into consideration the notion of register, at least according to the definitionin the present paper. Studies in the field of MT concerned with translation errors of new domains are covering only the lexical level (Irvine et al., 2013), as the authors operate solely with
the notion of domain (field of discourse) and not register (which includes more parameters, as
described in Section 2.2 above). Research on adding in-domain bilingual data to the training
material of SMT systems (Eck et al., 2004; Wu et al., 2008) or on application of in-domain comparable corpora (Laranjeira et al., 2014; Irvine and Callison-Burch, 2014) consider the notionof domain. However, further register features are mostly ignored.
Domains reflect what a text is about, i.e. its topic. So, consideration of domain alone wouldclassify news reporting on certain political topics together with political speeches discussing
the same topics, although they belong to different registers. We expect that texts from the latter (political speeches) translated with a system trained on the former (news) would be lacking
in persuasiveness, argumentation and other characteristics reflected in their lexico-grammaticalfeatures, e.g. imperative verbal constructions used to change the addressee’s opinion, or interrogatives as a rhetorical means, etc. The similarity in domains would cover only the lexicallevel, in most cases terminology, ignoring the lexico-grammatical patterns specific for the given
register, as shown by Lapshinova-Koltunski and Pal (2014) in their discussion on domain vs.
register. Although some NLP studies employing web resources are arguing for the importanceof register conventions, as by Santini et al. (2010), register remains out of the focus of machinetranslation. One of the few works addressing the relevance of register features for machinetranslation is Petrenz (2014), in which the author uses text features to build cross-lingual register classifiers.

3

Methodology and Resources

3.1

Research Questions

Following the assumption that translated language should normalise the linguistic features (likethose described in Section 2.2 above) in order to adapt them to target language conventions,
we use two different classification methods, KNN and SVM, to prove that in terms of registersettings 1) machine translations correspond to human reference translations to a greater extent
than 2) to comparable original non-translated texts in the same language. This requires a twofold experiment design. In the first experiment, we use German original data for training andGerman machine translations for testing. Based on classification accuracy we can determine the“closeness” of machine translations to comparable non-translated texts in the same language.
For the second experiment, we use a different data set applying the same classification methods.
Human reference translations are used as training data and machine translations as test data.
In this way we can observe the relation between machine translations and comparable nontranslated texts in the same language, as well as between machine translations and referencetranslations.
We also aim at answering the following questions:

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 217

(i) Do German machine translations correspond to comparable German non-translated originals?
(ii) Are German machine translations closer to human reference translations than to comparable original German texts?
(iii) What are the main parameters influencing the classification outcome?
Our assumption is that machine translations will comply more with register standards ofhuman-produced translated texts rather than with non-translated texts written by humans, as it
was shown by Lapshinova-Koltunski and Vela (2015).
3.2

Feature Selection

For our analysis, we select a set of features derived from register studies described in Section
2.2 above. These features represent lexico-grammatical patterns of more abstract concepts, i.e.
textual cohesion expressed via pronominal coreference or other cohesive devices, evaluative
patterns (e.g. it is interesting/important that, etc.) and others. The selected features reflect
linguistic characteristics of all texts under analysis, are content-independent (do not contain terminology or keywords), and are easy to interpret yielding insights on the differences between
the analysed variables. For instance, we use groupings of specific types of phrases (e.g. nominal, verbal, etc.) instead of part-of-speech n-grams, as they are easier to interpret as n-grams.
The set of selected features for our analysis is outlined in Table 1. The first column denotes
the extracted and analysed patterns, the second represents the corresponding linguistic features,and the third denotes the three context parameters according to register theory as previously
described in Section 2.2.
The number of nominal and verbal parts-of-speech, chunks and nominalisations (ungnominalisations) reflects participants and processes in the field parameter. The distribution of
abstract or general nouns and their comparison to other nouns gives information on the vocabulary (parameter of field). Modal verbs grouped according to different meanings (Biber et al.,1999), and evaluation patterns express modality and evaluation, i.e. the parameter of tenor.
Content words and their proportion to the total number of words in a text represent lexicaldensity, which is an indicator of the parameter of mode. Conjunctions, for which we analyse
distributions of logico-semantic relations, belong to the parameter of mode as they serve as
discourse-structuring elements. Reference expressed either in nominal phrases or in pronouns
reflects textual cohesion (mode). Overall, we define 21 features representing subtypes of thecategories given in Table 1.
patternnominal and verbal chunks
ung-nominalisations and general nouns
modals with the meanings of permission, obligation, volition
evaluative patterns
content vs. functional words
additive, adversative, causal, temporal, modal
conjunctive relations
3rd person personal and demonstrative pronouns

featureparticipants and processes
vocabulary and stylemodality
evaluation
lexical density
logico-semantic relations

parameter
field
tenor

mode

cohesion via reference

Table 1: Features under analysis

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 218

However, for the final interpretation, a reduced number of features is used, which resultsfrom the validation step described in Section 3.4.1 below.
3.3

Data

In the first experimental setting, the training data set consists of German non-translated texts(GO=German originals) extracted from CroCo (Hansen-Schirra et al., 2012), a corpus of bothparallel and comparable texts in English and German. The dataset contains 108 texts which
cover seven registers: political essays (ESSAY), fictional texts (FICTION), manuals (INSTR), popular-scientific articles (POPSCI), letters to shareholders (SHARE), prepared political speeches (SPEECH), and tourism leaflets (TOU). The decision to include this wide rangeof registers is justified by the need for heterogeneous data for our experiment. Therefore, thedataset contains both frequently machine-translated texts, e.g. SPEECH, ESSAY and INSTR,
and those, which are commonly not translated with MT systems, such as FICTION or POPSCI.
The totals number of tokens in GO is 252711.
The corresponding test data set is smaller and includes 50 texts translated from English
into German with a rule-based (RBMT) and a statistical (SMT) machine translation system.
The rule-based machine translations were produced with the rule-based system SYSTRAN6(Systran, 2001). The statistical machine translations were produced with a Moses-based system1 , trained with EUROPARL (Koehn, 2005), a parallel corpus containing texts from the proceedings of the European parliament. The total number of tokens in RBMT and SMT comprise
127865 and 124462 respectively. Both variants contain translations of the same texts belonging
to the same registers as in the originals (training data).
In the second setting, English to German human translations (HT) – extracted from the
CroCo corpus – are used for training, and comprise 100 texts (262655 tokens). For testing we
use the same data as in the previous setting – machine translated texts (RBMT and SMT). Bothtraining and test data are translations of the same source texts, which corresponds a common
setting in MT evaluation. Table 2 gives an overview of the number of texts, sentences andtokens in both experiment settings.

Texts
Sentences
Tokens

Train
GO
108
15736
252711

Setting 1
Test
RBMT
SMT
50
50
6195
6131
127865 127865

Train
HT
100
13077
262655

Setting 2
Test
RBMT
SMT
50
50
6195
6131
127865 127865

Table 2: Statistics for the data sets used in experiments
To extract the occurrences of register features described in Section 3.2, we annotate bothtraining and test sets with information on token, lemma, part-of-speech, syntactic chunks andsentence boundaries using Tree Tagger (Schmid, 1994). The availability of these annotationlevels in both corpora allows us to analyse certain lexico-grammatical patterns (see Section 3.2)
required for register-sensitive analysis of translation. The features are then defined as linguistic
patterns and modelled as regular expressions for the Corpus Query Processor (Evert, 2005),
available within the CWB tools (CWB, 2010).
3.4Classification Approaches

For our classification task, we train two models by using two different classifiers for each experiment setting: k-nearest-neighbors (KNN), a non-parametric method, and support vector
1 We did not perform tuning for the SMT system.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 219

machines (SVM) with a linear kernel, a supervised method, both commonly used in text classification. The models are then tested on German translations, and in this way, we obtain classification performance scores for:
• KNN and SVM, having German originals as training data and machine translations(English-German) as test data;• KNN and SVM, having human translations (English-German) and again machine translations (English-German) as test data2 .
The performance scores are then judged in terms of precision, recall and f-measure. Thesescores are class-specific and indicate the results of automatic assignment of register labels tocertain machine-translated texts. In case of precision, we measure the class agreement of thedata with the positive labels given by the classifier. For example, there are ten German fictional
texts in our data. If the classifier assigns FICTION labels to ten texts only, and all of them reallybelong to FICTION, then we will achieve the precision of 100%. With recall, we measure if alltranslations of a certain register were assigned to the register class they should belong to. So,
if we have 10 fictional texts, we would have the highest recall if all of them are assigned with
the FICTION label. F-measure combines both precision and recall, and is understood as the
harmonic mean of both.
3.4.1 K-Nearest Neighbors (KNN)
When using KNN, the input consists of the K closest training examples in the feature space3 ,and the output is a class membership. This method is instance-based, where each instance iscompared with existing ones using a distance metric, and the distance-weighted average of theclosest neighbours is used to assign a class to the new instance, see (Aha et al., 1991; Witten
et al., 2011).
For our experiments we have to determine the final number for K and the most appropriate number of features used for classification. By measuring the distribution of errors during
training (by performing 10-fols cross-validation) we determined the best K=114 and the final
number of features for our setting. For our classification analysis we work with the tuple (numberOfFeatures=17, K=11), performing classification on the German translation (test) data byusing the knn library in Weka (Hall et al., 2009). The final list of features include:→ total words – words per text
→ content words – content (lexical) words→ NP chunks – nominal chunks→ VP chunks – verbal chunks→ chunks – chunks per text
→ nominal – nominal part-of-speech categories→ verbal – verbal part-of-speech categories→ adversative – adversative conjunctive relations→ causal – causal conjunctive relations→ temporal – temporal conjunctive relations→ modal – modal conjunctive relations→ ung-nominalisations – nominalisations formed with ung-suffix→ pron – personal pronouns→ dempron – demonstrative pronouns→ pronnp – nominal phrases filled with pronouns→ gnouns – general nouns→ modals denoting permission
2 Note that human and machine translations are translation variants of the same English source text.
3 In our experiments, the features are quantified by their frequency in the corresponding data set.
4 The final value for K was chosen from an interval between 3 and 19

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 220

3.4.2 Support Vector Machines (SVM)
When using SVM models (Vapnik and Chervonenkis, 1974), the learning algorithm tries to findthe optimal boundary between classes by maximising the distance to the nearest training data ofeach class. Given German labelled training data, the algorithm outputs an optimal hyperplane
which categorises new instances, here German translations5 . We use the same list of features forthe classification with SVM as in the classification with KNN. We perform SVM classification
with a 10-fold cross-validation.
The cross-validation in the training phase has shown that registers SPEECH and SHARE
show low accuracy, especially in the first experiment setting (when German original data isused). For this reason, we we exclude these registers from further analyses.
3.5

Experimental Setup

In the first setting, both classifiers are supposed to store all cases from German originals (108data points) with the corresponding register labels available. New cases from test data, which
are machine translations in this case (50+50 data points), are then classified, i.e. assigned register labels. Classification is performed on the basis of distance function measure, for which
Euclidean distance is used. The results of automatic assignment are indicated with scores (precision, recall and f-measure), which are used to measure the class agreement of the data withthe positive labels given by the classifiers.
In the second setting, both classifiers store all labelled cases available in human translations(100 data points). The trained model is then applied on the same machine translations as in thefirst setting (50+50 data points). And again, we use precision, recall and f-measure to judge theclass agreement of the data with the positive labels given by the classifiers.

4Classification Results

As already mentioned above, the results of both classification algorithms are analysed in terms
of precision, recall and f-measure. In case of precision, we measure the class agreement of thedata with the positive labels given by the classifier. Our assumption is that precision values
would indicate if the test data correspond to the training data in terms of the register settings.
Hence, in the first experiment setting, the higher the precision, the better a machine translation
corresponds to comparable originals, whereas in the second setting, the higher the precision,
the better a machine translation corresponds to human translations.
4.1

Setting 1: Originals vs. Machine Translations

RBMT
SMT

ESSAY
KNN SVM
0.36
0.00
0.48
0.00

FICTIONKNN SVM
0.86
0.75
0.86
0.80

INSTR
KNN SVM
0.17
0.55
0.33
0.60

POPSCIKNN SVM
0.53
0.00
0.46
0.00

TOU
KNN SVM
0.50
0.32
0.46
0.29

Table 3: F-measure scores for classification per machine translation variant and register in thefirst setting
Table 3 provides the confusion matrix for the five registers and the two MT outputs split
on the two classification methods. Concerning both the classification method and the MT system, we notice that FICTION is the register performing best, achieving an f-measure of 0.86%
for the combination RBMT-SVM and SMT-SVM. Based on the f-measure we observe that
5 One of the reasons why SVM are often used is their robustness towards overfitting, as well as their ability to map
to a high-dimensional space.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 221

ESSAY performs well for the classification with KNN but fails with SVM. INSTR and TOU
perform similar in terms of f-measure for both KNN and SVM, the lowest f-measure values being measured for INSTR in the combination RBMT-KNN. POPSCI fails for the combinations
RBMT-SVM and SMT-SVM, but performs well for KNN (0.53% for RBMT and 0.46% for
SMT). These results imply that overall, FICTION performs best for both translation types andboth classifiers, which indicates that translated fictional texts obey best to original fictional texts
in terms of register settings.
Obviously, the data used for developing/training MT systems as well as the classification
method play an important role. Registers like ESSAY (political essays) and INSTR (manuals)
are usually used for training SMT systems, whereas registers like FICTION (fictional texts)
and TOU (tourism leaflets) are less likely used for training MT systems. The more surprising
are here the results for FICTION. The big difference in the results for both classifiers can be
explained in the distinction between these two classification techniques. KNN uses all training
data (predefined neighbours) in classification, while for SVM the maximised distance (margin) to the nearest example of each class plays a crucial role (all non-support vectors being
discarded), thus influencing the difference in the classification outcome.
The overview of the performance of both MT systems in Table 4 (split on register andclassification method) reveals that SMT performs better than RBMT in certain combinations forthe registers ESSAY, FICTION and INSTR. The classification failure of the RBMT and SMT
produced POPSCI translations for SVM indicates that SVM is not the appropriate classification
method for this kind of texts. The fact that we observe contradictory results with both classifiers
for ESSAY and POPSCI prevents us to claim that certain registers are generally more difficult
to be identified in translated data than others.
registerESSAY
FICTIONINSTR
POPSCI
TOU

KNN
SMT
RBMT=SMT
SMT
RBMT
RBMT

SVM
RBMT=SMT
SMT
SMT
RBMT=SMT
RBMT

Table 4: Performance of KNN and SVM across registers based on f-measure in the first setting

RBMT
SMT

Precision
KNN SVM
0.43
0.24
0.50
0.32

RecallKNN SVM
0.61
0.56
0.61
0.53

F-Measure
KNN SVM
0.50
0.32
0.54
0.34

Table 5: Classification accuracy per machine translation variant in the first setting
In the last step, we analyse the performance of the MT systems disregarding registers,
see Table 5. We notice that results do not differ much in the MT systems under analysis,
which means that register agreement between non-translated and machine-translated texts is
not dependent on the method involved in translation. This is proven by Pearson’s chi-square
test, which confirms our observation, as for both the KNN and the SVM results, p-value ishigher than 0.05 (0.85 and 0.58 respectively).
4.2

Setting 2: Reference Translations vs. Machine Translations

The same experiments and analysis steps as in Section 4.1 are performed for the second setting,where human reference translations are used as training data and hypothesis translations as test

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 222

data. However, we observe a strong improvement in the results presented in Table 6. The
registers ESSAY and FICTION achieve the best performance, showing an f-measure of up to
100%. Over all, f-measure remains over 50% for all registers, which means that classifiers
perform also well for TOU, POPSCI and INSTR. These results can, in fact, serve as indicatorsthat human and machine translations have more similarities, sharing register features.

RBMT
SMT

ESSAY
KNN SVM
0.93
1.00
0.93
1.00

FICTIONKNN SVM
1.00
1.00
0.50
0.80

INSTR
KNN SVM
0.54
0.73
0.62
0.67

POPSCIKNN SVM
0.71
0.77
0.60
0.73

TOU
KNN SVM
0.57
1.00
0.75
0.83

Table 6: F-measure scores for classification per machine translation variant and register in thesecond setting
In this setting, ESSAY matches register features of human translations best, leading to theassumption that the texts used for developing and training MT systems play a key role. In contrast to the results in the first setting, the difference between the results of the two classification
methods for ESSAY is minor. The lowest value is scored by INSTR with an f-measure of 0.54
for RBMT-KNN.
Table 7 demonstrates that, different from the results in the first experiment in Section 4.1,
RBMT performs better then SMT when compared to human reference translations, with some
exceptions. RBMT-INSTR and RBMT-TOU are better classified than the same registers translated with the SMT system, if the results from KNN are taken into account.
registerESSAY
FICTIONINSTR
POPSCITOU

KNN
RBMT=SMT
RBMT
SMT
RBMT
SMT

SVM
RBMT=SMT
RBMT
RBMT
RBMT
RBMT

Table 7: Performance of the classification methods across registers based on the f-measure in
the second setting
Similar tendencies are observed if we average the scores for registers.

RBMT
SMT

Precision
KNN SVM
0.81
0.90
0.82
0.84

RecallKNN SVM
0.75
0.90
0.65
0.80

F-Measure
KNN SVM
0.75
0.90
0.68
0.80

Table 8: Classification accuracy per translation variant in the second setting
We observe in Table 8 that, in terms of MT system, RBMT performs almost always better
than SMT, and in terms of classification method applied, SVM performs always better than
KNN. However, significance test show that the difference between both machine translation
systems is not significant, as the computed p-values exceed 0,05 for both KNN and SVM scores(0.78 and 0.97 respectively), confirming not only our assumption that reference and hypothesis
translations are similar if register features are considered, but also that both machine-translated
variants are similar as well.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 223

5Conclusion

The results of the presented experiments have proven our assumption that machine translations
share their register features rather with human produced translations than with human produced
non-translated texts, regardless the method involved in translation.
Setting1: Original vs. Machine Translation The results of the first experiment show that
register settings of most German machine translations do not comply with the register settingsof non-translated German, this being shown also by Lapshinova-Koltunski and Vela (2015).
This is especially valid for ESSAY and POPSCI, which show low performance in most classification scenarios, not adapting the target language register conventions. The good performancefor FICTION is an indicator that fictional texts adapt to the conventions of the target language.
However, as known from Neumann (2013), German and English fictional texts share a lot offeatures, which might also influence the performance of the classifiers. We suppose that the influence of the source language register conventions6 might also have an impact on the outcome.
To prove this, we would need to perform additional experiments, in which English source texts
should be used as training data.
Furthermore, as the performed significance tests have shown that the difference between
RBMT and SMT is not meaningful, our suggestions apply regardless the translation method
involved. In case of SMT, the results are apparently influenced by the training data used, asclassification performs better for registers which are commonly used for SMT training, i.e. like
ESSAY and INSTR. Off-the-shelf RBMT systems, like the one used here, are developed tocover a more general aspect of language which also essentially complicate the adaptation ofregister features.
Setting 2: Reference vs. Machine Translation The results of the second experiment presented in Section 4 have proven our assumption that the overlap between hypothesis and reference translations is higher than between hypothesis translations and comparable non-translated
texts. On the one hand, this corresponds to our intuition in Lapshinova-Koltunski and Vela
(2015), where we show that both human and machine translations do not correspond to comparable German originals, suggesting that both machine and manual translations should havemore in common. In fact, this is also shown by Lembersky et al. (2012), who demonstrate thatthe BLEU score can be improved if they apply language models compiled from translated texts
and not non-translated ones. They also show that language models trained on translated texts fit
better to reference translations in terms of perplexity. In fact, this confirms our claim that machine translations comply more with translated rather than with non-translated texts producedby humans. This results in the improvement of the BLEU score, but not necessary leading to abetter quality of machine translation.
Following the results from both experimental setting, we argue that register features should
be integrated into MT evaluation process, as an additional layer to the already existing automatic
metrics. As future work, we would like to test this hypothesis by combining and correlating theresults presented here with state-of-the-art evaluation metrics.

References
Aha, D. W., Kibler, D., and Albert, M. (1991). Instance-based Learning Algorithms. Machine
Learning, 6(1):37–66.
Baroni, M. and Bernardini, S. (2006). A new Approach to the Study of Translationese:
Machine-learning the Difference between Original and Translated Text. Literary and Linguistic Computing, 21(3):259–274.
6 Here, we mean “shining through” of the source language as defined by Teich (2003).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 224

Biber, D. (1988). Variation Across Speech and Writing. Cambridge University Press, Cambridge.
Biber, D. (1995). Dimensions of Register Variation. A Cross Linguistic Comparison. Cambridge
University Press, Cambridge.
Biber, D., Johansson, S., Leech, G., Conrad, S., and Finegan, E. (1999). Longman Grammar of
Spoken and Written English. Longman, Harlow.
Bojar, O., Buck, C., Callison-Burch, C., Haddow, B., Koehn, P., Monz, C., Post, M., SaintAmand, H., Soricut, R., and Specia, L., editors (2013). Proceedings of the 8th Workshop on
Statistical Machine Translation (WMT). ACL.
Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M., and Specia,
L., editors (2014). Proceedings of the Ninth Workshop on Statistical Machine Translation.
Association for Computational Linguistics, Baltimore, Maryland, USA.
Callison-Burch, C., Osborne, M., and Koehn, P. (2006). Re-Evaluation the Role of Bleu in Machine Translation Research. In Proceedings of the 11th Conference of the European Chapterof the Association for Computational Linguistics, pages 249–256.
Comelles, E. and Atserias, J. (2014). VERTa Participation in the WMT14 Metrics Task. In
Proceedings of the 9th Workshop on Statistical Machine Translation (WMT), pages 368–375,
Baltimore, Maryland, USA. Association for Computational Linguistics.
CWB (2010). The IMS Open Corpus Workbench. accessed February 2015.
De Sutter, G., Delaere, I., and Plevoets, K. (2012). Lexical Lectometry in Corpus-based Translation Studies: Combining Profile-based Correspondence Analysis and Logistic Regression
Modeling. In Oakes, M. P. and Meng, J., editors, Quantitative Methods in Corpus-based
Translation Studies: a Practical Guide to Descriptive Translation Research, volume 51,
pages 325–345. John Benjamins Publishing Company, Amsterdam, The Netherlands.
Denkowski, M. and Lavie, A. (2014). Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the Workshop on Statistical MachineTranslation (WMT).
Diwersy, S., Evert, S., and Neumann, S. (2014). A Semi-supervised Multivariate Approachto the Study of Language Variation. In Szmrecsanyi, B. and Wälchli, B., editors, Linguistic variation in text and speech, within and across languages, pages 174–204. De Gruyter,
Berlin.
Doddington, G. (2002). Automatic Evaluation of Machine Translation Quality Using N-gram
Co-Occurrence Statistics. In Proceedings of the 2nd International Conference on Human
Language Technologies (HLT), pages 138–145.
Eck, M., Vogel, S., and Waibel, A. (2004). Improving Statistical Machine Translation in the
Medical Domain using the Unified Medical Language system. In Proceedings of the 20th
International Conference on Computational Linguistics (COLING-2004), pages 792–798,
Geneva, Switzerland.
Evert, S. (2005). The CQP Query Language Tutorial. IMS Stuttgart. CWB version 2.2.b90.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 225

Gonzàlez, M., Barrón-Cedeño, A., and Màrquez, L. (2014). IPA and STOUT: Leveraging
Linguistic and Source-based Features for Machine Translation Evaluation. In Proceedingsof the 9th Workshop on Statistical Machine Translation (WMT), pages 394–401, Baltimore,
Maryland, USA. Association for Computational Linguistics.
Guerberof, A. (2009). Productivity and Quality in the Post-editing of Outputs from Translation
Memories and Machine Translation. International Journal of Localization, 7(1).
Gupta, R., Orăsan, C., and van Genabith, J. (2015). ReVal: A Simple and Effective MachineTranslation Evaluation Metric Based on Recurrent Neural Networks. In Proceedings of the2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon,
Portugal.
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. (2009). The
WEKA Data Mining Software: An Update. SIGKDD Explor. Newsl., 11(1):10–18.
Halliday, M. (2004). An Introduction to Functional Grammar. Arnold, London.
Halliday, M. and Hasan, R. (1989). Language, Context and Text: Aspects of Language in a
Social-semiotic Perspective. Oxford University Press, Oxford.
Hansen-Schirra, S., Neumann, S., and Steiner, E. (2012). Cross-linguistic Corpora for the Studyof Translations. Insights from the Language Pair English-German. de Gruyter, Berlin, NewYork.
House, J. (2014). Translation Quality Assessment. Past and Present. Routledge.
Irvine, A. and Callison-Burch, C. (2014). Using Comparable Corpora to Adapt MT Models
to New Domains. In Proceedings of the ACL Workshop on Statistical Machine Translation(WMT).
Irvine, A., Morgan, J., Carpuat, M., III, H. D., and Munteanu, D. S. (2013). Measuring MachineTranslation Errors in New Domains. TACL, 1:429–440.
Koehn, P. (2005). Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the 10th Machine Translation Summit, pages 79–86, Phuket, Thailand.
AAMT.
Kruger, H. and van Rooy, B. (2012). Register and the Features of Translated Language. AcrossLanguages and Cultures, 13(1):33–65.
Kurokawa, D., Goutte, C., and Isabelle, P. (2009). Automatic Detection of Translated Text andits Impact on Machine Translation. In Proceedings of MT-Summit XII.
Lapshinova-Koltunski, E. and Pal, S. (2014). Comparabilty of Corpora in Human and MachineTranslation. In Proceedings of the Seventh Workshop on Building and Using Comparable
Corpora, Reykjavik, Iceland. LREC.
Lapshinova-Koltunski, E. and Vela, M. (2015). Measuring ’Registerness’ in Human and Machine Translation: A Text Classification Approach. In Proceedings of the 2nd Workshop on
Discourse in Machine Translation (DiscoMT), pages 000–000, Lisbon, Portugal. Association
for Computational Linguistics. fr.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 226

Laranjeira, B., Moreira, V., Villavicencio, A., Ramisch, C., and Finatto, M. J. (2014). Comparing the Quality of Focused Crawlers and of the Translation Resources Obtained from them.
In Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno,A., Odijk, J., and Piperidis, S., editors, Proceedings of the Ninth International Conferenceon Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. European Language
Resources Association (ELRA).
Lembersky, G., Ordan, N., and Wintner, S. (2012). Language Models for Machine Translation:
Original vs. Translated Texts. Computational Linguistics, 38(4):799–825.
Neumann, S. (2013). Contrastive Register Variation. A Quantitative Approach to the Comparison of English and German. De Gruyter Mouton, Berlin, Boston.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A Method for Automatic
Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318.
Petrenz, P. (2014). Cross-Lingual Genre Classification. PhD thesis, School of Informatics,
University of Edinburgh, Scotland.
Quirk, R., Greenbaum, S., Leech, G., and Svartvik, J. (1985). A Comprehensive Grammar of
the English Language. Longman, London.
Santini, M., Mehler, A., and Sharoff, S. (2010). Riding the rough waves of genre on the web. In
Mehler, A., Sharoff, S., and Santini, M., editors, Genres on the Web: Computational Models
and Empirical Studies, pages 3–30. Springer.
Schmid, H. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. In International Conference on New Methods in Language Processing, pages 44–49, Manchester, UK.
Stanojević, M. and Sima’an, K. (2014). BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414–419, Baltimore,
Maryland, USA. Association for Computational Linguistics.
Steiner, E. (2004). Translated Texts. Properties, Variants, Evaluations. Peter Lang Verlag,
Frankfurt/M.
Systran (2001). Past and Present. Technical report.
Teich, E. (2003). Cross-Linguistic Variation in System und Text. A Methodology for the Investigation of Translations and Comparable Texts. Mouton de Gruyter, Berlin.
Vapnik, V. N. and Chervonenkis, A. J. (1974). Theory of Pattern Recognition. Nauka.
Vela, M. and Hansen-Schirra, S. (2006). The Use of Multi-level Annotation and Alignment for
the Translator. In Proceedings of Translating and the Computer 28 (ASLIB).
Vela, M., Neumann, S., and Hansen-Schirra, S. (2007). Querying Multi-layer Annotation andAlignment in Translation Corpora. In Proceedings of the Corpus Liguistics Conference (CL).
Vela, M., Schumann, A.-K., and Wurm, A. (2014a). Beyond Linguistic Equivalence. An Empirical Study of Translation Evaluation in a Translation Learner Corpus. In Proceedingsof the EACL 2014 Workshop on Humans and Computer-assisted Translation, pages 47–56,
Gothenburg, Sweden. Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 227

Vela, M., Schumann, A.-K., and Wurm, A. (2014b). Human Translation Evaluation and its
Coverage by Automatic Scores. In Proceedings of MTE Workshop at LREC 2014, Reykjavik,
Iceland. European Language Resources Association (ELRA).
Vela, M. and Tan, L. (2015). Predicting Machine Translation Adequacy with Document Embeddings. In Proceedings of the 10th Workshop on Statistical Machine Translation, pages
382–390, Lisboa, Portugal. Association for Computational Linguistics.
Vela, M. and van Genabith, J. (2015). Re-assessing the WMT2013 Human Evaluation withProfessional Translators Trainees. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation (EAMT).
Witten, I. H., Frank, E., and Hall, M. A. (2011). Data Mining: Practical Machine Learning
Tools and Techniques. Morgan Kaufmann, Burlington, Massachusetts.
Wu, H., Wang, H., and Zong, C. (2008). Domain Adaptation for Statistical Machine Translation
with Domain Dictionary and Monolingual Corpora. In Scott, D. and Uszkoreit, H., editors,
Proceedings of the 22nd International Conference on Computational Linguistics (COLING2008), pages 993–1000, Manchester, UK.
Zampieri, M. and Vela, M. (2014). Quantifying the Influence of MT Output in the Translators
Performance: A Case Study in Technical Translation. In Proceedings of the EACL Workshop
on Humans and Computer-assisted Translation (HaCat), pages 93–98.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 228

Error-Tolerant Speech-to-Speech Translation
Rohit Kumar
rkumar@bbn.com
Sanjika Hewavitharana
shewavit@bbn.com
Nina Zinovieva
nzinovie@bbn.com
Matthew E. Roy
mroy@bbn.com
Edward Pattison-Gordon
epgordon@bbn.com
Raytheon BBN Technologies, 10 Moulton Street, Cambridge, MA, USA, 02138

Abstract
Recent efforts to improve two-way speech-to-speech translation (S2S) systems have focused on developing error detection and interactive error recovery capabilities. This article
describes our current work on developing an eyes-free English-Iraqi Arabic S2S system that
detects ASR errors and attempts to resolve them by eliciting user feedback. Here, we report
improvements in performance across multiple system components (ASR, MT and error detection). We also present a controlled evaluation of the S2S system that quantifies the effectof error recovery on user effort and conversational goal achievement.

1. Introduction
Over the past decade, considerable progress has been made in developing usable, two-way
speech-to-speech (S2S) translation systems that enable real time cross-lingual spoken communication [1][2]. Conventionally, S2S systems comprise a pipeline of three speech and language technology components: automatic speech recognition (ASR), machine translation(MT) and text-to-speech synthesis (TTS).
While each of these components technologies have continued to improve in performance,each is data-driven and its performance will degrade when faced with novel vocabulary. For
example, large-vocabulary ASR systems are incapable of recognizing out-of-vocabulary(OOV) words, MT systems cannot translate unseen source words and TTS often mispronounces novel words (often high-value concepts like names and technical terminology). A
combination of these deficiencies can render S2S systems unusable, especially on conversational topics not well represented in the training data.
Different approaches to addressing component failures in the context of S2S systems
have been explored. There have also been attempts at joint optimization of ASR and MT, as
well as MT and TTS [23][24][25]. Most recently, S2S system that actively detect and recoverfrom failures have been built and evaluated [16][26]. In this work, we report our recent efforts
to extend and enhance the usability of S2S translation systems through active error detection
and recovery.
To address system robustness, we have operationalized recent advances in ASR and statistical MT (SMT) into our real-time S2S system. These advances have dramatically improved
ASR and SMT performance, leading to a more robust S2S pipeline, and yet they are fastenough for real-time use. To address the problem of novel vocabulary, we have developed
interfaces that enable non-technical users to rapidly enrich our S2S system with new words
and phrases that are relevant to emerging use cases. Finally, we detect potential ASR errors
before they are amplified by being sent through the SMT and TTS components. A simple in-Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 229

teraction strategy then allows users to correct the translation in the event of an error. Section 3
evaluates an end-to-end S2S system augmented with this capability.

2. BBN’s Speech-to-Speech Translation System
2.1.

Architecture

Figure 1. A unidirectional S2S system pipeline

Figure 2. BBN’s S2S System with SuperMic
Two-way S2S systems comprise a pair of symmetric unidirectional pipelines of ASR, SMT
and TTS components. Each pipeline serves as a communication channel in one direction. Figure 1 depicts one of these pipelines for our S2S system. Spoken input is converted by the ASR
into a word lattice and a 1-best, whole-sentence hypothesis. The SMT produces the target
language translation of the 1-best ASR hypothesis. The ASR error detector (AED) analyzes
the word lattice to identify and rank potentially erroneous spans in the ASR hypothesis. This
analysis, along with the current discourse state, drives a rule-based action selection module,
which decides whether the translation should be transmitted to the listener or a clarification
prompt should be presented to the speaker.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 230

Our English-Iraqi Arabic S2S system uses ASR, AED and SMT components developed at BBN and off-the-shelf English and Iraqi Arabic TTS. The system is deployed on an
off-the-shelf mobile computing device shown in Figure 2. The system is self-contained andruns the entire pipeline fully on this device. To enable eyes-free use, the platform is augmented with a pair of proprietary audio I/O devices, also shown in Figure 2. Each device,referred to as a “SuperMic”, offers a push-to-talk button, a high-quality close-talking microphone and a speaker.
2.2.

System Components

Automatic Speech Recognition: The baseline ASR system is built using data from the
DARPA TransTac English-Iraqi Arabic parallel two-way spoken dialogue collection [3]. It isbased on the BBN Byblos system, which uses a multi-pass decoding strategy where models of
increasing complexity are used in successive passes in order to refine the recognition hypotheses [4]. Speech is modeled as the output of context-dependent phonetic hidden Markov models (HMMs), whose outputs are mixtures of multi-dimensional diagonal Gaussians. Byblos
uses various forms of parameter tying, including state tied mixture (STM) triphone models
and state clustered tied mixture (SCTM) quinphone models. The models were trained on a setof acoustic features, part of which were obtained by using neural networks as describedahead.
Acoustic features: We use neural networks (NNs) to generate stacked bottleneck(SBN) features [5][6]. The SBN structure contains two NNs. The input features of the first
NN are 24 critical-band energies obtained with a Mel filter-bank, with online mean and variance normalization applied. 15 frames of these features are stacked and a Hamming window
multiplies the time evolution of each parameter. Finally, DCT is applied, of which 0th to 15th
coefficients are retained. The first NN consists of two hidden layers, each with 1,500 nodes,
followed by a bottleneck layer. The bottleneck (BN) outputs from the first NN are stacked,
down-sampled, and taken as an input vector for the second NN. This second NN is also a bottleneck layer that is roughly the same size as the first NN. Both NNs were trained to classify
phoneme states (5 states per phoneme). These targets were generated by forced alignment
with baseline, perceptual linear prediction (PLP) models and remained fixed during the training. The final feature stream was built by concatenation of 9 frames of the PLP features together with the bottleneck layer output from the second NN. Finally, region dependent transformation (RDT) [7] is performed to estimate a discriminative feature projection to reduce thedimensionality to 46.
The English acoustic model was trained on approximately 200 hours of transcribed
English speech from TransTac data, and the Iraqi Arabic acoustic model was trained on about
600 hours of transcribed speech from TransTac data. We tested ASR performance on held-out
development sets, consisting of 13,074 and 3,354 utterances for Iraqi Arabic and English,
respectively. Word error rate (WER) on these sets is 16.7% and 7.9%, respectively. For bothlanguages, the WER is better than our previously reported ASR performance in this application [15], attributable to recent implementation of SBN features within Byblos.
Statistical Machine Translation: Our SMT system is based on the state-of-the-art, string-tostring hierarchical decoder described in [8]. In addition to the log-linear combination of generative components such as forward and backward rule probabilities, lexical translation probabilities, n-gram language models, etc., this system further incorporates two multi-layer NN
scoring components, viz. the neural network joint model (NNJM) and the neural network lexical translation model (NNLTM). The NNJM estimates the probability of a hypothesized target word t, conditioned on both the n-1 preceding target words and an m-word source context,

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 231

centered at the source word s that t is affiliated with. The NNLTM estimates the probability
that a source word s translates to a target word t (or NULL if none), given only t’s source context.
These models are trained with one hidden layer, and therefore leverage the robust performance of multi-layer neural networks, but, following [8], we achieve look-up speeds on apar with n-gram language models by precomputing the trained hidden layer (avoiding feedforward computations at run-time) and by training the model to produce approximately normalized output (avoiding costly softmax computations across the whole target vocabulary at
run-time). To further speed up the decoder, we do not perform n-best hypothesis re-ranking,and we also tighten up various parameters in the decoder’s beam search, leading to average
decoding speeds of approximately 41 words/second.
We trained both our English-to-Iraqi Arabic (E2I) and Iraqi Arabic-to-English (I2E)
systems on the Iraqi-English parallel text portions of the DARPA TransTac corpus [3], which
we word-aligned using GIZA++ [9] and extracted hierarchical rules from using the method of[10]. We trained 4-gram Kneser-Ney language models on the respective target side of the
corpus for each system. The NNJM is also limited to a 3-word target history, and both neuralnetwork models have an 11-word source context window. The log-linear combination of allcomponents (including neural network models) is tuned using k-best optimization with an
expected BLEU objective function [11] on held-out development data.
Table 1 shows model performance using the BLEU [12] and translation edit rate (TER)[13] metrics on a held-out blind set of the TransTac corpus. The addition of the neural network models has improved SMT performance significantly in both directions (+3 BLEU for
English-to-Iraqi +2 BLEU for Iraqi-to-English), over our best-performing prior system [17].

E2I
I2E

BLEU↑
19.1
33.6

TER↓
60.1
50.0

Table 1. BLEU and TER scores on held-out TransTac data
ASR Error Detection: The ASR error detector (AED) component performs a word level
analysis of the 1-best ASR hypothesis. We trained a conditional random field (CRF) modelwith the same data used to train our ASR and using automatically generated error annotationby comparing ASR outputs to reference transcriptions. Words in the ASR hypothesis are represented using the following features: (1) ASR confidence; (2) language model perplexity; (3)whether a 1-best word is present in the (pruned) confusion network; (4) density of the corresponding slot in the confusion network; and (5) an indication if the word is commonly misrecognized. The performance of the AED component was evaluated on a collection of utterances designed to be representative of errors encountered by S2S systems [15]. Based on the
ASR WERs reported earlier, we choose operating points corresponding to 1% false alarm atword level for English and 2% false alarm for Iraqi Arabic. The corresponding detection rates
for AED were 36.4% and 19.8% respectively. These significant improvements over our previous operationalized AED implementation [21] are attributable to both improvements in
ASR but also use of new features and classifier formulation for AED [14].
Word-level AED analysis is aggregated to identify error spans (contiguous spans ofdetected word-level errors). To prevent very large spans from being identified, we apply aheuristic that temporarily increases the operating threshold of the AED to localize only thehighest confidence sub-span within large error spans. The results of ASR error detection areused to select an appropriate system action.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 232

Note that the AED approach employed in this work only uses resources available fortraining and evaluation of the ASR which makes this approach extensible to new languages.
Furthermore, features used by AED are based on rich information produced by the ASR. The
need for tighter integration of the ASR error detection capability with the ASR makes a casefor implementation of AED as a commonly available module within modern ASRs.
Interactive Error Recovery: Error recovery is presented as an optional module which users
can choose to enable. When error recovery is enabled, action selection filters out very small
error spans (<0.25 seconds long) identified by the AED analysis and ranks the spans by the
maximum error confidence of their constituent tokens. Furthermore, action selection is influenced by the discourse state. For example, if three consecutive attempts fail to resolve an error, action selection by-passes error recovery. Table 2 shows three excerpts of an English
speaker using the S2S system to communicate with an Iraqi Arabic speaker. Figure 3 summarizes the dialog model of error recovery.
Excerpt I. No error is detected
Hello, My name is Sergeant Jones. I came here to check on a woman that
Speakercame in here yesterday due to bleeding after childbirth.
hello my name is sergeant jones i came here to check on a woman that came
ASR
in here yesterday due to bleeding after childbirth‫مرحبا آني اسمي العريف جونز آني إجيت هنا حتى أتشيك على المرة اللي إجا هنا البارحة بسبب‬Translation‫النزيف بعد والدة‬Excerpt II. Error is detected (True Detection)
Speaker I'm doing good. My name is Sergeant Edwards. How are you doing today?
ASR i'm doing good my name is sergeant at words how are you doing today
i'm doing good my name is sergeant at words how are you doing today. May
Clarification not be able to translate [Edwards]. Is that a name? Say Yes. Otherwise, say
Go Ahead.
Speaker Yes
ASR yes
Translation‫[ شلونك اليوم‬Edwards] ‫آني زين اسمي العريف‬Excerpt II. Error is detected (False Alarm)Y e a h about that; I'm gonna have to talk to my superiors and get back to
Speakeryou.
ASR yeah about that i'm gonna have to talk to my superiors and get back to you
yeah about that i'm gonna have to talk to my superiors and get back to you.
Clarification May not be able to translate [Y e a h]. Is that a name? Say Yes. Otherwise,
say Go Ahead.
Speaker Go Ahead
ASR go ahead
Translation‫إي هاي آني رح أحكي ويا المسؤولين مالتي وأرجع لك‬
Table 2. Excerpts of interaction with the English-Iraqi Arabic S2S System
Excerpt I in Table 2 shows the S2S system behavior when no error is detected in theuser input (HT). In this case, a translation is presented to the listener (ST). Excerpts II and III
shows the case where the AED finds an error (a true detection in II and a false alarm in III). In
each case, the speaker is asked to confirm (SC) if the audio corresponding to the top ranked
error span corresponds to a name (or a concept that could be transferred as-is to the other lan-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 233

guage). The user can respond (HR) by saying “yes” to confirm the accurate error detection
and localization as shown in excerpt II, or skip the system’s attempt to recover from a potential error by issuing the “go ahead” command. If error detection is confirmed, the audio segment corresponding to the error span is spliced into the translation through the source-targetalignment produced by SMT. In addition to commands shown in excerpt II and III, the user
can also choose to rephrase the input which is analyzed as a new input.

Figure 3. Dialog model of error recovery in S2S system
System behaviors in terms of action selection and prompt generation are symmetric inboth directions (i.e. E2I and I2E). We note that, given the user-mediated nature of this error
recovery strategy, successful cross-lingual concept transfer depends not only on component
performance, but also on the appropriateness of users’ responses to clarifications

3. Evaluation
3.1.

Experiment Design & Data Collection

DARPA Broad Operational Language Technologies (BOLT) is a three-phase research focused on advancing the state of the art in translation technologies. Under the third phase ofthis program, an evaluation of error tolerant S2S systems was conducted by NIST over 5 daysin January 2015.
Nine speaker pairs, each comprised of one native English speaker and one native Iraqi
Arabic speaker, were tasked with communicating with one another using only a S2S system.
The speakers were placed in two rooms separated by a see-through glass wall that served as asound barrier. Each speaker pair interacted with each system over two 120-minute-long sessions. The S2S system was configured to perform ASR error detection in only one of the twosessions. The ordering of the two sessions (with and without error detection) was balanced
across speaker pairs.
Each speaker pair was provided 16 conversational scenarios to be accomplished ineach session. The scenarios assigned one of the two speakers a conversational driver role andthe other a respondent role. The driver of the scenario was provided a conversational objective, such as ascertaining the extent of damage due to a hypothetical natural disaster. Speakers

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 234

were asked to achieve the conversational objective for each scenario in less than 8 minutes,
and then to move on to the next scenario until either all scenarios were completed or the 120
minutes had expired (whichever came first). The scenarios cover a range of conversational
domains relevant to military and humanitarian operations as well as everyday conversational
topics like sports, family and pets.
Before using the S2S system, speakers received 30 minutes of training which included
a 4-minute-long video demonstrating SuperMic features and the error recovery functionality,
as well as several minutes of free-form interaction for additional practice.
Each conversational trial collected was scored by 6 independent judges along threemetrics. These metrics, listed below, used a 7 point scale ranging from -3 (strongly disagree)
to +3 (strongly agree).


Goal: The initiator of the dialog achieved his/her goal.

Quality: The overall quality of the translations was adequate.

Clarification: The clarification(s) helped during the dialog.

Note that the clarification scale is applicable only when the S2S system was configured
to perform error recovery. In this paper, we will report average computed over the multiple
judgements.
3.2.

Results

Clarification
ON#Trials ↑113
Avg. Trial Duration ↓396.2
Turns per Trial (En) ↓9.2
Turns per Trial (IA) ↓9.7#Clarifications (En)
191
#Clarifications (IA)229
ASR Performance
WER (En) ↓5.7
WER (IA) ↓6.8
OOV Rate (En) ↓0.9
OOV Rate (IA) ↓1.8
ASR Error Detector Performance
Recall (En) ↑30.4
Recall (IA) ↑24.0
SMT PerformanceTER (E2I) ↓58.9
TER (I2E) ↓49.8
UNK rate (En) ↓0.57
UNK rate (IA) ↓
0.42

OFF
125
378.2
8.0
8.5
5.6
7.0
0.8
2.2
58.5
51.7
0.44
0.44

Table 3. Statistics of trials collected at BOLT evaluation of
BBN’s S2S system & Component performance metrics
We logged 238 conversational trials over 18 sessions of use of our S2S system. S2S system
configured for ASR error detection and recovery was used in 9 of these sessions during which

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 235

113 trials were logged. Table 3 provides summary statistics about these trials and reports theperformance of key system components (ASR, AED, SMT) in these trials both with and without error recovery (Clarification ON or OFF).
ASR performance, reported as WER and OOV rate, is consistent across the two typesof sessions. The WER is significantly lower than on the development sets, especially for Iraqi
Arabic (IA). This is likely because the development set was collected in a human-mediatedcross-lingual communication setting unlike the computer-mediated setting in which the S2S
system is used. AED performance is reported as recall at the operating points mentioned in
section 2.2.3. Despite a more conservative choice of operating point, AED is able to accurately detect a larger fraction of erroneous words for English. SMT performance is measured bothas TER on 1-best ASR hypothesis and as the rate of untranslatable input words (UNKs). TER
is comparable to development sets. UNK rate for English is lower than Iraqi Arabic.
While error recovery appears to not harm ASR and MT performance, we note that itachieves lower conversational throughput. Speakers have 10% fewer conversations when error recovery is enabled, and the average duration of a conversational trial increased by 5%.
User effort, quantified as the number of turns spoken per trial, increases by 15% with error
recovery dialogs. Also, we found that 9% of the English error clarification sub-dialogs required more than one turn (7% for Iraqi Arabic). This is an improvement over previously reported measures of error recovery cost [16], where it was found that erroneous inputs consumed 1.4 clarification turns.

Correlations
Clarification
Words per turnOOV Rate↓
WER↓
AED Recall ↑UNK Rate↓
TER↓

English
ON
OFF
-0.32
-0.35
0.73
0.05
0.31
-0.59
-0.06
0.31
-0.06
-0.02
0.05

Iraqi Arabic
ON
OFF
-0.30
-0.45
-0.28
-0.14
0.33
0.12
-0.19
0.42
0.05
0.17
-0.20

Table 4. Correlations between metrics and user effort across different speaker pairs
We found significant variation in all of the metrics reported in Table 3 across different
speaker pairs and across different domains covered in the scenarios. The standard deviation
for user effort across different speaker pairs (s.d.=2.89) was higher than across different domains (s.d.=1.26). Table 4 characterizes the effect of different system components on user
effort by reporting Pearson correlations of various metrics with user effort. The correlations
are computed over the variations among the speakers.
Without error recovery, higher WER corresponds to fewer turns taken by the English
speaker. This is likely to be indicative of fewer concepts discussed by the speaker. More English OOVs lead to increased effort when error recovery is enabled. However, use of OOVs by
the Iraqi speaker does not increase the number of turns. We found that the correlation between
OOV rate and AED Recall for English is positive (r=0.18), but its negative for Iraqi Arabic(r=-0.23). This is due to the difference in AED performance between the two languages. For
both the speakers, more untranslatable tokens lead to an increase in effort when error recovery
is enabled.
Finally, we examine the human judgements of the conversational trails. Averages forthe three scales are shown in Table 5. The use of error recovery does not improve goal
achievement or quality of translation. However, a closer examination of the scenarios that

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 236

were trialed with both error recovery enabled and disabled (N=112) shows that for difficult
scenarios, error recovery infact improves conversational goal achievement. Difficult scenarios
are defined as subset of scenarios that had below average goal achievement score in the baseline system (i.e. without error recovery).

Clarification
Goal ↑
Quality ↑Clarification ↑#Difficult Trials ↑Goal (Difficult Trials) ↑Goal (Easy Trials) ↑

ON
2.27
1.79
1.70
38
2.10
2.47

OFF
2.40
1.89
74
2.00
2.73

Table 5. Judgement scores for BBN’s S2S systems
Intuitively, this can be interpreted as indication that error recovery is only helpfulwhen the users are having a conversation that the S2S system has difficulty with. For easy
scenarios, error recovery does not help. One of the immediate design implications of this for
S2S systems is to empower the users with the ability to turn error recovery on only whenneeded. However, user enabled error recovery assumes that the users are able to judge when
the S2S system is having difficulty with a conversation.

4. Conclusion
Application of interactive error recovery have been investigated for multiple spoken language
technologies including spoken dialog systems [18][19] and S2S systems [16][20]. Our priorwork attempted to address various types of errors encountered due to imperfections of S2S
system components. In contrast to that, the work presented in this paper builds on recent improvements in component performance and focuses only on errors introduced by misrecognition of input speech. This leads to simplification of interaction design as well as reduction in
the cost of error recovery, quantified here in terms of user effort and time. Extending the needto minimize the cost of error recovery, we have principally chosen conservative false-alarm
rate for AED based on language specific WERs. While in the work presented here we focus
on an eyes-free use case, another configuration of BBN’s S2S system employs a touch screeninterface along with visual cues to resolve errors without requiring spoken commands.

References[1]

S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J. S. Zhang, H. Yamamoto, E. Sumita and S. Yamamoto, "The ATR Multilingual Speech-to-Speech Translation System",IEEE Trans. on Audio, Speech, and Language Processing, 14(2), pp. 365- 376, 2006.

[2]

D. Stallard, R. Prasad, P. Natarajan, F. Choi, S. Saleem, R. Meermeier, K. Krstovski, S. Ananthakrishnan and J. Devlin, “The BBN TransTalk Speech-to-Speech Translation System”, Speech andLanguage Technologies, InTech, (Ed. Ivo Ipsic), pp. 31-52, 2011.

[3]

F. Choi, S. Tsakalidis, S. Saleem, C. L. Kao, R. Meermeier, K. Krstovski, C. Moran, K. Subramanian, D. Stallard, R. Prasad, P. Natarajan, “Recent Improvements in BBN’s English/Iraqi Speech-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 237

to-Speech Translation System”, IEEE Spoken Language Technology Workshop, pp. 245–248,
2008.
[4]

L. Nguyen and R. Schwartz, “Efficient 2-pass n-best Decoder”, DARPA Speech Recognition
Workshop, pp. 167–170, 1997.

[5]

S. Tsakalidis, R. Hsiao, D. Karakos, T. Ng, S. Ranjan, G. Saikumar, L. Zhang, L. Nguyen, R.
Schwartz and J. Makhoul, “The 2013 BBN Vietnamese Telephone Speech Keyword Spotting System”, IEEE International Conference on Acoustic, Speech and Signal Processing, pp. 7829-7833,
2014.

[6]

M. Karafiat, F. Grezl, M. Hannemann, K. Vesely, and J. H. Cernocky, “BUT Babel system for
spontaneous Cantonese”, Interspeech, pp. 2589-2593, 2013.

[7]

T. Ng, B. Zhang, S. Matsoukas, and L. Nguyen, “Region Dependent Transform on MLP Features
for Speech Recognition”, Interspeech, pp. 221–224, 2011.

[8]

J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz and J. Makhoul. “Fast and Robust Neural
Network Joint Models for Statistical Machine Translation”, Proc. of the Association for Computational Linguistics, pp. 1370-1380, 2014.

[9]

F. J. Och and H. Ney. “A Systematic Comparison of Various Statistical Alignment Models”,
Computational Linguistics, 29(1). pp. 19-51, 2003.

[10] D. Chiang. “Hierarchical Phrase-based Translation”, Computational Linguistics, 33(2), pp. 201228, 2007[11] A. Rosti, B. Zhang, S. Matsoukas, and R. Schwartz. “BBN System Description for WMT10 System Combination Task”, Proc. of WMT/MetricsMATR, pp. 321-326, 2010[12] K. Papineni, S. Roukos, T. Ward and W.-J. Zhu. “Bleu: a Method for Automatic Evaluation of
Machine Translation”, Proc. of the Association for Computational Linguistics, pp. 311-318, 2002
[13] M. Snover, B. Dorr, R. Schwartz, L. Micciulla and J. Makhoul, “A Study of Translation Edit Ratewith Targeted Human Annotation”, Proc. of the Association for Machine Translation in the Americas, pp. 223-231, 2006[15] Y.-C. Tam, Y. Lei, J. Zheng, and W. Wang, “ASR Error Detection Using Recurrent Neural Network Language Model and Complementary ASR,” Proc. of ICASSP, pp. 2331-2335, 2014[16] R. Kumar, M. Roy, S. Ananthakrishnan, S. Hewavitharana and F. Choi. “Interactive Error Resolution Strategies for Speech-to-Speech Translation Systems”, Proc. of the SIGDIAL, pp. 142-144,
2013.
[17] S. Ananthakrishnan, D. Mehay, S. Hewavitharana, R. Kumar, M. Roy and E. Kan. “LightlySupervised Word Sense Translation Error Detection and Resolution in an Interactive Conversational Spoken Language Translation System,” Machine Translation, Kluwer Academic Publishers,
vol. 29, no. 1, pp. 25-47, 2015.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 238

[18] M. Turunen, and J. Hakulinen, “Agent-based Error Handling in Spoken Dialogue Systems”, Proc.
of Eurospeech, pp. 2189–2192, 2001[19] D. Bohus, and A. I. Rudnicky, "Sorry, i didn’t catch that!- an investigation of non-understanding
errors and recovery strategies", Proc. of SIGDIAL, pp. 128-143, 2005[20] E. Pincus, S. Stoyanchev and J. Hirschberg, “Exploring Features For Localized Detection of
Speech Recognition Errors,” Proc. of SIGDIAL, pp. 132—136, 2013[21] W. Chen, S. Ananthakrishnan, R. Kumar, R. Prasad, and P. Natarajan, “Automated ASR error
detection in a conversational spoken language translation system”. Proc. of Intl. Conf. on Acoustics, Speech and Signal Processing, pp. 7418-7422, 2013.
[22] Zhang, R., Kikui, G., Yamamoto, H., Watanabe, T., Soong, F., and Lo, W. K. “A unified approachin speech-to-speech translation: integrating features of speech recognition and machine translation”, Proc. of 20th COLING, Stroudsburg, PA, USA, 2004[23] He, X. and Deng, L. “Optimization in Speech-Centric Information Processing: Criteria and techniques”, Proc. of ICASSP, 2012, p. 5241-5244[24] Matsoukas, S., Bulyko, I., Xiang, B., Nguyen, K., Schwartz, R. and Makhoul, J. "Integrating
Speech Recognition and Machine Translation," Proc. of ICASSP, 2007, p. 1281- 1284
[25] Suhm, B., Myers, B. and Waibel, A. "Interactive recovery from speech recognition errors in speechuser interfaces," Proc. of 4th ICSLP, 1996. p.865-868[26] N. F. Ayan, A. Mandal, M. Frandsen, J. Zheng, A. Kathol, F. Bechet, B. Favre, A. Marin, T.
Kwiatkowski, M. Ostendorf, L. Zettlemoyer, P. Salletmayr, J. Hirschberg, and S. Stoyanchev,“Can you give me another word for hyperbaric?: Improving speech translation using targetedclarification questions,” Proc. of ICASSP, 2013

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 239

Mixed-Domain vs. Multi-Domain
Statistical Machine Translation
Matthias Huck
Alexandra Birch
Barry Haddow

mhuck@inf.ed.ac.uk
a.birch@ed.ac.uk
bhaddow@inf.ed.ac.uk
School of Informatics, University of Edinburgh, 10 Crichton Street, Edinburgh EH8 9AB, UK

Abstract
Domain adaptation boosts translation quality on in-domain data, but translation quality fordomain adapted systems on out-of-domain data tends to suffer. Users of web-based translation
services expect high quality translation across a wide range of diverse domains, and what makes
the task even more difficult is that no domain label is provided with the translation request.
In this paper we present an approach to domain adaptation which results in large-scale, general
purpose machine translation systems. First, we tune our translation models to multiple individual domains. Then, by means of source-side domain classification, we are able to predict thedomain of individual input sentences and thereby select the appropriate domain-specific model
parameters. We call this approach multi-domain translation.
We develop state-of-the-art, domain-adapted translation engines for three broadly-defined domains: TED talks, Europarl, and News. Our results suggest that multi-domain translation
performs better than a mixed-domain approach, which deploys a system that has been tuned ona development set composed of samples from many domains.

1

Introduction

Domain adaptation is a common approach to significantly improve machine translation quality
on input documents from a given domain. Domain adaptation techniques for statistical machine translation (SMT) have been extensively studied and are well established (Federico andBertoldi, 2012). In practice, machine translation systems are often engineered to perform wellon the domain of one specific application. Most research on domain adaptation assumes thatany prospective input data originates from a single domain, and the characteristics of this domain are known beforehand, e.g. by means of existing samples from the same domain which
can be employed for training and tuning. The adaptation task is then defined as utilizing a small
amount of in-domain training resources effectively in order to learn system parameters that are
more appropriate for translating in-domain input. The in-domain training resources constitutea minor fraction of the overall training data only, the majority of which has a domain mismatchwith the designated application.
The downside of systems that have been highly tweaked towards the characteristics of asingle domain is a diminished translation quality on out-of-domain data (Haddow and Koehn,2012). Online translation systems, on the other hand, are usually designed for open-domain
scenarios where the domain of the input text is not predefined. Being able to take advantageof the benefits of domain adaptation while not having to compromise quality on out-of-domain
data would be desirable for online systems.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 240

A viable utilization of domain adaptation approaches in open-domain online translation
systems comes in two components:• A number of different parameter sets, each tuned to optimize translation quality on texts
from a specific domain.
• A text classifier that predicts the domain of foreign-language input data prior to decoding.
As the input data is not labeled with a domain (and, in an open-domain setting, may even
originate from a new domain), the text classifier first has to assign the most likely class froma set of multiple known domains. The decoder is then reconfigured with a domain-specificparameter set (e.g., a weight vector), which should be the most appropriate one for achieving
high translation quality on the current input. We refer to this approach as multi-domain SMT.
In this work we investigate different methods for domain classification:• Classification based on the scores of language models (LMs) which have been interpolatedwith interpolation weights tuned on in-domain development sets.
• Maximum entropy text classifiers trained on medium-sized training corpora.
• Maximum entropy text classifiers trained on the same smaller domain-specific development sets which are employed for tuning the machine translation systems.
An obvious alternative method for building an open-domain online translation system istuning on a corpus containing samples of texts from all known domains, which collectively
are considered to be representative for the application. We refer to this approach as mixeddomain SMT. A difficulty here is the choice of the corpus samples in a way that brings about
good performance across all domains. However, a high-quality generic system with a single
parameter set that does not depend on a domain label is appealing.
In the empirical part of this paper, we compare multi-domain and mixed-domain SMT on
the English→German, English→Italian, English→Portuguese, and English→Greek language
pairs using training corpora of diverse origin, totalling tens of millions of parallel sentences.

2

Related WorkA significant amount of research on domain adaptation for SMT has been conducted in recent
years. Some methods which are commonly used are:• Tuning of the decoder model weights (Och and Ney, 2002) on an in-domain development
set (Pecina et al., 2012).
• Model combination (of language models, translation models, or reordering models) via
interpolation or other schemes, e.g. phrase table fill-up (Foster and Kuhn, 2007; Koehn
and Schroeder, 2007; Nakov, 2008; Bisazza et al., 2011; Niehues and Waibel, 2012; Chenet al., 2013).
• Data selection (Moore and Lewis, 2010; Axelrod et al., 2011).
• Instance weighting (Matsoukas et al., 2009; Foster et al., 2010; Shah et al., 2012; Mansour
and Ney, 2012).
• Further exploitation of in-domain monolingual data (Ueffing et al., 2007; Bertoldi andFederico, 2009; Schwenk and Senellart, 2009; Lambert et al., 2011).
• Domain-specific features, e.g. binary features indicating the provenance of phrase pairs as
implemented in the open-source Moses toolkit (Durrani et al., 2013b) or “domain augmentation” (Clark et al., 2012).
However, few authors have tackled the question of how to benefit from domain adaptationin scenarios where a domain label of the input is not present. An important aspect of ourapproach to multi-domain MT is the need for domain classification.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 241

Xu et al. (2007) perform domain classification for a Chinese→English task. The domains
are newswire and newsgroup. The classifiers operate on whole documents rather than on individual sentences. The authors propose two techniques for domain classification. Their first
technique is based on interpolated LMs: a general-domain LM is interpolated with LMs whichwere trained on in-domain development sets, resulting in a number of domain-specific interpolated LMs. The interpolation weight is heuristically chosen. The classifier computes LM
perplexities over input documents and assigns the domain with the lowest perplexity. Theirsecond technique is based on a metric which measures similarity wrt. vocabulary.
Banerjee et al. (2010) conduct machine translation experiments with classification of two
technical documentation domains, availability and security in the area of computing. Empiricalresults on Chinese→English and English→Chinese tasks are presented. The authors build a
Support Vector Machine (SVM) classifier using Term Frequency Inverse Sentence Frequency
features over bigrams of stemmed content words. Classification is carried out on the level ofindividual sentences. The SVM is trained on the SMT training corpora (∼226k sentences intotal). Several setups with different domain-adapted and domain-agnostic systems are evaluated. The authors show that a pipeline with the SVM classifier is effective in multi-domaintranslation.
Wang et al. (2012) distinguish generic and patent domain data in experiments on 20 language pairs. For domain classification, the authors rely on averaged perceptron classifiers withvarious phrase-based features. The machine translation development sets serve as training data
for the classifiers. An interesting aspect of their translation experiments is that they utilize amulti-domain optimization in order to jointly tune weights for all domains in a single run oflattice MERT (Macherey et al., 2008).
In a related strand of research, source-side text classifiers have recently been employed
in order to detect Arabic dialects and select SMT systems accordingly (Salloum et al., 2014;
Mansour et al., 2014).

3

Text Domains

Our application scenario is an online translation service with the requirement to provide highquality translation not only of texts from a single domain, but of a wider range of text types.
We therefore study a use case where the translation system is supposed to perform well on thefollowing domains: TED talks, Europarl, and News. These three domains are fairly coarsegrained. Different documents from one of the domains are mostly not consistent regarding thecovered topics. While all three domains comprise heterogeneous topics, the domains are set
apart from each other by means of text style.
TED talks are transcripts of spoken language from short public presentations. The presentations often cover scientific subjects which are expressed in layman’s terms and in an informalmanner. TED talks are not spontaneous speech. They are however designed to be entertaining.
Europarl texts are transcripts of speeches on political matters from parliamentary proceedings.
News texts are written news articles.
TED talks, Europarl, and News could be described as “genres”. We denote them as domains throughout this paper because the term “domain” is well established in related machinetranslation research literature and often used in a broad sense.
TED talks, Europarl, and News have been highly relevant domains in recent machine translation research. The International Workshop on Spoken Language Translation1 (IWSLT) hosts
a yearly open evaluation campaign which focuses on the translation of TED talks since 2011
(Federico et al., 2011). The European Parliament Proceedings Parallel Corpus (Koehn, 2005)
has been an influential resource for machine translation research ever since its first release over
1 http://www.iwslt.org

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 242

a decade ago. It is freely available and includes parallel text for 21 European languages. Test
sets and training data that enables research on machine translation of texts from the News domain have regularly been released for the shared translation task of the Workshop on Statistical
Machine Translation2 (WMT). The WMT “newstest” corpora have become important test sets
to measure progress in machine translation between English and several European languages.
Previous research has indicated that divergences of domains such as TED talks, Europarl,and News empirically matter for machine translation. For instance, Ruiz and Federico (2014)
systematically analyze characteristics of TED talk transcripts and News Commentary texts andpoint out the differences in detail for English-German.

4

Adaptation Techniques

In order to adapt systems to a domain, we leverage previously proposed techniques. First,
we tune the model weights on in-domain development sets. Secondly, we linearly interpolate
language models: rather than training a single large LM on all the target-side data, we train
separate models on each corpus and interpolate them based on weights that minimize perplexity
over the development set, resulting in a new, domain-adapted large LM that can be used by
the decoder. Finally, we add binary features indicating the provenance of phrase pairs: if aphrase pair has been seen in a particular training corpus, a binary indicator associated withthe respective training corpus fires on application of that phrase pair during decoding. This
increases the amount of features by a number equal to the number of parallel training corpora.

5

Domain Classification

A domain classifier is required for multi-domain SMT on unlabeled input data. The domain of
the source-side text we receive for translation is unknown and we need to predict it in order toselect appropriate decoder parameters.
We investigate classification based on source-side LMs as well as different variations of a
maximum entropy classifier.
Unlike Xu et al. (2007), who classify documents, we predict the domain label on the levelof single sentences. Sentence-level classification has the advantage that document boundaries
do not need to be present, and we are able to decode an unstructured incoming stream of sentences.
5.1

Source LM Classifier

Classification based on source-side LMs predicts the domain label from LM scores. We train
separate LMs on the source side of each parallel training corpus. Then we create adapted LMs
for each domain by linearly interpolating those source language LMs, where the interpolation
weights are tuned to minimize perplexity on the source side of the respective in-domain development set. The classifier computes LM scores with each of the domain-adapted source LMs
and selects the domain label according to maximum score. Note that for the scores to be on acomparable level, all domain-adapted source LMs should be interpolations of the same set ofindividual LMs.
Besides classifying sentences rather than documents, our method differs from the one proposed in (Xu et al., 2007) with respect to another aspect: Xu et al. (2007) interpolate LMs
trained on the respective in-domain development sets with a single huge generic LM. Disadvantages of their method are (1.) the tiny size of the development corpora in terms of LM training,and (2.) the necessity of setting the interpolation weights heuristically. We overcome thesedrawbacks by resorting to a more straightforward framework of reserving the in-domain devel2 http://www.statmt.org/wmt15/translation-task.html

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 243

opment sets for tuning source LM interpolation weights. Furthermore, we argue that source-side
scores of the interpolated LMs employed in our classifier may to some extent resemble targetside scores of the interpolated LMs which are applied in the respective domain-adapted SMT
systems.
5.2

Maximum Entropy Classifiers

Maximum entropy text classifiers can utilize a larger number of features in order to predict thelabel (Berger et al., 1996). We incorporate features from single words, pairs of adjacent words,
the first word of the sentence, and the last word of the sentence. The model is trained with
L-BFGS (Byrd et al., 1995) and regularized using a Gaussian prior.
We build maximum entropy (ME) classifiers under two different training conditions: using
the MT development sets (which are rather small) as training data, and using selected othercorpora as training data (which might not always exactly match what is defined as in-domain to
the MT systems, as the development sets essentially constitute the domains).
In a further flavor of our ME classifiers, in addition to the previously described features, we
include source LM indicator features in the ME model. To create these features, we score thesentence with the same domain-adapted source LMs as employed by the source LM classifier.
For each of the domain-adapted LMs, an associated source LM indicator feature fires if therespective LM yields the maximum LM score.
Overall, we end up with four variations:
MEtrain Classifier trained on medium-sized training corpora with the basic set of features.
MEtrain+lm Classifier trained on medium-sized training corpora with the basic set of features
plus source LM indicator features.
MEdev Classifier trained on the MT development sets with the basic set of features.
MEdev+lm Classifier trained on the MT development sets with the basic set of features plus
source LM indicator features.

6

Experimental Setup

We use Moses (Koehn et al., 2007) for machine translation, MGIZA++ (Gao and Vogel, 2008)
to train word alignments, KenLM (Heafield, 2011) for LM training and scoring, SRILM(Stolcke, 2002) for LM interpolation, and the Stanford Classifier3 for maximum entropy textclassification. We present experimental results on English→German, English→Italian, English→Portuguese, and English→Greek translation tasks.
6.1

Training Data

Our SMT systems are trained with the following bilingual corpora:• TED from WIT3 (Cettolo et al., 2012)• Europarl (Koehn, 2005)• JRC-Acquis 3.0 (Steinberger et al., 2006)• DGT’s Translation Memory (Steinberger et al., 2012) as distributed in OPUS (Tiedemann,
2012)• OPUS European Central Bank (ECB)• OPUS European Medicines Agency (EMEA)• OPUS EU Bookshop• OPUS OpenSubtitles4
3 http://nlp.stanford.edu/software/classifier.shtml
4 http://www.opensubtitles.org

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 244

Parallel training corpus
sentences
source running words
source vocabulary
target running words
target vocabulary

En→De
23.97 M
464.82 M
2.21 M
421.67 M
3.81 M

En→It
29.55 M
445.97 M
1.54 M444.26 M
1.62 M

En→Pt
32.15 M
427.53 M
1.32 M
424.03 M
1.39 M

En→El
30.79 M
417.45 M
1.33 M
367.84 M
2.19 M

Table 1: Statistics of the overall parallel training data after preprocessing.
Monolingual training corpus
Wikipedia
sentences
running words
vocabularyNews
sentences
running words
vocabularyDe
35.80 M695.84 M
6.50 M
159.66 M
2940.44 M
8.83 M

It
15.37 M
387.51 M
2.42 M———Pt
9.27 M
216.71 M
1.82 M———El
1.98 M
44.58 M
0.93 M———

Table 2: Statistics of additional monolingual training data after preprocessing.
• WMT News Commentary• WMT CommonCrawl• SETimes (Tyers and Alperen, 2010)
Statistics of a concatenation of all bilingual training corpora are presented in Table 1.
For language modeling on the target side, we furthermore add monolingual corpora fromrecent (April 2015) Wikipedia database dumps5 and—for German—the News Crawl corpora
provided for the WMT 2015 shared translation task. Plain text was obtained from the Wikipedia
XML dumps with the Wikipedia Extractor6 tool. Statistics of the additional monolingual training corpora are presented in Table 2.
6.2

Machine Translation Systems

Word alignments are created by aligning the data in both directions and symmetrizing the two
trained alignments (Och and Ney, 2003; Koehn et al., 2003). We extract phrases up to a maximum length of five. The MT systems comprise these features:• Phrase translation log-probabilities, smoothed with Good-Turing smoothing (Foster et al.,2006), and lexical translation log-probabilities in both directions.
• Phrase penalty and word penalty.
• Distance-based distortion cost.
• A hierarchical lexicalized reordering model (Galley and Manning, 2008).
• A 5-gram operation sequence model (Durrani et al., 2013a).
• Seven binary features indicating absolute occurrence count classes of phrase pairs.
• Sparse phrase length features.
• Sparse lexical features for the top 200 words.
• A 5-gram LM with modified Kneser-Ney smoothing (Kneser and Ney, 1995; Chen andGoodman, 1998). We discard singleton n-grams of order three and higher.
Feature weights are optimized to maximize B LEU (Papineni et al., 2002) with batch MIRA
(Cherry and Foster, 2012) on 1000-best lists. We prune the phrase table to a maximum of 100
5 http://dumps.wikimedia.org
6 https://github.com/bwbaugh/wikipedia-extractor

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 245

best translation options per distinct source side and apply a minimum score threshold of 0.0001
on the source-to-target phrase translation probability. We use cube pruning in decoding. Pop
limit and stack limit are set to 1000 for tuning and to 5000 for testing. We disallow reordering over punctuation. Furthermore, Minimum Bayes Risk decoding is employed for testing.
Translation quality is measured in truecase with B LEU.
6.2.1 Development and Test Sets
Wherever possible, we tune and test on common sets as distributed on http://matrix.
statmt.org/test_sets/list/ and https://wit3.fbk.eu/.
The domainadapted Europarl systems are tuned on the test2006 set. The domain-adapted TED systems
are tuned on a concatenation of TED-dev2010 and TEDX-dev2012 (En→De), on a concatenation of TED-dev2010 and TEDX-dev2014 (En→It), and on TED-dev2010 (En→Pt).
An English→Greek translation task was so far never organized as part of any of the IWSLT
evaluation campaigns and for that reason no common TED sets exist for that language pair.
However, the 2012-02 release of the WIT3 corpus contains a parallel corpus of 84 831 EnglishGreek sentence pairs. We reserved every fifth sentence of this data for tuning and testing. Of
the tuning and testing part of the corpus, we assign every fourth sentence to the test set and therest to the tuning set.7 The domain-adapted News systems are tuned on a concatenation of thenewstest2008-2012 sets (En→De) and on newstest2009 (En→It). We use newssyscomb2009 as an English→Italian News domain test set for lack of other English-ItalianNews test sets. Note that newssyscomb2009 is a small set of only 502 sentences. No News
test data was available to us for the English→Portuguese and English→Greek language pairs,
so we experiment with only two domains (TED and Europarl) on these tasks.
The Portuguese TED development and test sets are Brazilian Portuguese whereas the Europarl sets are European Portuguese. The two Portuguese dialects have a number of differencesin written language. Marujo et al. (2011) give a brief overview.
6.2.2 Domain-Adapted SMT
For our domain adaptation experiments, we first tune the systems with the features describedabove on the respective in-domain development set (TED-tuned, Europarl-tuned, News-tuned).
We next replace the large baseline LM with a domain-specific interpolated LM (+ LM interp.).
We then add binary features indicating the provenance of phrase pairs (+ LM interp. + indicatorfeat.).
6.2.3 Mixed-Domain SMT
We build mixed-domain SMT systems by tuning on a development corpus containing samples oftexts from all domains. We include a balanced amount of development data from the different
domains in the mixed-domain development set in order to avoid a bias towards any specificdomain.
The mixed-domain systems (Mixed-domain-tuned) are tuned on a concatenation of
TED-dev2010 and TEDX-dev2012 and Europarl test2006 and newstest2009(En→De), on a concatenation of TED-dev2010 and TEDX-dev2014 and Europarl
test2006 and newstest2009 (En→It), on a concatenation of TED-dev2010 and every second sentence from Europarl test2006 (En→Pt), and on a concatenation of every sixth
sentence from our Greek TED development set and the full Europarl test2006 (En→El).
The LMs for the mixed-domain systems are trained on the full target language monolingual
training data, not interpolated from individual LMs. Binary features indicating the provenanceof phrase pairs are not used.
7 We end up with English-Greek TED corpus sizes of 67 865 sentences for training, 12 725 sentences for tuning, and
4 241 for testing.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 246

ClassifierLM
MEtrain
MEtrain+lm
MEdev
MEdev+lm

En→De
78.1
83.1
84.8
78.1
81.8

Accuracy [%]En→It En→Pt
90.2
96.9
92.4
96.6
92.5
96.9
83.1
91.0
88.4
96.9

En→El
96.5
96.7
96.6
93.0
96.6

Table 3: Domain classifier accuracies on the English side of a concatenation of all test sets forthe respective language pairs.
6.2.4

Multi-Domain SMT

Multi-domain systems classify the input sentence with a domain classifier. They parameterize
the decoder according to the predicted domain label. We use the parameters of the Domaintuned + LM interp. + indicator feat. MT setups. We evaluate five multi-domain system per
language pair, one for each of our domain classifiers.
6.2.5

Oracle-Domain SMT

In an oracle domain setup, we assume that the correct domain label of each input sentence isgiven. We can parameterize the decoder according to the gold-standard domain label.
6.3

Domain Classifiers

The MEtrain classifiers are trained on the source language side of the TED portion of the training
data and fractions of both the Europarl portion of the training data and the English News Crawl
2014 corpus as provided for the WMT 2015 shared translation task. Again, we include a balanced amount of data from the different domains (e.g. 10% of the Europarl data and 1% of theNews Crawl 2014 data for English→German) in order to not give preference to any of the domains. The MEdev classifiers are trained on the MT development sets as used for mixed-domain
MT tuning.
While building common English domain classifiers would be possible, we decided to train
separate ones for each task and utilize the data resources from the respective language pair.

7

Experimental Results

Domain classifiers. The accuracies of the domain classifiers are presented in Table 3. We report accuracies (micro-averaged F1) on a concatenation of all test sets for each of the language
pairs with the source LM classifier and four variations of the ME classifier (cf. Section 5). Naturally, accuracies are higher for the tasks where only two domains have to be distinguished(En→Pt, En→El) than on the tasks with three domain classes (En→De, En→It). Accuracies
are generally of a high level, even for the simple source LM classifier. We are going to evaluate in MT experiments whether any of the differences in classification accuracy carry over totranslation quality of multi-domain systems.
Translation quality. Tables 4-7 contain B LEU scores obtained with all MT systems on the test
sets from the various domains for the four language pairs. The TED test sets are the common
IWSLT tests sets, the Europarl test sets have been downloaded from matrix.statmt.org,
and the News test sets are the standard ones from the WMT shared tasks. We test all systems on
the sets from all domains, in particular, domain-adapted systems are tested on out-of-domain
sets as well. We also report B LEU scores on concatenations of all test sets (all) in order tomeasure overall performance in open-domain scenarios.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 247

System (En→De)
TED-tuned+ LM interp.
+ LM interp. + indicator feat.
Europarl-tuned+ LM interp.
+ LM interp. + indicator feat.
News-tuned+ LM interp.
+ LM interp. + indicator feat.
Mixed-domain-tuned
Multi-domain, LM classifier
Multi-domain, MEtrain classifier
Multi-domain, MEtrain+lm classifier
Multi-domain, MEdev classifier
Multi-domain, MEdev+lm classifier
Oracle-domain

2010
24.3
24.2
24.4
24.4
23.1
22.8
23.7
23.3
23.4
24.6
24.4
24.3
24.3
24.2
24.3
24.4

TED
2011 201226.7 22.9
26.8 22.7
26.7 23.1
26.3 22.8
24.8 22.0
24.7 21.7
26.2 22.2
25.8 22.1
25.7 22.2
26.9 23.1
26.8 23.1
26.7 22.8
26.7 23.0
26.8 22.8
26.9 22.9
26.7 23.1

2013
25.3
24.6
25.0
25.2
24.0
24.0
24.5
24.2
24.4
25.3
24.9
24.7
24.9
24.7
24.9
25.0

Europarl2007 2008
21.9 21.7
21.2 20.9
20.9 20.7
22.4 22.1
22.5 22.2
22.6 22.1
21.5 21.4
21.3 21.2
21.4 21.0
22.2 22.0
22.5 22.1
22.5 22.1
22.5 22.2
22.5 22.1
22.5 22.1
22.6 22.1

2013
20.2
19.5
19.5
20.4
19.4
19.3
20.6
20.4
20.5
20.6
20.1
20.3
20.4
20.3
20.3
20.5

News
2014 2015
20.8 22.9
20.1 22.2
20.2 22.3
21.2 23.0
19.7 21.8
19.5 21.6
21.1 22.9
20.9 22.6
20.9 22.6
21.2 23.2
20.6 22.5
20.7 22.5
20.8 22.5
20.8 22.5
20.7 22.5
20.9 22.6

all
22.4
21.9
21.9
22.6
21.8
21.7
22.2
21.9
21.9
22.7
22.4
22.4
22.5
22.4
22.5
22.5

Table 4: English→German experimental results (truecase B LEU scores).
System (En→It)
TED-tuned+ LM interp.
+ LM interp. + indicator feat.
Europarl-tuned+ LM interp.
+ LM interp. + indicator feat.
News-tuned+ LM interp.
+ LM interp. + indicator feat.
Mixed-domain-tuned
Multi-domain, LM classifier
Multi-domain, MEtrain classifier
Multi-domain, MEtrain+lm classifier
Multi-domain, MEdev classifier
Multi-domain, MEdev+lm classifier
Oracle-domain

2010
26.8
27.0
27.3
24.6
23.9
24.2
26.8
26.8
26.9
26.9
27.3
27.3
27.3
27.3
27.3
27.3

TED
2011 201226.6 27.3
27.3 27.7
27.7 28.0
24.3 25.5
24.4 25.6
24.3 25.8
26.5 27.2
27.4 28.0
27.5 27.9
26.7 27.0
27.7 27.9
27.6 27.8
27.6 27.9
27.7 27.9
27.8 27.9
27.7 28.0

2013
32.9
33.5
33.4
30.5
30.0
29.9
32.5
33.3
33.1
32.5
33.4
33.3
33.3
33.1
33.3
33.4

Europarl2007 200824.6 25.3
24.3 24.7
24.1 24.6
24.9 25.6
24.9 25.4
25.0 25.5
24.6 25.3
24.7 25.1
24.6 25.1
25.0 25.6
24.9 25.5
25.0 25.5
25.0 25.5
25.0 25.4
24.9 25.5
25.0 25.5

News
syscomb2009
27.5
27.3
27.5
26.5
25.2
25.2
27.7
27.8
27.8
27.9
27.3
26.1
25.8
27.4
27.6
27.8

all
26.9
27.0
27.0
25.8
25.6
25.7
26.9
27.2
27.1
27.0
27.2
27.1
27.1
27.1
27.2
27.2

Table 5: English→Italian experimental results (truecase B LEU scores).
Domain adaptation. As expected, the domain-adapted systems perform better on in-domain
data than when evaluated in a cross-domain experiment. The in-domain systems outperform systems tuned on out-of-domain development sets. More aggressive domain adaptation via LM interpolation and binary provenance indicator features gives mixed results. For
English→German, we basically do not observe gains over in-domain tuning with any of thetwo adaptation methods. They just further reduce quality on out-of-domain data. For English→Italian, we observe gains with both methods on in-domain test sets on top of the
TED-tuned system, but not on top of the Europarl-tuned and News-tuned systems. For English→Portuguese and English→Greek, we see larger gains mostly due to LM interpolationand on TED-domain adaptation.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 248

System (En→Pt)
TED-tuned+ LM interp.
+ LM interp. + indicator feat.
Europarl-tuned+ LM interp.
+ LM interp. + indicator feat.
Mixed-domain-tuned
Multi-domain, LM classifier
Multi-domain, MEtrain classifier
Multi-domain, MEtrain+lm classifier
Multi-domain, MEdev classifier
Multi-domain, MEdev+lm classifier
Oracle-domain

2010
32.1
33.7
33.7
29.9
26.7
26.5
32.3
33.7
33.6
33.7
33.3
33.7
33.7

2011
34.0
35.2
35.6
32.0
28.4
27.7
34.1
35.4
35.4
35.4
34.9
35.4
35.6

TED
2012
34.8
36.1
36.2
31.0
27.5
27.1
33.4
35.9
35.9
35.9
35.4
35.9
36.2

2013
33.9
34.9
35.4
31.6
28.6
28.0
33.2
35.1
35.0
35.1
34.6
35.1
35.4

2014
32.8
34.6
35.2
31.2
28.2
27.6
33.5
35.0
34.8
34.9
34.3
35.0
35.2

Europarl2007 200827.6 27.7
25.5 25.2
24.9 24.5
30.1 30.1
30.3 30.1
30.4 30.1
29.4 29.5
30.4 30.1
30.4 30.1
30.4 30.1
30.2 29.9
30.4 30.1
30.4 30.1

all
30.6
30.2
30.0
30.7
29.1
28.8
31.6
32.6
32.6
32.6
32.3
32.6
32.8

Table 6: English→Portuguese experimental results (truecase B LEU scores).
System (En→El)

TED

TED-tuned+ LM interp.
+ LM interp. + indicator feat.
Europarl-tuned+ LM interp.
+ LM interp. + indicator feat.
Mixed-domain-tuned
Multi-domain, LM classifier
Multi-domain, MEtrain classifier
Multi-domain, MEtrain+lm classifier
Multi-domain, MEdev classifier
Multi-domain, MEdev+lm classifier
Oracle-domain

28.2
29.0
28.9
27.0
25.7
25.7
27.5
28.9
28.9
28.9
28.8
28.9
28.9

Europarl2007 200825.3 24.6
24.7 24.3
24.6 24.2
25.6 25.0
25.6 25.3
25.6 25.1
25.6 25.0
25.6 25.1
25.6 25.1
25.6 25.1
25.6 25.1
25.6 25.1
25.6 25.1

all
26.3
26.4
26.3
26.0
25.7
25.6
26.2
26.8
26.8
26.8
26.7
26.8
26.8

Table 7: English→Greek experimental results (truecase B LEU scores).
On TED, our domain-adapted English→Italian and English→Portuguese systems outperform the best submissions from recent IWSLT evaluation campaigns by several B LEU points.8
On News, our domain-adapted English→German systems are on par with the best phrase-based
system submissions at the WMT shared translation task.9
Mixed-domain vs. multi-domain SMT. Looking at the performance on the concatentationof all test sets, mixed-domain SMT yields a higher B LEU score than any of the domain-adapted
systems on two out of four language pairs (En→De: +0.3; En→It: -0.2; En→Pt: +0.9; En→El:-0.2). Apart from English→Portuguese, the differences are small.
Multi-domain SMT clearly outperforms mixed-domain SMT for English→Portuguese (upto +1.0 on all) and English→Greek (up to +0.6 on all). The choice of the domain classifier
barely matters wrt. translation quality. Due to its compact model, the MEdev classifier would
for instance be a reasonable choice despite not providing the highest classification accuracy.
8 MT English→Italian: +4.3 points B LEU on tst2013 (Cettolo et al., 2013). MT English→Portuguese: +2.8
points B LEU on tst2014 (Cettolo et al., 2014).
9 http://matrix.statmt.org

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 249

20

15

15

10

10

5

5

German

26

.3
26

BLEU

21
.9
21
.7
21
.9
22
.7
22
.5
22
.5

25

20

0

.2
26
.8
26
.8

8

30

28
.

.1

.0
27
.2
27
.2

27

.0
27

.7
27

25

BLEU

25

.6

30

25

30

6

35

32

35

.6
32
.

40

.0

40

.8

Mixed-domain-tuned
Multi-domain
Oracle-domain

31

TED-tuned
Europarl-tuned
News-tuned

Italian0Portuguese

Greek

Figure 1: B LEU scores on a concatenation of all test sets.
Compared to oracle-domain SMT, which is equivalent to choosing the respective indomain translation from the Domain-tuned + LM interp. + indicator feat. system, the best
multi-domain results are on the same level of quality across the board, with a maximum drop of
0.2 points B LEU (En→Pt).
We visualized the results in a couple of plots (Figures 1-4). We use Domain-tuned as ashortcut for Domain-tuned + LM interp. + indicator feat. in all plots, i.e. the in-domain system
results in the plots include LM interpolation and the provenance indicator. Multi-domain in theplots is the variant based on the MEdev+lm classifier.
B LEU histograms on the concatenation of all test sets are shown in Figure 1. The figure
illustrates the results we just discussed, on the concatentation of all test sets. In Figures 2–4 weplotted average B LEU differences wrt. the in-domain system on the TED, Europarl, and News
test sets. In terms of averaged B LEU scores over the in-domain test sets, in-domain systems are
up to 7.8 points B LEU better than out-of-domain systems. Multi-domain SMT is up to 1.7 points
B LEU better than mixed-domain SMT but can also perform minimally worse in some cases, forinstance English→German TED and News, where mixed-domain SMT performs better thanin-domain SMT. However, multi-domain SMT is typically on par with in-domain SMT.

8

Conclusion

While mixed-domain tuning worked for half of the language pairs, our results indicate thatmulti-domain SMT is the more reliable choice. Multi-domain SMT is always on par within-domain SMT translation quality and under some circumstances mixed-domain SMT can perform much worse.
Multi-domain SMT can be easily implemented with a domain classifier and by allowing
for run-time reconfiguration of the decoder with domain-specific weight vectors. Satisfactorydomain classification accuracy can be achieved with a simple and compact maximum entropy
text classifier trained on the small MT development sets and applied at the sentence level.
For scenarios where the model is expected to translate a wide variety of input text, theapproach presented in this paper balances ease of implementation with high performance.

Acknowledgements
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements no 644333 (TraMOOC) and 644402 (HimL).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 250

Mixed-domain-tuned
Multi-domain

1
0

0.2

0

0

0

-0.1

-1

Avg. BLEU diﬀerence on TED wrt. in-domain system

Avg. BLEU diﬀerence on TED wrt. in-domain system

TED-tuned
Europarl-tuned
News-tuned

-0.3
-0.8

-0.9
-1.5

-2
-3

-3.1

-4
-5
-6
-7
-8
German

1
0

0

0

0

-0.2
-1
-1.4

-2

-1.9

-3
-3.2
-4
-5
-6
-7
-8

-7.8

Italian

Portuguese

Greek

Avg. BLEU diﬀerence on Europarl wrt. in-domain system

Avg. BLEU diﬀerence on Europarl wrt. in-domain system

Figure 2: Average B LEU differences on TED test sets.
1
0

0

-0.3

-1
-2

-1.6

0.1

0
-0.1

-0.1

-0.4
-0.9

-1.2

-3
-4
-5
-6
-7
-8
German

1
0

0

0

0

0
-0.1

-1

-0.8

-1.0

-2
-3
-4
-5
-6

-5.6

-7
-8

Italian

Portuguese

Greek

Avg. BLEU diﬀerence on News wrt. in-domain system

Figure 3: Average B LEU differences on Europarl test sets.
1
0

0

0.3

0
-0.2

-1

-0.7

0.1
-0.2

-0.3

-1.2
-2
-2.6

-3
-4
-5
-6
-7
-8
GermanItalianFigure 4: Average B LEU differences on News test sets.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 251

References
Axelrod, A., He, X., and Gao, J. (2011). Domain Adaptation via Pseudo In-domain Data Selection. In
Proc. of the Conf. on Empirical Methods for Natural Language Processing (EMNLP), pages 355–362,
Edinburgh, Scotland, UK.
Banerjee, P., Du, J., Li, B., Naskar, S., Way, A., and van Genabith, J. (2010). Combining Multi-Domain
Statistical Machine Translation Models using Automatic Classifiers. In Proc. of the Conf. of the Assoc. for Machine Translation in the Americas (AMTA), Denver, CO, USA.
Berger, A. L., Della Pietra, S. A., and Della Pietra, V. J. (1996). A Maximum Entropy Approach to Natural
Language Processing. Computational Linguistics, 22(1):39–72.
Bertoldi, N. and Federico, M. (2009). Domain Adaptation for Statistical Machine Translation with Monolingual Resources. In Proc. of the Workshop on Statistical Machine Translation (WMT), pages 182–189,
Athens, Greece.
Bisazza, A., Ruiz, N., and Federico, M. (2011). Fill-up versus Interpolation Methods for Phrase-based
SMT Adaptation. In Proc. of the Int. Workshop on Spoken Language Translation (IWSLT), pages 136–
143, San Francisco, CA, USA.
Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. (1995). A Limited Memory Algorithm for Bound Constrained
Optimization. SIAM Journal on Scientific Computing, 16(5):1190–1208.
Cettolo, M., Girardi, C., and Federico, M. (2012). WIT3 : Web Inventory of Transcribed and Translated
Talks. In Proc. of the Annual Conf. of the European Assoc. for Machine Translation (EAMT), pages
261–268, Trento, Italy.
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., and Federico, M. (2013). Report on the 10th IWSLT
Evaluation Campaign. In Proc. of the Int. Workshop on Spoken Language Translation (IWSLT), pages
15–32, Heidelberg, Germany.
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., and Federico, M. (2014). Report on the 11th IWSLT
Evaluation Campaign, IWSLT 2014. In Proc. of the Int. Workshop on Spoken Language Translation(IWSLT), pages 2–17, Lake Tahoe, CA, USA.
Chen, B., Foster, G., and Kuhn, R. (2013). Adaptation of Reordering Models for Statistical MachineTranslation. In Proc. of the Human Language Technology Conf. / North American Chapter of the
Assoc. for Computational Linguistics (HLT-NAACL), pages 938–946, Atlanta, GA, USA.
Chen, S. F. and Goodman, J. (1998). An Empirical Study of Smoothing Techniques for Language Modeling. Technical Report TR-10-98, Computer Science Group, Harvard University, Cambridge, MA,
USA.
Cherry, C. and Foster, G. (2012). Batch Tuning Strategies for Statistical Machine Translation. In Proc.
of the Human Language Technology Conf. / North American Chapter of the Assoc. for Computational
Linguistics (HLT-NAACL), pages 427–436, Montréal, Canada.
Clark, J. H., Lavie, A., and Dyer, C. (2012). One System, Many Domains: Open-Domain Statistical Machine Translation via Feature Augmentation. In Proc. of the Conf. of the Assoc. for Machine Translationin the Americas (AMTA), San Diego, CA, USA.
Durrani, N., Fraser, A., and Schmid, H. (2013a). Model With Minimal Translation Units, But DecodeWith Phrases. In Proc. of the Human Language Technology Conf. / North American Chapter of the
Assoc. for Computational Linguistics (HLT-NAACL), pages 1–11, Atlanta, GA, USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 252

Durrani, N., Haddow, B., Heafield, K., and Koehn, P. (2013b). Edinburgh’s Machine Translation Systems
for European Language Pairs. In Proc. of the Workshop on Statistical Machine Translation (WMT),
pages 114–121, Sofia, Bulgaria.
Federico, M., Bentivogli, L., Paul, M., and Stueker, S. (2011). Overview of the IWSLT 2011 Evaluation
Campaign. In Proc. of the Int. Workshop on Spoken Language Translation (IWSLT), pages 11–27, San
Francisco, CA, USA.
Federico, M. and Bertoldi, N. (2012). Practical Domain Adaptation in SMT. Tutorial at the Conf. of
the Assoc. for Machine Translation in the Americas (AMTA). http://www.mt-archive.info/
10/AMTA-2012-Bertoldi-ppt.pdf.
Foster, G., Goutte, C., and Kuhn, R. (2010). Discriminative Instance Weighting for Domain Adaptationin Statistical Machine Translation. In Proc. of the Conf. on Empirical Methods for Natural Language
Processing (EMNLP), pages 451–459, Cambridge, MA, USA.
Foster, G. and Kuhn, R. (2007). Mixture-Model Adaptation for SMT. In Proc. of the Workshop on
Statistical Machine Translation (WMT), pages 128–135, Prague, Czech Republic.
Foster, G., Kuhn, R., and Johnson, H. (2006). Phrasetable Smoothing for Statistical Machine Translation.
In Proc. of the Conf. on Empirical Methods for Natural Language Processing (EMNLP), pages 53–61,
Sydney, Australia.
Galley, M. and Manning, C. D. (2008). A Simple and Effective Hierarchical Phrase Reordering Model. In
Proc. of the Conf. on Empirical Methods for Natural Language Processing (EMNLP), pages 847–855,
Honolulu, HI, USA.
Gao, Q. and Vogel, S. (2008). Parallel Implementations of Word Alignment Tool. In Software Engineering,
Testing, and Quality Assurance for Natural Language Processing, SETQA-NLP ’08, pages 49–57,
Columbus, OH, USA.
Haddow, B. and Koehn, P. (2012). Analysing the Effect of Out-of-Domain Data on SMT Systems. In
Proc. of the Workshop on Statistical Machine Translation (WMT), pages 422–432, Montréal, Canada.
Heafield, K. (2011). KenLM: Faster and Smaller Language Model Queries. In Proc. of the Workshop on
Statistical Machine Translation (WMT), pages 187–197, Edinburgh, Scotland, UK.
Kneser, R. and Ney, H. (1995). Improved Backing-Off for M-gram Language Modeling. In Proceedingsof the Int. Conf. on Acoustics, Speech, and Signal Processing, volume 1, pages 181–184, Detroit, MI,USA.
Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In Proc. of the MT
Summit X, Phuket, Thailand.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open Source
Toolkit for Statistical Machine Translation. In Proc. of the Annual Meeting of the Assoc. for Computational Linguistics (ACL), pages 177–180, Prague, Czech Republic.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical Phrase-Based Translation. In Proc. of the Human
Language Technology Conf. / North American Chapter of the Assoc. for Computational Linguistics(HLT-NAACL), pages 127–133, Edmonton, Canada.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 253

Koehn, P. and Schroeder, J. (2007). Experiments in Domain Adaptation for Statistical Machine Translation. In Proc. of the Workshop on Statistical Machine Translation (WMT), pages 224–227, Prague,
Czech Republic.
Lambert, P., Schwenk, H., Servan, C., and Abdul-Rauf, S. (2011). Investigations on Translation Model
Adaptation Using Monolingual Data. In Proc. of the Workshop on Statistical Machine Translation(WMT), pages 284–293, Edinburgh, Scotland, UK.
Macherey, W., Och, F., Thayer, I., and Uszkoreit, J. (2008). Lattice-based Minimum Error Rate Training
for Statistical Machine Translation. In Proc. of the Conf. on Empirical Methods for Natural Language
Processing (EMNLP), pages 725–734, Honolulu, HI, USA.
Mansour, S., Al-Onaizan, Y., Blackwood, G., and Tillmann, C. (2014). Automatic Dialect Classificationfor Statistical Machine Translation. In Proc. of the Conf. of the Assoc. for Machine Translation in theAmericas (AMTA), Vancouver, BC, Canada.
Mansour, S. and Ney, H. (2012). A Simple and Effective Weighted Phrase Extraction for Machine Translation Adaptation. In Proc. of the Int. Workshop on Spoken Language Translation (IWSLT), pages
193–200, Hong Kong.
Marujo, L., Grazina, N., Luis, T., Ling, W., Coheur, L., and Trancoso, I. (2011). BP2EP - Adaptationof Brazilian Portuguese texts to European Portuguese. In Proc. of the Annual Conf. of the European
Assoc. for Machine Translation (EAMT), pages 129–136, Leuven, Belgium.
Matsoukas, S., Rosti, A.-V. I., and Zhang, B. (2009). Discriminative Corpus Weight Estimation for
Machine Translation. In Proc. of the Conf. on Empirical Methods for Natural Language Processing(EMNLP), pages 708–717, Singapore.
Moore, R. C. and Lewis, W. (2010). Intelligent Selection of Language Model Training Data. In Proc.
of the Annual Meeting of the Assoc. for Computational Linguistics (ACL), pages 220–224, Uppsala,Sweden.
Nakov, P. (2008). Improving English-Spanish Statistical Machine Translation: Experiments in Domain
Adaptation, Sentence Paraphrasing, Tokenization, and Recasing. In Proc. of the Workshop on Statistical
Machine Translation (WMT), pages 147–150, Columbus, OH, USA.
Niehues, J. and Waibel, A. (2012). Detailed Analysis of Different Strategies for Phrase Table Adaptationin SMT. In Proc. of the Conf. of the Assoc. for Machine Translation in the Americas (AMTA), San
Diego, CA, USA.
Och, F. J. and Ney, H. (2002). Discriminative Training and Maximum Entropy Models for Statistical
Machine Translation. In Proc. of the Annual Meeting of the Assoc. for Computational Linguistics(ACL), pages 295–302, Philadelphia, PA, USA.
Och, F. J. and Ney, H. (2003). A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19–51.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a Method for Automatic Evaluationof Machine Translation. In Proc. of the Annual Meeting of the Assoc. for Computational Linguistics(ACL), pages 311–318, Philadelphia, PA, USA.
Pecina, P., Toral, A., and van Genabith, J. (2012). Simple and Effective Parameter Tuning for Domain
Adaptation of Statistical Machine Translation. In Proc. of the Int. Conf. on Computational Linguistics(COLING), pages 2209–2224, Mumbai, India.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 254

Ruiz, N. and Federico, M. (2014). Complexity of Spoken Versus Written Language for Machine Translation. In Proc. of the Annual Conf. of the European Assoc. for Machine Translation (EAMT), pages
173–180, Dubrovnik, Croatia.
Salloum, W., Elfardy, H., Alamir-Salloum, L., Habash, N., and Diab, M. (2014). Sentence Level Dialect Identification for Machine Translation System Selection. In Proc. of the Annual Meeting of the
Assoc. for Computational Linguistics (ACL), pages 772–778, Baltimore, MD, USA.
Schwenk, H. and Senellart, J. (2009). Translation Model Adaptation for an Arabic/French News Translation System by Lightly-Supervised Training. In Proc. of the MT Summit XII, Ottawa, Canada.
Shah, K., Barrault, L., and Schwenk, H. (2012). A General Framework to Weight Heterogeneous Parallel
Data for Model Adaptation in Statistical MT. In Proc. of the Conf. of the Assoc. for Machine Translationin the Americas (AMTA), San Diego, CA, USA.
Steinberger, R., Eisele, A., Klocek, S., Pilos, S., and Schlüter, P. (2012). DGT-TM: A freely Available
Translation Memory in 22 Languages. In Proc. of the Int. Conf. on Language Resources and Evaluation(LREC), pages 454–459, Istanbul, Turkey.
Steinberger, R., Pouliquen, B., Widiger, A., Ignat, C., Erjavec, T., Tufis, D., and Varga, D. (2006). The
JRC-Acquis: A Multilingual Aligned Parallel Corpus with 20+ Languages. In Proc. of the Int. Conf.
on Language Resources and Evaluation (LREC), pages 2142–2147, Genoa, Italy.
Stolcke, A. (2002). SRILM – an Extensible Language Modeling Toolkit. In Proc. of the Int. Conf. on
Spoken Language Processing (ICSLP), volume 3, Denver, CO, USA.
Tiedemann, J. (2012). Parallel Data, Tools and Interfaces in OPUS. In Proc. of the Int. Conf. on Language
Resources and Evaluation (LREC), pages 2214–2218, Istanbul, Turkey.
Tyers, F. M. and Alperen, M. S. (2010). South-East European Times: A parallel corpus of Balkan languages. In Procȯf the LREC Workshop on Exploitation of Multilingual Resources and Tools for Centraland (South-) Eastern European Languages, pages 49–53, Malta.
Ueffing, N., Haffari, G., and Sarkar, A. (2007). Semi-supervised model adaptation for statistical machinetranslation. Machine Translation, 21(2):77–94.
Wang, W., Macherey, K., Macherey, W., Och, F., and Xu, P. (2012). Improved Domain Adaptation for
Statistical Machine Translation. In Proc. of the Conf. of the Assoc. for Machine Translation in theAmericas (AMTA), San Diego, CA, USA.
Xu, J., Deng, Y., Gao, Y., and Ney, H. (2007). Domain Dependent Statistical Machine Translation. In
Proc. of the MT Summit XI, pages 515–520, Copenhagen, Denmark.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 255

Korean-to-Chinese Word Translation using
Chinese Character Knowledge
Yuanmei Lu
Toshiaki Nakazawa

lu@nlp.ist.i.kyoto-u.ac.jp
nakazawa@pa.jst.jp

Japan Science and Technology Agency, Tokyo, 102-8666, Japan

Sadao Kurohashi

kuro@nlp.ist.i.kyoto-u.ac.jp
Graduate School of Informatics, Kyoto University, Kyoto, 606-8501, Japan

Abstract
In this paper, we present a way of translating Korean words to Chinese using Chinese character information. A mapping table of Korean and Chinese characters is constructed and used
to obtain possible combinations as translation candidates. The candidates are ranked by the
combination score which accounts for the possibility of the character combination and context
similarity score, which indicates contextual information among words. Parallel resources like
Wikipedia aligned data or Wiktionary data are used for preliminary translation and also usedduring ranking candidates.

1 Introduction
The quality of the statistical machine translation heavily correlates with the amount of parallel
corpora used, and improving the lexical coverage of the parallel corpora plays an importantrole in reducing the number of out-of-vocabulary (OOV) words. However, the vocabularies oflanguages keep growing over time, especially in the case of technical terms. It is impossible tocover all the newly appearing words by augmenting the parallel corpora and therefore we needto prepare bilingual dictionaries for the new words, or translate them separately, for example,
using the transliteration technique.
There are some parallel dictionaries available for limited language pairs and limited domains. In addition, we can extract parallel resources from Wikipedia. It offers hyper-linked
pages of the same topic in different languages, and the title pairs of the linked pages can beused as a parallel dictionary. However, the coverage is not sufficient for both cases especiallyfor technical terms. In some cases, titles aligned by hyper-linked pages can not be seen as exacttranslations of each other since many of them may just be related to similar concepts. Anotherlimitation in this domain is that although there may exist enough resources between English
and other languages, there are very few resources between the non-English languages, such as
Korean, Chinese and Japanese.
Korean, Chinese and Japanese use Chinese characters. In Japan, they use Kanji, which
originated from China, while Korean uses Sino-Korean vocabularies, in which characters(Hangul) can be converted to corresponding Chinese characters (Hanzi). Just as Japanese writing includes both Kana and Kanji, Korean contains two kinds of characters–Hangul and Hanja.
Hangul are Korean characters that are most commonly used and seen, and many Sino-Korean
words have their Hanja writings–Hanja. For the sake of clarity refer to Table 1 which contains
a few examples of Hangul and Hanja.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 256

Hangul (Korean)
Hanja (Korean)
Kanji (Japanese)
Hanzi (Chinese)

㥙㩁
愛情
愛情
爱情

㕤㗴
勞動
労働
劳动

Table 1: Difference of characters written in Hangul, Hanja, Kanji and Hanzi

In reality, Hanja writing is seldom used in modern Korean language. It is used in several technical writings to emphasize the definition of certain words. It is not hard to see thatin many cases, the same Sino-Korean word may written in different Hanja that represent various meanings according to the context. Thus, the Hanja play important roles in word sense
disambiguation (WSD), especially when it comes to terminology words.
Even though the forms are different, most of the vocabularies in these three languages(Korean, Chinese and Japanese) have one-to-one correspondence with respect to the characters.
Using this characteristic, we can perform word translation and use the result to construct a terminological or scientific dictionary, which can further be used in machine translation systems
for the above mentioned languages. However, there is neither a publicly available HangulHanzi-Kanji mapping table, nor is there an official parallel dictionary between Sino-Korean andChinese/Japanese (there exists an online dictionary for common words, but not for terminological words) on the web.
In this paper, we propose a method of translating Korean words (mainly terminologicalwords) into Chinese and Japanese using the Chinese character knowledge. The input is Korean sentences, and our objective is the translation of terminological words in these sentences.
We treat nouns as potential terminological words. A morphological analyzer is employed forextracting nouns, as the pre-processing of our model. We also construct a Hangul-to-Hanzi
mapping table and use it to generate translation candidates, since Hangul and Hanzi have a
one-to-one correspondence. Then we rank the candidates using the character combination andcontextual similarity scores. We also apply a machine learning method for interpolating the two
aforementioned scores of candidates.

2 Sino-Korean Words
2.1 Chinese Words in Korean and Japanese
In Asian languages like Korean and Japanese, majority of words are borrowed or adopted fromother languages, especially when considering terminological words (Matsuda et al. (2008)).
Apart from borrowing/adopting words phonetically like transliteration, a majority of words inthese languages are originally borrowed from ancient Chinese. An example is the adoption of
Kanji words (漢字語) of Japanese, or Sino-Korean words of Korean.
According to Studies on the Vocabulary of Modern Newspapers III published by National
Institute for Japanese Language and Linguistics, Kanji (Chinese characters used in Japanese)
accounts for over 70% of the readable content in newspapers (National Institute for Japanese
Language and Linguistics (1972)). The Kanji words such as “使用 (use) ” are preferably usedthan the native Japanese words such as “使う (to use) ” in Japanese formal writings.
The situation is similar in Korean wherein a significant portion of the words are composedof Sino-Korean. The Standardized Korean Language Dictionary by National Institute of the
Korean Language (NIKL) published in 2004 had 57% of its content as Sino-Korean words (鄭虎聲 (2000)); the Survey of Korean Vocabulary frequency, which was conducted in 1956, has
shown that about 70% of the frequently used words are Sino-Korean (文教部 (1956)). Nowadays, the percentage of Sino-Korean words of spoken language being generally used is grad-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 257

ually decreasing but these kinds of words are still frequently used in formal writings, such as
newspapers and dissertations. Most of the Sino-Korean words are not written in Hanja directly,
but in Hangul. However, we can convert them into Hanja and further to Hanzi because there
is a correspondence among them. Actually, some papers are published with combinations ofHangul and Hanzi in order to specify definitions of vocabularies or emphasize them.
2.2

Related Work

There are some previous work on character conversion, both within language or between two
languages (Chen and Lee (2000), Huang et al. (2004)). During character conversion, a bilingual dictionary is needed for candidate selection. For a low resource language like Korean,a bilingual dictionary containing enough data, including polysemous words, is often hard to
obtain. Moreover, unlike sentence translation, character translation often ignores the context
information of the input source sentence.
Chinese character knowledge is widely used in cross-language information retrieval(Hasan and Matsumoto (2000)), or translation of names of people (Wang et al. (2007, 2008,2009)). During translation, they select named entities by removing the postpositions or theendings, by applying the maximum matching algorithm. For Sino-Korean words that are written using same Hangul word but expressing different meanings according to various context
environments (ambiguous words), they adopt some mutual information score to evaluate theco-relation between the query term and the candidates. In languages such as Japanese and Korean, there is more ambiguity about where word boundaries should be, wherein some particles
may also be a part of a noun, or a verb, their method of extracting target words are often notefficient.
Moreover, unlike information retrieval, the machine translation method also has to ensure
the meaning of the sentence in order to be fluent. In other words, we should also consider
context features of the sentences.
Since Korean characters are phonograms, we can find corresponding Hangul characters forgiven Hanzi characters. Actually, almost all of the Hanzi can be converted to one (or in somerare cases, many) Korean characters. Huang et al. constructed a Chinese-Korean Character
Transfer Table (CKCT Table) to reflect the correspondence between Hanzi and Hangul (Huang
and Choi (2000)). It is reported that the table contains 436 Hangul with corresponding 6763
Hanzi. Practically, there are 4888 common Hanja used in Korean (KATS (Korean Agency forTechnology and Standards) (1997), Hanyang Systems (1992)). Moreover, the number of dailyused Hanzi in Korea numbers around 18001 , while 3500 Hanzi are required to learn for practical
Chinese character level test2 . Obviously, most of the Hanzi in their table cannot be considered
as practical ones. After all, the table is non-public to ordinary users.

3

Proposed Method

Figure 1 gives an overview of our Korean-to-Chinese terminological word translation methodology using Chinese character knowledge. The determined translation result is marked in adarker color. Since our objective is to perform terminological word translation, and in scientificdocuments, terminological words are often Sino-Korean words, mostly nouns, we focus on thetranslation of Korean nouns to Chinese.
Given a Korean sentence, we first need to extract the nouns for which we use morphological analyzers to extract Korean nouns (Section 3.1). In the example in Figure 1, after morphological analysis, Korean words, 㳜㲦㝝, 㑵㩓, 㑀㩁 are extracted from the input sentence. We
1 Wikipedia:㡜㧌㲢㨙
2 Korea Foreign Language Evaluation Institutehttp://www.pelt.or.kr/cs/10/main/main.aspx

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 258

Figure 1: Korean-to-Chinese Word Translation Systemthen look up the Chinese translations of the Korean words in a Korean-Chinese parallel dictionary (Section 3.2). There are three different cases for a given word: unique translation (only
1 candidate per word), multiple translations (more than 1 candidate per word) and untranslatable. The words cannot be translated by the parallel dictionary are further processed whereinpossible Chinese character combinations are generated as the translation candidates, using the
Hangul-Hanzi mapping table constructed in Section 3.3. The candidates are ranked by twoscores: combination score and context similarity score (Section 3.4). The combination scoreaccounts for the possibility of the Chinese character sequences and is calculated using the Chinese web corpus, whereas the context similarity score considers the context feature of the input
sentences and that of the sentences in the Chinese web corpus. Finally, we interpolate these twoscores to handle the score of each translation candidate. The candidate with the highest scoreis considered as the final translation result. During the whole translation procedure, some data
selection methods are involved.
3.1 Extracting Nouns
In Korean sentences, words are separated by spaces among them wherein each word may contain one or more morphological elements. On the other hand, a functional word may havedifferent morphologies under different conditions Li et al. (2013). For example, in Koreansentences:

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 259

(1) 㲡㑰 㦂 㐀㖰
school DAT go‘Go to school.’(2) 㟼 㐀㔴㛽㖰rain NOM fall
‘It rains.’
In the first sentence, the word “㲡㑰㦂” is composed of “㲡㑰 (noun)” and “㦂 (particle)”,while in the two sentences, “㐀” may act as either a verb (as in 㐀㖰) or a particle (as in 㟼㐀).
To get terminologies from a sentence, some methods of extraction, which involves processes
like tokenization and POS tagging should be implemented. However, there is no freely available
tool to accomplish this task of extracting Korean terminological words. Instead, since most of
the terminological words are nouns, we can use a Korean morphological analyzer to extract
nouns from the given Korean sentences and treat them as terminological words.
For the given Korean sentences that contain Sino-Korean vocabularies, we extract nouns
from these sentences with the help of a morphological analyzer, during which also performs
word tokenization. After comparing several POS tag sets (KKMA (2011)) that mainly used for
Korean language, we treat NNP and NNG as POS type for terminological words.
3.2 Translation by Dictionary MatchingSome of the Korean words are translated into Chinese with a parallel dictionary as an initialstep. We use a dictionary of Wikipedia and Wiktionary aligned data to achieve this.
As a multilingual online encyclopedia, Wikipedia titles can be used as parallel data formany languages. In our model, we use the aligned Wikipedia title pairs of Chinese and Korean
and Wiktionary data as a parallel dictionary. In addition, for aligned Wikipedia titles, we apply
the following processes to improve the quality and coverage of the parallel dictionary.
• Make full use of redirect pages of each page, and validate the correctness using the first
sentence (definition sentence) of each page to augment the parallel dictionary. Definition
sentences of some pages containing more than one key words are analyzed to determine
whether these key words are synonyms, by checking whether the definition sentence contains the word “㙚㖙” and “㳓㧹”, which means “or” between the key words. These
synonyms have the same Chinese translation result.
• Convert Chinese characters of traditional Chinese into simplified Chinese for normalization. This step is done because translation result in dictionary matching step will be used
as context feature of each of the character combination candidates (simplified Chinesecharacters).
On the other hand, we collect Sino-Korean words from Korean Wiktionary. Some of the
Wiktionary pages have marked Chinese information which serves to indicate that the Chinesecontent is a translation of the title (語源: 漢字).
During the preliminary translation, some words may have more than one translation result(Chinese alignment). Some Wikipedia titles may contain brackets within them, the brackets
often contain information for disambiguation. For example, consider a Korean word 㨻㨙. The
word is contained in two titles 㨻㨙 and 㨻㨙 (㥳㥱㲡), having aligned Chinese “电子” (electricity) and “转字” (transliteration), respectively. We combine titles of shared Korean words(without brackets) as translation candidates. The example above, 㨻㨙 has two aligned Chinese, 电子 and 转字 translations. Korean words that have more than one Chinese translations

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 260㲢㨙闲, 韩, 恨, 限, 汉姐, 字, 磁, 子, 仔, 姿, 刺, 自, 资, 瓷

Table 2: A portion of the mapping table

are considered ambiguous. In the Korean sentences we work with, we have Korean words belonging to both the above mentioned types. In the case of Korean words with unique Chinesetranslations, we do not need any further processing. However, in the case of multiple candidates,
we compare the context vectors of the candidates calculated with the formerly translated results
and determine the most appropriate candidate as the translation result. The Korean nouns whichcannot be translated with the parallel dictionary are subjected to further processing.
3.3

Generating Translation Candidates

The purpose of this step is to generate possible translation candidates by combining Hanzi
characters converted from the Hangul characters, using the Hangul-Hanzi mapping table. For
instance, using the mapping table in Table 2, we can generate the translation candidates for the
Korean word 㲢㨙 (Chinese character, 汉字) and the possible character combinations couldbe:㲢㨙 (Chinese character, 汉字): 闲姐, 韩姐, ...汉字, 汉子....
Whether these combinations have the proper meaning or not is still unknown. Most ofthem may not have practical meanings. So we need to select the most appropriate combination.
3.4

Rank the Translation Candidates

In this step, we utilize the segmented Chinese web corpus as a filter. First of all, we check theexistence with the corpus and unify the POS type of input Korean words and their Hanzi combinations. According to Xia (2000), in Chinese, there are three POS tags for nouns: NR (Proper
Noun), NT (Temporal Noun) and NN (Other Noun). Thus, according to these definitions, we
extract all NN and NR type nouns from the segmented Chinese web corpus that contain POS information (see Shen et al. (2013)) as a Chinese noun corpus. After that, we generate all possiblecombinations and check whether they are included in this corpus. This step plays a significant
role for reducing the cost of our model. In order to determine the most appropriate translations
among the selected combinations, we utilize combination score and context similarity score.
3.4.1 Combination Score
Combination score Scombi measures the strength of the link between the characters. Here, we
utilize a language model to get the score. The language model returns the possibility (log score)of each combination according to the Chinese web corpus (see as equation (1)). We acquire
the score of the original character sequence of the candidate and also the score for the reverse
sequence. Equation (2) indicates the way the language model computes the score for the reverse
sequence. Suppose that the combination consists of n characters, c1..nn−1∏

S(c1..n ) = log(

P (ci |c1..i−1 ))(1)P (ci |ci+1..n ))

(2)

i=1
n−1
∏

S(cn..1 ) = log(

i=1

For example, the scores for “汉字语” and “语字汉”may be calculated as,
S(汉字语) = log(P (汉) × P (字 |汉) × P (语|汉字)),Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 261

S(语字汉) = log(P (语) × P (字 |语) × P (汉|语字))We combined these two scores by simply adding them. As the length of the word increases,
the score would decrease drastically. We divide the sum with length-1 for normalization. The
following formula defines the combination score. For words with many characters, the scorewill be much smaller after each multiplication and thus we divide the score with the length of
the word. n here indicates the number of characters the word contains.
Scombi (c1 ..n ) =S(c1..n ) + S(cn..1 ),n−1(n ≥ 2)

When n=1 (words with single character), we utilize unigram score as their combination scores.
For example, the combination score for “汉字语” may be calculated aslog(P (汉) × P (字 |汉) × P (语|汉字))Scombi (汉字语) =+ log(P (语) × P (字 |语) × P (汉|语字))3−1

Since the potential number of combinations may run into tens of thousands we conduct
a selection again before moving on to the next step. We sort combinations according to theirscores, and keep only the combinations whose scores are not lower than the highest score -2.
For example, suppose Scombi (汉字)= -5.7126 is the highest one among the combinations, weremove combinations whose combination score is lower than -7.7126. If we still have many
candidates, we keep top 10 candidates among them.
3.4.2 Context Similarity Score
Now that we have obtained candidates using the most possible combinations. To acquire themost proper ones for the given sentences, we consider context features. For each combination,
context vector is constructed using the Chinese corpus. We use sentences which contain eachcombination as the context window, so the element of the vector is the co-occurrence Chinesecharacters (for character-based context vector) and frequency of them. We ignore 125 stop
words (characters) such as 的 and 了 3 . Some combinations that have higher frequency often
contain bigger and wider range of count values in their context vectors. For words with thissituation, co-occurrence characters with lower occurrence frequency are less effective as context
information. Moreover, considering all of the co-occurrence characters is impractical since itwill only increases complexity of time and space. We set the threshold as 100, that is to say,we ignore combinations that occur less than 100 times. Of course, there may also be somecombinations such that all of their co-occurrence characters appear less than 100 times. For
these candidates, we keep all these low-frequency characters as context information.
We also construct another context vector with the information of the input Korean sentence.
We use the formerly dictionary translated Korean words (Section 3.2).
The context similarity score Scontext is defined as the cosine similarity of the two context
vectors–one created using the Chinese corpus that contains Chinese sentences and the otherusing the translation results using the dictionary.
3.4.3 Interpolation
The combination score is useful to examine whether the combination is appropriate or not, andthe context similarity score is helpful for selecting the the most appropriate one according to
the context features where two or more combinations have practical meanings. Therefore, we
3 https://code.google.com/p/verymatch/downloads/detail?name=stopwords.txt

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 262

Hanja强界
計Meaning in Korean㐌㲣㪓㑅㢆

Hangul㐌㑇㑇

Table 3: The Korean-Chinese aligned resource
interpolate two scores obtained in the former two sub chapters and calculate the score of eachtranslation candidate S(cand) as follows:
S(cand) = αScombi + (1 − α)Scontext
The specified value of α (0 ≤ α ≤1) is determined using a method described in next
chapter. The character combination with the highest score is regarded as the final translation
result. If there are more than 2 words that cannot be translated by Wikipedia or Wiktionary,
we process the procedure in sequence (from beginning to end of the sentence) and use thetranslation result as context feature to the remaining unknown words.

4 Experiment
4.1 Settings
Chu et al. had produced a Chinese character mapping table for Japanese (Kanji), Traditional
Chinese (TC) and Simplified Chinese (SC) Chu et al. (2012). Thus, for constructing a tablethat contains mapping relationship between Korean Hangul and Chinese Hanzi, we need toconstruct rather Kanji-Hangul or Hanzi-Hangul tables and merge them. We collected HangulHanja mapping information from the web.
• We acquired 1365 Hanja characters with their aligned Hangul from a freely accessible
webpage4 . These characters are contained by words whose frequency is higher than 5965
times in some Sino-Korean corpus (there is no specific information about the mentioned
Sino-Korean corpus).
• There are also some materials that contain Hanja characters that are used in practice. We
collected 3500 Chinese characters that are required for Hanja level tests PELT (2005), as
mentioned in Section 2.2, to get most generally used Hanja characters with their aligned
Hangul.
• As we mentioned previously, more data is needed to guarantee the coverage of the generally used Hanja characters. In other to collect as much data as possible about Hanja andtheir Hangul alignment, we crawled additional content from some Wikia pages5 , whichcontain more than 20000 Hangul-Hanja correspondence. Of course most of them are notused in general.
We merged these data to get a Hangul-to-Hanja alignment, and combined this table withthe one created by Chu et al. (2012). This combination is possible because most Hanja are traditional Chinese characters, and have one-to-one correspondences with Hangul. Hanja-Hangul
tables mentioned above may contain a large amount of unused characters, which can affect time
efficiency and the correctness of candidates. We checked the compatibility of the table withsome Hanja dictionaries obtained from the web6,7 . There are also Hanja characters that cannot
4 http://korean.nomaki.jp/site_j/kanji16.html
5 㡐㧌㨙:Masoris/hani converter.js - 㲢㨙㧦㯅- Wikia
6 http://hanja.naver.com (㕐㨋㞣㲢㨙㡐㨻)
7 http://small.dic.daum.net/index.do?dic=hanja (Daum㲢㨙㡐㨻)

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 263

testsetαPrecision(%)

1
0.44
93.75
(15 / 16)2
0.44
100.00(5 / 5)

3
0.20
72.22
(13 / 18)

4
0.44
90.00
(9 / 10)

5
0.44
92.31
(12 / 13)

Table 4: The α-Precision relation for each test set
be converted to Hanzi and so we ignore them. This way, we can create a Hangul-Hanja-KanjiHanzi mapping table.
4.1.1 Experimental data
For our experiments, we obtained 100 Korean sentences from several technical documents (onnatural science, such as biology, chemistry, physics, earth science) from the web, which contain3281 words in total8 (1014 nouns, as translation object).
Totally, we have 5368 Hanzi characters map to 481 Hangul characters in our HangulHanzi mapping table. The size of the aligned title dictionary is around 7.1MB (Wikipedia
6.6MB, Wiktionary 495KB) and that of the Chinese web corpus that contains ordinary Chinese
sentences for calculating the context score and combination score of each translation candidate,is 47GB, among them are 184MB of NN, NR words. For querying the web corpus, we used the
KenLM Heafield (2011) language model with smoothing technique included, ignoring start andend symbols and utilizes a character based process.
We used Google Translate as one of the baselines and compared the result with reference.
We checked the precision with exact-match mechanism and obtained a precision of 38.17% 9 .
4.2 Result
We manually set the reference translation results for the test data, and calculated precision forevaluation. To obtain the best value for α during interpolation, we tried 5-fold cross validation.
We divided the test set into 5 parts and recursively selected four of them to get the α that gives
the best score and used it on the remaining part to test the performance of the translation (seeTable 4). We calculated precision of each test set for evaluation, and the best-performing α isthe one with which we can get the highest precision. Figure 2 demonstrates the variation of
precision with the change of α, for a randomly selected test set, testset 4.
We obtained an α for each test set, took the mean value and used it on the test sets again.
The translation results are as shown in Table 5. (a) shows when we do not consider ambiguityof words in the dictionary, while (b) shows the results obtained when we determine ambiguous
words with currently determined translation results (Chinese). The last columns of the twotables shows the result obtained with mean value of α, (α=0.18 in (a) and α=0.39 in (b)). Using
the mean value of α, the precision dropped. (c) demonstrates the results when considering
the scores separately as well as together. The column “+Combi” and “+Context” shows thetranslation result by only considering combination scores and context features, separately, and“+Combi +Context” shows result by considering both combination scores and context features.
In another experiment, we utilized a machine learning algorithm that utilizes these featuresin order to get a better translation result.
We employed SVM rank, Support Vector Machine for Ranking. Taking a Support Vector
approach, the resulting training problem is tractable even for large numbers of queries and large
numbers of features Joachims (2002, 2006).
8 words are separated by space in the sentence
9 https://docs.google.com/spreadsheets/d/1hFiIjaZ7SiSzIRp8Nyk8WRW68D3yRGipD4zeh2D4G4k/
edit#gid=0

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 264

Figure 2: Relationship between α and precision of testset 4

We used combination score of normal sequence and reversed one separately as features,with the third one as context similarity score. For the two combination scores, in order to pickcandidates to process, we implemented a selection mechanism. We first added two scores, thensorted the sum and got the candidates among top 10 scores. We queried combinations scores ofthese candidates respectively and utilized as value of these two features.
On the other hand, the value of the third feature–context score, was calculated with formerly determined translation results in the given sentence. We again divided the sentences into
5 parts, and performed a 5 fold cross validation. As a result, the precision is 93.55%. Thismethod outperforms the former one, is fast and has a low resource requirement.
4.3 Discussions
Among the 1014 words that is analyzed as nouns by the morphological analyzer, 59 are original
Korean words, which are neither adopted words nor terminological words, 46 are transliterated
words. Analysis of mistranslations is presented in Table 6.
When analyzing the experimental results, we found that among the 13 words that could
not be translated, 11 of them are contain phonetic parts within them, such as 㜭㭢㖦㪏 (mechanism), 㣨㣟㯰 (system). These words are untranslatable since they could neither be looked up
in the dictionary (Wikipedia aligned data and Wiktionary data) nor did they have Hanzi mapping
for the phonetic characters.
There is a compound Korean word “㡴㥪㧼㥊㡐㲡 (西洋音乐史学)”, consisting of threewords, “㡴㥪 (西洋)”, “㧼㥊 (音乐)”, “㡐㲡 (史学)”. During the generation of Hanzi combination, the generated combination could not be found in the web corpus. We did use a corpus that
contains compound words (nouns), and found that it contains “西洋音乐” and “音乐史学”, but
no 西洋音乐史学 in it. For compound words that are space separated and do not have translation candidates in the dictionary, we checked the existence of their components in the dictionary
and obtained candidates by combining the translations of the individual components. However,for compound words that contain more than 4 characters and do not have a space among them,
the number of character combination increases drastically as n (length of the word) increases,
which leads to an increase in space and time complexity. Thus, for words in this case, preprocessing during the morphological analysis phase gives better results. For example, for longwords (containing more than 4 characters and do not have space among them), we can implement a split-and-analyze step before moving on to the translation step. We attempt to do further
separation by adding spaces among characters and run the morphological analyzer again, until
two divided parts are both analyzed as nouns. In this method, we successfully segmented “㡴㥪㧼㥊㡐㲡” into “㡴㥪” and “㧼㥊㡐㲡”, and confirmed that they can be translated by theProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 265

Dictionary
Correct
InCorrect
NoTranstion
Precision(%)

646
234
75
73.41
(646/880)+Combination +Context(α=0.18)
43
19
13
69.35
(43/62)(a) Experimental results for not having consider
ambiguous words in dictionary

Correct
InCorrect
NoTranstion
Precision(%)

Dictionary
Sole
Multi
549
190
35
106
371
75
94.01
64.19
(549/584) (190/296)+Combination +Context(α=0.39)
50
12
13
80.65
(50/62)(b) Experimental results when considering
ambiguous words in dictionary

Correct
InCorrect
NoTranstion
Precision(%)+ Combi+Context

44
18
13
70.97
(44/62)

47
15
13
75.80
(47/62)+Context+ Combi
50
12
13
80.65
(50/62)(c) Experimental results when considering
ambiguous words in dictionary (for each score)

Table 5: Experimental results

Wiki-dictionary (Wikipedia and Wiktionary).
In our experiments, instead of considering all combination candidates (obtained using the
web corpus), we used several selection rules like: sort the candidates according to combination
scores in the web corpus and select candidates to whose score differs by two points from thecandidate with the highest combination score. From the candidates, we selected up to 10 candidates that have higher scores. We also utilize the segmented Chinese web corpus as a filter.
We check the existence with the corpus and unify the POS type of input Korean words and theirHanzi combinations. However, this kind of filtering sometimes excludes “useful” candidates.
For example, 㧮㡐㨾 is another word that could not be translated. All of the character combinations of it, including correct translation 类似点 is not contained in the web corpus, thus theword did not have any translation candidate. There are also other words, such as 㦖㝛㨙 and㡜㨾㐀, which have candidates in the web corpus but correct translation is filtered during theprocedure. It is mainly caused by the segmentation error of the web corpus.
Table 7 shows some good and bad example of translation results when using combinationand context scores. The words being discussed are under-lined and the correct translations
are marked with “*”. The table contains combination scores, context scores and interpolated
candidate scores. Determined candidates by each score are marked in bold.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 266

24 failed to find correct translation result in wik data
correct candidate 82 words in Wik dictionary
has lower score
12 words consider combi & context sim

InCorrect trans

No trans result

2 have no combination as nouns in web corpus (NN,NR)
11 are transliterated words
Table 6: Error analysis of mistranslations

Good example:㧌㥚㧷㚿㟙㯢 㨋㦣㨋 㡵㬻㗼㖙 㑅㧍㦂㗪 㟼㣦㲢 㳇㡜㨋㨎㥱㔣㜶, 㨋㘺㖙 㑀㲥 㨙㛽㦂 㗪㫳㲢 㨋㦣㨋 㒔 㩳㧦㨆㦊㚠㨋㦣㘞㑙 㩁㨻㒟㨺 㡜㳒㨚㧌㧺㲠㐳㗽㖰.
Korean Candi
Combi Context Context+Combi
现象* -10.5459 -2.2506-5.4858㳇㡜悬赏-10.3712 -3.4574-6.1538
Bad example:㨋 㟛㑡㞪㧹 㟛㲩㚳㑙 㐇㗪㐀 㕭㥉㡴 㐀㞸㧏 㧙㢙㦂㗂㲩㡴㗪 㕭㧹 㐇㗪㛵 㔣㯍㔶㖰.
Korean Candi
Combi Context Context+Combi
感度* -12.1544 -2.0072-5.9646㐇㗪感到-10.9311 -2.0130-5.4910
Table 7: Examples of good and bad translations.
(* indicates correct translation result)

5 Conclusions
In this paper, we present an automatic Korean-to-Chinese terminological translation mechanismthat uses Chinese character knowledge. The ultimate goal of this work is to use the translation
result for constructing useful resources in machine translation between Korean and Chinese.
A morphology analyzer was used to extract nouns as Sino-Korean words. We used aligned
Wikipedia title data and Hangul-Hanja information in Wiktionary to obtain reference translations. Some polysemous words may have more than one Chinese translation in the dictionary.
To select the most proper one, we compared their context features within the sentence. In order to rank candidates, we both considered context information and probabilities of occurrencein the web corpus. We carried out character-based translation, for which segmented Chinese
web corpus is used to create a character-based and word-based context vector for each translation candidate. Some incorrect reference translation results (insufficiency in alignment data or
failed to be selected correctly with context similarity) caused some incorrect translation results.
Moreover, some candidates which are correct translations were excluded from consideration
since have too lower combination score among the candidate set.
In the future, we intend to determine candidates that are less ambiguous and easy to determine during candidate selection, to guarantee more contextual information. Moreover, we wantto evaluate the proposed approaches on larger test data.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 267

References
Chen, Z. and Lee, K.-F. (2000). A New Statistical Approach to Chinese Pinyin Input. In
Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, ACL’00, pages 241–247, Stroudsburg, PA, USA. Association for Computational Linguistics.
Chu, C., Nakazawa, T., and Kurohashi, S. (2012). Chinese Characters Mapping Table ofJapanese, Traditional Chinese and Simplified Chinese. In Proceedings of the Eighth Conference on International Language Resources and Evaluation (LREC 2012), pages 2149–2152,
Istanbul, Turkey.
Hanyang Systems (1992). KS X 1001:1992 (CJKV Information Processing, Appendix L).
http://examples.oreilly.com/cjkvinfo/AppL/ksx1001.pdf.
Hasan, M. M. and Matsumoto, Y. (2000). Chinese-Japanese cross language information retrieval: a Han character based approach. In Proceedings of the ACL-2000 workshop on Word
senses and multi-linguality, pages 19–26. Association for Computational Linguistics.
Heafield, K. (2011). KenLM: Faster and Smaller Language Model Queries. In Proceedingsof the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187–197,
Edinburgh, Scotland, United Kingdom.
Huang, J.-X., Bae, S.-M., and Choi, K.-S. (2004). A Statistical Model for Hangeul-Hanja
Conversion in Terminology Domain. Association for Computational Linguistics.
Huang, J.-X. and Choi, K.-S. (2000). Chinese-Korean Word Alignment Based on Linguistic
Comparison. In ACL.
Joachims, T. (2002). Optimizing search engines using clickthrough data. In Proceedings of theeighth ACM SIGKDD international conference on Knowledge discovery and data mining,
pages 133–142. ACM.
Joachims, T. (2006). Training linear SVMs in linear time. In Proceedings of the 12th ACM
SIGKDD international conference on Knowledge discovery and data mining, pages 217–
226. ACM.
KATS (Korean Agency for Technology and Standards) (1997).
KS X 1001 Hanja.
http://jinjucci.korcham.net/cms/board/Download.jsp?fileId=
IUAjJDU3MDI2LS0kJA==.
KKMA (2011). Korean Part Of Speech tagging table.
documents/?doc=postag.

http://kkma.snu.ac.kr/

Li, S., Wong, D. F., and Chao, L. S. (2013). Experiments with POS-based restructuring andalignment-based re-ordering for statistical machine translation. ACL 2013, page 82.
Matsuda, M., Takahashi, T., Goto, H., Hayase, Y., Nagano, R. L., and Mikami, Y. (2008).
Technical Terminology in Asian Languages: Different Approaches to Adopting Engineering
Terms. In IJCNLP, pages 25–32.
National Institute for Japanese Language and Linguistics (1972). Studies on the Vocabulary ofModern Newspapers Vol.III.
PELT (2005). Korea Foreign Language Evaluation Institute. http://www.hanja.com/
content/test01.php.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 268

Shen, M., Kawahara, D., and Kurohashi, S. (2013). Chinese word segmentation by mining
maximized substrings. In Proceedings of the Sixth International Joint Conference on Natural
Language Processing, pages 171–179.
Wang, Y.-C., Lee, Y.-H., Lin, C.-C., Tsai, R. T.-H., and Hsu, W.-L. (2007). Korean-Chinese
Person Name Translation for Cross Language Information Retrieval.
Wang, Y.-C., Tsai, R. T.-H., and Hsu, W.-L. (2008). Learning Patterns from the Web to Translate
Named Entities for Cross Language Information Retrieval. In IJCNLP, pages 281–288.
Wang, Y.-C., Tsai, R. T.-H., and Hsu, W.-L. (2009). Web-based pattern learning for namedentity translation in Korean–Chinese cross-language information retrieval. Expert Systems
With Applications, 2(36):3990–3995.
Xia, F. (2000). The part-of-speech tagging guidelines for the Penn Chinese Treebank (3.0).
文教部 (1956). Survey of Korean Vocabulary frequency.
鄭虎聲 (2000). Statistical analysis of vocabularies in Standardized Korean Language Dictionary.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 269

Topic Adaptation for
Machine Translation of E-commerce Content
Prashant Mathur
Marcello Federico∗prashant@fbk.eu
federico@fbk.eu

FBK, Trento, Italy

Selçuk Köprü
Sharam Khadivi
Hassan Sawaf

skopru@ebay.com
skhadivi@ebay.com
hsawaf@ebay.com

eBay Inc., San Jose (CA), USA

Abstract
We describe effort to improve machine translation of item titles found in a large e-commerce
inventory through topic modeling and adaptation. Item titles are short texts which typicallycontain brand names that do not have to be translated, and item attributes whose translation
often depends on the context. Both issues call for robust methods to integrate context information in the machine translation process in order to reduce translation ambiguity. We survey
both existing topic adaptation approaches and propose novel methods that augment the standard
phrase-table models with sparse features and dense features measuring the topic match between
each phrase-pair and the input text. We report extensive experiments on the translation of item
titles from English into Brazilian Portuguese, and show the impact of topic adaptation bothwith and without domain adaptation.

1

Introduction

Domain adaptation and topic adaptation can be seen as complementary methods to cope withthe variability between training and testing data in machine translation. Under this perspective,
domain modeling typically assumes training data partitioned according to human defined labels,
while topic modeling builds on fuzzy clustering of the data and automatically learned labels. In
statistical machine translation, knowledge about the domain or topics of the input can be leveraged to bias the system towards training data matching the same labels of the input. Potential
advantages of topic adaptation over domain adaptation is that fuzzy clustering can better copewith data sparseness than hard clustering, and that automatic labels do not require any manual intervention. On the other side, however, domain adaptation becomes difficult to beat whentraining data can be effectively and naturally partitioned into in-domain and out-of-domain data.
In this paper we discuss the application of domain and topic adaptation to an e-commerce
online MT system (Guha and Heger, 2014), whose target is the translation of user queries andall item titles, descriptions, and specifics shown in the search result pages. In particular, ourinvestigation focuses on the translation of item titles, which consists of concise user-generated
texts describing each item put on sale. Item titles differ in several ways from text genres typically considered in machine translation research. Titles are usually short texts, of maximum∗Most of the work was carried out during a visiting period of the first two authors at eBay Inc.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 270

100 characters, with a simple syntactic structure, and containing brand names, feature values,
as well as specific jargon. Their translation poses several challenges (Sanchez and Badeka,2014), such as the correct rendering of proper names, which can often be confused with common names, and the correct translation of product features, which often depends on the context.
From a statistical learning perspective, MT of item titles is also hard because of the large varietyof content present in eBay’s inventory, which are very unevenly populated but also significantly
overlapping in terms of linguistic content.
The idea that we follow in this work is to employ topic modeling to better translate English item titles to a foreign language. Since we have a relatively small amount of bilingual
in-domain data compared to bilingual out-of-domain data and English in-domain monolingual
data, we aim to apply topic adaptation on the bilingual data and to train a topic model on themonolingual data (see Section 2). Then, we enrich in-domain and out-of-domain parallel datawith topic information and embed this in the translation model of the MT system. At testingtime, we infer the topics of the input and use them to dynamically adapt the MT system. In
this work we survey and compare different topic adaptation methods from the recent literatureand measure their impact on translation performance with and without domain adaptation. We
report experimental result with a Moses-based phrase-based system on the English - Brazilian Portuguese language pair. This particular pair is one of the active language pairs at eBay.
While for domain adaptation we deploy a state-of-the-art method, for topic adaptation we investigate the use of new sparse features which we compare against other features proposed in
the literature.
The rest of the paper is arranged as follows. We first introduce topic modeling applied
to our e-commerce content in Section 2, then we survey previous work on topic adaptation for
statistical MT in Section 3, afterwards we present our take on topic adaptation in Section 4, andfinally we report on our experimental set-up and results in Sections 5 and 6. We end the paperwith some conclusions and future work.

2

Topic Modeling

2.1

Content

In eBay, machine translation plays an important role to facilitate cross-border trade between
sellers and buyers with different languages (Guha and Heger, 2014). eBay is a marketplace
where sellers can advertise their items on the site and buyers can search for the items andthen electronically bid for them. To enable a trade between buyers and sellers with different
languages, at least four types of texts need to be translated: queries, item titles, descriptions,and item specifics. This work focuses on the translation of item titles, which are concise andusually very informative descriptions of the items put on sale. For instance, the item title:
new men’s white jekyll & hyde jeans winston designer regular fit shirt size s-xxl
specifies, in order, the condition, target gender, color, brand, designer, fit, item type, and sizeof the product. Common challenges in the translation of eBay’s user generated content in general, and of titles (Sanchez and Badeka, 2014) and queries (Picinini, 2014) in particular, are theproper rendering of proper names and the translation of words which can have multiple senses,
depending on the context in which they appear. For example, the word “age” might have different meanings if context is “baby”, “wine” or “collectibles”. Similarly, the word “j-hook” might
have different meanings if context is “motors” or “garden”.
The core idea of this work, hence is to apply topic modeling to efficiently represent thecontext of single words or expressions in order to improve the accuracy of their translation by aphrase-based statistical MT system. In the following, we describe the monolingual data and the
topic model that we used for this purpose.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 271

2.2

Sampling

The amount of monolingual item titles available to eBay is huge and very unevenly distributed
across different levels of categories of eBay’s inventory. Actually, items in eBay’s inventory areclassified according to a hierarchical taxonomy. The hierarchy itself contains 34 top-level categories (L1) with varying degrees of depth in each category (L2=400 and L3=4000 categories).
For example, the top level category “Books” has eleven second-level categories (L2) and eachof those categories have anywhere from four to thirty categories, with many of these having
subcategories as each topic becomes more and more specific. All traded items are placed in theleaves of this hierarchy.
For the sake of experimentation, we first perform sampling in order to collect a morebalanced and manageable collection of titles. Hence, starting from a collection of billions of
item titles, after stratified sampling from each L2 category we end up with a collection of 4.3M
item titles. Using a uniform sampling method, we further sub-sampled data (from each L1
category) to around 708K item titles, which is actually used to train the topic model.
2.3

Models

Topic models can be trained from an arbitrary document or sentence collections with different
methods, such as probabilistic latent semantic analysis (Hofmann, 1999), hidden topic Markov
models (Gruber et al., 2007), and latent Dirichlet allocation (LDA) (Blei et al., 2003). For all
methods, there are existing software tools that allow to train topic models after specifying thedesired number of topics and a few training options. All tools permit then to use a training
model to infer a topic distribution for a given sentence. In our case, we deployed a LDA modeltrained with the Standford TMT1 tool. In particular, we worked with a training configuration
using 30 topics (similar to size of L1 categories) and 1000 iterations. We also experimented
with different number of topics but empirically found 30 to be the optimum number. Moreover,
we excluded all item titles with less than 5 words, all words occurring less than 3 times, and allwords made of less than 3 characters. The pruning steps were performed to exclude from themodel item titles providing too little context, words which are too infrequent, and very frequentand short words that to not bear any topic information.
In the following table, we report the 15 most relevant words of the first 10 topics trainedwith the LDA model. As can be seen, some of the topics are easily recognizable, e.g. T01(toys), T04 (electronics), T05 (photo), and T09 (fashion), and T10 (hunting and fishing). The
other topics look instead combinations of multiple categories, e.g. T06 seems a combination of
pet supplies and fashion.
T01
T02
T03
T04
T05
T06
T07
T08
T09
T10

doll high barbie monster little dolls girl lot fashion dress pony and american hair withfree shipping fish aquarium beads tank diy ship round glass pcs wholesale plastic water craft
screen car baby lcd seat glass replacement touch protector cover cloth diaper safety holder glasses
digital control module with remote power sensor switch board lcd meter system arduino air kit
camera lens canon nikon mount digital video with sony gopro dslr camcorder adapter black herodress pet dog clothes coat long winter size sweater women puppy warm apparel jacket fashion
nail set art brush hair color makeup gel eye kit cream polish powder tool skins
figure anime movie poster action disney hot mask toys sex toy series japan prop batman
size black jacket mens shoes leather boots large blue womens nwt medium ski white men’s
knife steel tool stainless set blade folding with pocket gun black tools handle hunting fishing

Table 1: LDA topic model of item titles: top 15 relevant words of the first 10 topics.
Finally, by using the same tool, the topic model is applied to annotate with topics all the
1 http://nlp.stanford.edu/software/tmt/tmt-0.4/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 272

sentences in the source side of all the available parallel data. As a result, each sentence of theparallel training data is associated with a topic vector or distribution.

3

Related Work

Topic models have been investigated in the statistical MT literature to enhance both linear or
hierarchical phrase-based models with additional context-topical information derived from thetraining and testing data.
Previous works have approached this issue under different perspectives, also providing
different levels of integration of topic information. For instance, in (Gong et al., 2011) and (Ruizand Federico, 2011) authors devised and exploited cross-lingual topic models to generate topic
relevant target words for an entire document or sentence. This differs from other approaches
and our one, which instead exploit monolingual topic models in the source language to directly
enhance the selection process of single phrase translations pairs during decoding.
In (Eidelman et al., 2012), topic dependent lexical probabilities are directly integrated ina hierarchical phrase-based system. The authors start with topic distributions inferred at the
document level on the source side of the parallel data. After extraction of the translation rules,
topic-conditional translation probabilities are inferred by computing the expectation over the
topic vectors observed in all the sentences where each translation rule was extracted from. At
decoding time, these probabilities are weighted by the topic prior inferred on the test document.
This results in a set of sparse features, one per topic, which are tuned on a development set.
In this work, we implement and evaluate a similar set of sparse features for a phrase-based
decoder.
In (Su et al., 2012) topic models are used to off-line adapt a phrase table trained on outof-domain parallel data by using in-domain monolingual data. Topic distributions are inferredthrough hidden topic Markov models trained on monolingual sentences. Two distinct topic
models are trained, one in-domain and one out-of-domain, which are mapped via a mixture
model. Finally, similar to (Eidelman et al., 2012) topic-conditioned phrase translation probabilities trained on out of domain data are weighted with the in-domain topic prior probabilities.
Contrary in our case, topic information is integrated just in the training phase to bias the translation model toward the in-domain data. Our purpose, instead, is to dynamically adapt thetranslation model at testing time.
In (Xiao et al., 2012) monolingual topic models are trained on the source side with LDA
and topic vectors are associated to hiero rules. Differently from (Eidelman et al., 2012), during
decoding topic posterior distribution on phrase-pairs are matched against the topic distributionof the test sentence. Matching is performed with the Hellinger divergence. In addition, a feature
measuring topic sensitivity of each phrase-pair is included based on the entropy function. In ourwork, we integrate and compare several dense features that compute the match between the topic
distributions of the input and of each phrase-pair candidate. Among them we also include thetwo features proposed by this work.
In (Hewavitharana et al., 2013) topic adaptation is performed in context of machine translation of task-driven conversations. Topics vectors are inferred via LDA on the source sideof in-domain parallel training data. At test time, the topic vector of the conversation is incrementally updated at each turn. During decoding, with a Moses-like phrase-based system, eachcandidate phrase-pair activates a feature function measuring the highest similarity between thecurrent topic vector and all topic vectors associated to the occurrences of the phrase-pair in the
training corpus. Similarity is computed by taking the complement of the Jensen-Shannon divergence. In our work, we also deploy this divergence measure to measure the match between
the topic vector of the input and the topic vector of each candidate phrase-pair.
Finally, in (Hasler et al., 2014a) the authors combine and compare domain adaptation andProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 273

topic adaptation in phrase-based statistical MT for the translation of texts from three different domains. Concerning topic adaptation, the standard Moses phrase-based feature functions
associated to the phrase-table are augmented three sets of dense feature functions: (i) two translation probabilities, (ii) one language model score and (iii) three topic distribution similarity
feature functions. The first set of features introduce source-to-target phrase probabilities that
account for topic information, the second set scores unigrams of the target phrase according totheir topic relevance, and the latter measures the similarity between the input topic distribution
and the topic distribution associated, respectively, to the whole phrase-pair, to the target phrase,
and to the most representative target of the phrase. Topics distributions are inferred similarly
to (Eidelman et al., 2012) at the level of whole speeches. Particular care is taken about sampling data of different domains, to avoid domain biases, as well as sampling the same amountof context data for each phrase-pair, to not avoid context bias. The authors main conclusion isthat topic adaptation helps especially if there is a high divergence between training and testingdomains. Moreover, topic vectors can be helpful also to predict the domain of test data when
topic domain vectors are used as proxy of data. As is the case with Hasler et al. (2014a), we
also compare topic adaptation with domain adaptation, but with respect to topic adaptation our
main emphasis is on the combination of sparse and dense features.

4

Topic Adaptation

4.1

Approach

We apply the topic model discussed in Section 2 to infer the topic distribution on the sourceside of all bilingual training data of our statistical MT system (details will follow in Section 5),
on the development set and on the evaluation set. Notice that we inferred the topic distribution
on out-of-domain training data at the sentence level rather than at paragraph or document level.
This choice is to keep annotation consistent with the training and testing conditions of the topic
model, which are performed on item titles.
Similar to previous works, we trickle-down topic information from the sentence level tothe phrase-pair level. By borrowing the notation from (Hasler et al., 2014b), we estimate theprobability p(t|s, k) of a target phrase t given a source phrase s and the topical information k(k ∈ 1 . . . K where K is the total number of topics), through the formula:Pd p(k|d) · c(t, s; d)(1)p(t|s, k) = P Ptd p(k|d) · c(t, s; d)Where c(t, s; d) is the count of target phrase t and source phrase s being extracted from sentence(our proxy for document) d in the training data, and p(k|d) is the probability of topic k in
sentence d in the training data.
In addition to the probabilities computed with (1), we also infer topic vector (distribution) φs,tfor each phrase-pair (s, t) extracted from the training data. Each component of the vector is theprobability p(k|s, t), which is inferred by averaging over all topic vectors corresponding to thesentences (d) in which the phrase pair was extracted from.
For each phrase pair we only keep the relevant topics and set the probability of the othertopics to zero. In particular, we compute the perplexity2 (P P ) of the topic distribution andkeep only the most probable dP P e topics. In general, the perplexity tells how many equally
likely topics can be represented with the number of bits of an optimal encoding of a topic
distribution. We have empirically observed that the ceiling of the PP typically identifies thepoint in the ranked list of topics after which there is a significant drop in probability. Moreover,P P gives us also a measure of the topic specificity of a phrase pair. In particular, the lower
2 Definition and properties of the perplexity function can be found for instance in (Federico and De Mori, 1998).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 274

the perplexity of the topic distribution is, the more topic-specific is the translation represented
by the phrase-pair phrase. On the contrary, phrase-pairs with high perplexity values reflecttranslations observed in sentences from many different topics, and thus identify translations not
weakly depending on their context. Hence, we expect that the translation model will mostly
benefit from topic information only for phrase-pairs with low perplexity. For this reason, we
add topic information p(t | s, k) and p(k | s, t) to every entry s, t of the phrase-table only when
the perplexity of φs,t is below a given threshold P Pmax and up to P P topics.
The information entered in the phrase-table is exploited to activate feature functions at test
time, by combining them with topic information inferred on the input sentence at test time.
Similarly to (Hasler et al., 2014b) we denote the input topic information with the vector φc ,where c stands for context representation. Notice, that we apply the same topic pruning strategy
explained before also for the input topic vector.
In the following, we present the list of feature functions that we have explored in this work.
4.2

Features

1. Joint Probability: The topic probability p(t|s, k) in itself is not enough to disambiguate
between the context. As an example, assume that our English-Portuguese phrase tablecontains the following two phrase pairs with corresponding probabilities of target given
source and topic:(a) age ||| idade ||| topic14 0.6 topic04 0.2(b) age ||| era ||| topic25 0.5 topic14 0.3
The two entries show two different translations of the English word age, whose probabilities vary according to the context they have been observed with. In particular, thetranslation idade has been observed with topic14 and topic04, while the translation era
has been observed with topic25 and topic14. However, this information can be properly
exploited only once the topic information of the input sentence is known.
Let us assume the following two input sentences and (pruned) context vectors:(a) early english bronze age period blade c. 1600 bc <topic25 0.9>(b) supre hempz age defying moisturizer- 1 bottle <topic14 0.7>
Our first feature function activates for each translation option a sparse feature for each
topic that occurs in both the context and the phrase-table with value:
f1k (t, s, c) = − log p(k|c) · p(t|k, s)
Going back to the first input sentence of our example, translation of age with idade would
not activate any sparse feature, while translation of age with era would activate sparsefeatures topic25 with score −log(0.45). For the second input sentence, both translationsof age with idade and era would activate sparse features topic14 with scores −log(0.42)
and −log(0.21), respectively. The plain interpretation of this example is that our sparsefeature function only rewards the translation era for the first sentence, while for the second
input sentence it rewards both translations but gives a higher score to the translation idade.
Notice, that our sparse feature is derived from the feature that Eidelman et al. (2012) proposed for hierarchical phrase-based decoding. This feature has a clear probabilistic interpretation: the product p(k|c) · p(t|s, k) computes the joint translation-topic probabilityp(k, t | s) by multiplying the translation probability conditioned to topic with the priortopic probability coming from the context.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 275

2. Geometric Mean: The same principle of 1. applies with our second basic feature, but thistime with the phrase table annotated with posterior probabilities p(k|s, t). Again, for eachphrase-pair and for topics present both in the phrase table and in the context, our sparsefeature takes the product of the topic probability on the input and the topic probability ofthe phrase pair.
f2k (t, s, c) = − log p(k|c) · p(k|t, s)
The intuitive interpretation behind this feature is to measure the level of each matched
topic with the geometric mean between the probabilities in the context and phrase-table.
Notice that the the above expression is equivalent to the log of the geometric mean but aconstant factor 0.5 which we assume to be absorbed by the feature weight.
While the previous features are sparse, in the sense that they are computed over single
topics thus resulting in K distinct features, the following feature are dense, i.e. it is a single
feature is computed for all topics. In particular, all features try to measure the similarity
between the topic distributions on the input and on each phrase pair.
3. Crude-Count: In this feature, we simply count the number of topics active in both input
and the phrase pair and normalize this count over the total number of topics in both thecontext and the phrase-pair. Formally:(|{k: p(k|c)·p(k|t,s)>0}|)f3 (t, s, c) = − log (|{k:p(k|c)+p(k|t,s)>0}|)

Notice that this feature as well as the remaining features are activated for phrase-pair only
if there is at least one common topic between context and the phrase-pair.
4. Cosine Similarity: This feature computes the cosine similarity between the input topic
vector (φc ) and the phrase-pair topic vector (φt,s ) .
f4 (t, s, c) = − log cos(φc , φt,s )This feature was proposed in (Hasler et al., 2014b).
5. JS Divergence: This feature computes Jensen-Shannon divergence between input topic
vector (φc ) and phrase-pair topic vector (φt,s ).
f5 (t, s, c) = − log JS(φc , φt,s )This feature was proposed in Hewavitharana et al. (2013).
6. Hellinger Divergence: Finally, this feature computes the similarity with Hellinger’s divergence.
f6 (t, s, c) = − log HD(φc , φt,s )1
HD(φc , φt,s ) = √2qppΣki=1 ( φc,i − φt,s,i )(2)This feature was proposed in (Xiao et al., 2012).
7. Sensitivity: We measure how sensitive the phrase pairs are to the topics. Here, the idea isto penalize the phrase pairs with the topic vectors of high entropy. High entropy of a topic
vector denotes that the phrase pair is susceptible to multiple topics, which means it occurs
in multiple context and hence topic vectors are not useful for any disambiguation.
f7 (t, s, c) = − log H(φt,s )This feature which can be computed offline, as it does not depend on the context, was also
proposed by Xiao et al. (2012).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 276

The first two sparse features and the other dense features are linearly combined with standard
features used in phrase based decoding. Hence, with the notation f1 + f3 + f4 + f5 + f6 + f7
we indicate that the standard translation score computed by the decoder is augmented with:K
X

λ1,k f1k (s, t) +

k=1

7
X

λi fi (s, t)

(3)

i=3

where the K + 5 λ-weights are tuned together with the other weights of the translation model.

5

Experiments

5.1

Task and Data

We evaluated our topic adaptation approach on the translation of item titles from English to
Portuguese (Brazilian). Parallel data used to train, tune and evaluate MT systems comes fromvarious publicly available collections, proprietary repositories and in house translated item titles. In particular, in-house translated items, descriptions, and specifics are here considered
as in-domain data while all the rest is regarded as out of domain data. For development andtesting purposes we use manually translated item titles for which two reference translations areavailable. Statistics on the amount of parallel data for each category are given in Table 2.

Segments
Tokens EN
Tokens PT

Train (Out-Domain)
5.28M
69M
70M

Train (In-Domain)
336K
2M
2M

Dev
1631
27K
31K

Test
1000
10K
11.6K

Table 2: Statistics of English-Portuguese parallel data.

5.2

MT Systems

This section describes the topic adapted MT systems and the two baseline MT systems developed for comparison purposes. All MT systems are built using the Moses toolkit (Koehn et al.,2007) and the linear weights for all systems are optimized using the k-best batch MIRA implementation provided in the Moses toolkit (Cherry and Foster, 2012). Performance of all systems
are reported in terms of BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) scores. Statistical significance tests were conducted using approximate randomization tests (Clark et al.,2011).
Baseline System In-domain and out-domain parallel data were taken in 10:1 ratio for training
the word alignments. Translation models along with operation sequence models (Durrani et al.,2011) were trained using the standard pipeline of Moses. Due to the nature of the item titles,
we did not use any lexicalized reordering model in the MT system. The distortion limit was
set to 6. On the target side, we built a trigram LM, using KenLM (Heafield, 2011) trained withmodified Kneser-Ney smoothing (Chen and Goodman, 1996).
Domain Adapted System The domain adapted system was built on top of the baseline system. An additional translation model was built using the in-domain data and then the fill-up
adaptation method (Bisazza et al., 2011) was applied to combine the in-domain and out-domain
phrase tables. Fill-up simply adds a provenance feature in the phrase table with a score of 1 if
the phrase pair is present in in-domain phrase table and 0 if it is from out-domain phrase table.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 277

Topic Adapted Systems To evaluate the performance of the topic adapted systems using thefeatures functions presented in Section 4.2 we followed a component analysis approach. Eachbasic sparse feature was added to the domain adapted system separately, shortly (1) DA+f1,(2) DA+f2. We built two separate systems because when we tag the topics in the phrase table,
we can either set them to a distribution of topics over phrase pair (P (k|s, t)) or a target-phrasetranslation probability given the source phrase and the topic (P (t|s, k)). Then we added theother dense features one by one, resulting in 10 distinct systems i.e. 1) DA+f1+f3 2) DA+f1+f43) DA+f1+f5 4) DA+f1+f6 5) DA+f1+f7 6) DA+f2+f3 7) DA+f2+f4 8) DA+f2+f5 9) DA+f2+f610) DA+f2+f7. Finally, we also combined all dense features together on top of basic sparsefeatures to build 1) DA+f1+f3+f4+f5+f6+f7 2) DA+f2+f3+f4+f5+f6+f7. To evaluate the impact
of topic adaptation independently from domain adaptation we performed the same analysis alsowith the baseline (BA) system.

6

Results and Discussion

6.1

Topic Model Analysis

Since, our method of topic adaptation depends on the quality of topic labels inferred on training
data, development and evaluation sets; first, we analyze the distribution of topic labels. Figure
1 shows the average probability mass for each topic in three sets. The plot shows that topics
represented in the training are not always very well covered in the development and evaluation
sets. Topics 14, 15 and 26 are very frequent in the training data, as a result when we project the
topic distribution to phrases in the phrase table these topics occur in more phrase pairs than any
other topic. Thus, as a consequence, these topics should have less discrimination power thanother topic features. To validate this hypothesis we also plotted the weights of the topic features
f1 after tuning them with k-batch MIRA. In fact, we can observe that the algorithm weights
them less compared to other topic features as shown by green lines in the figure.
An interesting note is that for some topic labels the tuning algorithm assigns negative
weights. A possible reason is that there is a topic-translation mismatch between training anddevelopment data. This can be explained by the fact that the training data contains both indomain and out-domain data while the development set contains only in-domain data. Hence,the possibility is that topics mostly occurring in phrase-pairs extracted from out-domain data
are being penalized to the advantage of topics mostly occurring in in-domain phrase-pairs.

Figure 1: Topic distribution on training, development, and test data set. Green bar shows theweight of each topic feature when tuned with MIRA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 278

6.2

MT experiments

Table 3 presents the results for the baseline, domain adapted and all topic adapted systems. We
split the results in three blocks: the first block of results shows the impact of adding sparse topic
features (f 1 and f 2) on top of the baseline (BA) system. The second block shows results with
the Joint Probability feature and the third shows results with the GeoMean feature. In the first
block, both features improve performance consistently (at least 0.3 gain in BLEU points) overthe baseline system and we even observe statistical significant improvements in BLEU scorewith the Joint Probability feature (f1). Since, our goal is to improve over domain adaptation wetested the rest of the features only on top of the domain adapted (DA) system.
SystemBaseline (BA)BA + f1
BA + f2

BLEU
36.99
37.52∗
37.28

TER
49.15
49.09
48.93

Domain Adapted (DA)DA + f1
DA + f1 + f3
DA + f1 + f4
DA + f1 + f5
DA + f1 + f6
DA + f1 + f7
DA + f1 + f3 + f4 + f5
DA + f1 + f3 + f4 + f5 + f6 + f7

39.42
39.56
39.66
39.70†
39.56
39.66
39.60
39.67
39.57

48.12
48.11
47.77†
47.75†
47.84
47.77†
47.84
47.77†
47.72†

Domain Adapted (DA)DA + f2
DA + f2 + f3
DA + f2 + f4
DA + f2 + f5
DA + f2 + f6
DA + f2 + f7
DA + f2 + f3 + f4 + f5
DA + f2 + f3 + f4 + f5 + f6 + f7

39.42
39.60
39.61
39.60
39.62
39.60
39.66
39.71†
39.53

48.12
47.86
47.79
47.76†
47.77†
47.78
47.68†
47.77†
47.68†

Table 3: BLEU and TER scores of systems on English to Portuguese data set showing impact ofsparse topic features against a weak (BA) and a strong (DA) baseline system (in bold). System
performance marked with ∗ and † show significant different results w.r.t baseline (BA) anddomain adapted system (DA) with p-value < 0.05.
The second block shows results with the topic adapted system using the Joint Probability
feature. Individually, the best features which give significant improvements in TER scores overthe DA system are the crude-count, cosine similarity and the Hellinger’s divergence feature,i.e. DA+f1+f3, DA+f1+f4, DA+f1+f6. Concerning the BLEU scores, only the cosine similarity
feature is significantly better than the domain adapted but on an average we observe 0.2 BLEU
points improvement per feature. We also observe significant gains in TER over the domain
adapted system when we combine all dense features together (see systems DA+f1+f2+f3+f4+f5
and DA+f1+f2+f4+f5+f6+f7).
The results in the third block are using the GeoMean sparse feature. The component wise
analysis shows that the cosine similarity, Hellinger’s divergence and the Sensitivity features give
statistically significant improvements (p < 0.05) over the DA system in terms of TER scores.
Average improvements of 0.2 BLEU points are observed across individual feature systems. In

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 279

terms of BLEU, our best system is the one which combines all the features together without
Hellinger’s divergence achieved statistically significant gains (p < 0.05) over the DA system(DA+f2+f3+f4+f5). In terms of TER, we observe statistically significant gains (p < 0.05) of
0.34 TER points in the best systems. These systems are the combination of GeoMean feature
with Sensitivity feature (DA+f2+f7) and the one which combines all dense features together
with the GeoMean feature (DA+f2+f3+f4+f5+f6+f7).
In Table 4 we show examples from the evaluation set where our system solved the problemsof context disambiguation and proper rendering of proper names. In the first example the source
title contains the words endurance which is a brand and colander which is the name of a cooking tool. Domain adapted system (DA) incorrectly translates endurance as resistência while
the topic adapted system (TA) correctly took the verbatim translation. In the same sentence,
DA translates colander as escorredor while TA correctly picked the more specific translation
escorredor de macarrão (colander for pasta). In the second example the source contains:
columbia river crkt which is a brand name. The DA system erronously translates river as rio
while the TA system produced the correct verbatim translation of the brand name.
Source
DA
TA
Ref1
Ref2

rsvp international 5-qt . endurance colander 1024
rsvp international 5-qt . resistência escorredor 1024
rsvp international 5-qt . endurance escorredor de macarrão 1024
rsvp international 5-qt . coador endurance 1024
escorredor de macarrão endurance de 5 quartos de galão da rsvp international 1024

Source
DA
TA
Ref1
Ref2

columbia river crkt crawford kasper lawks zytel knife !
rio columbia crkt crawford kasper lawks zytel faca !
columbia river crkt crawford kasper lawks zytel faca !
faca columbia river crkt crawford kasper lawks zytel !
faca canivete columbia river crkt crawford kasper lawks zytel !

Table 4: Examples from the domain adapted (DA) and topic adapted (TA) systems.

7

Conclusion

An open problem in machine translation is how to effectively handle and incorporate the context
information in the translation models that can help the system to properly disambiguate between
competing translation alternatives. In this paper we presented methods for topic adaptation forphrase-based machine translation that have been experimented in an e-commerce application
scenario. In particular, starting from state-of-the-art LDA topic modeling, we present several
feature functions (some of them new and others derived from the literature) that combine topic
information integrated in the phrase-table with topic information inferred on-the-fly on the input text. We report results on an English-Portuguese translation task of item titles that show
consistent and statistical significant improvements through topic adaptation over both a genericbaseline MT system and a domain adapted MT system. Our work shows that the use of sparsefeatures permits to identify and cope with topic-translation inconsistencies in the training data,
which from one side cannot be avoided when data from multiple and diverse sources are pooled
together, but from the other side calls for more refined methods to label such training data with
topic labels. In the future we plan to investigate feature regularization methods in order to selecttopic-features with high discrimination power.

Acknowledgements
The first author received support through a financial gift by eBay Inc. to FBK. The second
author received support by the EU H2020 funded MMT project (grant agreement No 645487),

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 280

by eBay Inc., and by FBK’s Mobility programme.

References
Bisazza, A., Ruiz, N., and Federico, M. (2011). Fill-up versus interpolation methods for phrasebased SMT adaptation. In IWSLT, pages 136–143.
Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent Dirichlet Allocation. the Journal ofmachine Learning research, 3:993–1022.
Chen, S. F. and Goodman, J. (1996). An empirical study of smoothing techniques for language
modeling. In Proceedings of the 34th Annual Meeting on Association for Computational
Linguistics, ACL ’96, pages 310–318, Stroudsburg, PA, USA. Association for Computational
Linguistics.
Cherry, C. and Foster, G. (2012). Batch tuning strategies for statistical machine translation.
In Proceedings of the 2012 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, pages 427–436, Montréal,Canada. Association for Computational Linguistics.
Clark, J. H., Dyer, C., Lavie, A., and Smith, N. A. (2011). Better hypothesis testing for statisticalmachine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics: Human Language Technologies:
short papers-Volume 2, pages 176–181. Association for Computational Linguistics.
Durrani, N., Schmid, H., and Fraser, A. M. (2011). A joint sequence translation model withintegrated reordering. In The 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June,
2011, Portland, Oregon, USA, pages 1045–1054.
Eidelman, V., Boyd-Graber, J., and Resnik, P. (2012). Topic Models for Dynamic Translation Model Adaptation. In Proceedings of the 50th Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers), pages 115–119, Jeju Island, Korea.
Association for Computational Linguistics.
Federico, M. and De Mori, R. (1998). Language modelling. In Mori, R. D., editor, Spoken
Dialogues with Computers, pages 199–230. Academy Press, London, UK.
Gong, Z., Zhang, M., and Zhou, G. (2011). Cache-based Document-level Statistical MachineTranslation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 909–919, Edinburgh, Scotland, UK. Association for Computational
Linguistics.
Gruber, A., Weiss, Y., and Rosen-Zvi, M. (2007). Hidden topic Markov models. In Proceedingsof the International Conference on Artificial Intelligence and Statistics, pages 163–170.
Guha, J. and Heger, C. (2014). Machine Translation for Global E-Commerce on eBay. In
Proceedings of the AMTA, volume 2: MT Users, pages 31–37.
Hasler, E., Blunsom, P., Koehn, P., and Haddow, B. (2014a). Dynamic Topic Adaptation forPhrase-based MT. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 328–337, Gothenburg, Sweden. Association
for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 281

Hasler, E., Haddow, B., and Koehn, P. (2014b). Combining domain and topic adaptation for
SMT. In Proceedings of the Eleventh Conference of the Association for Machine Translationin the Americas (AMTA), volume 1, pages 139–151.
Heafield, K. (2011). Kenlm: Faster and smaller language model queries. In Proceedings of the
Sixth Workshop on Statistical Machine Translation, pages 187–197. Association for Computational Linguistics.
Hewavitharana, S., Mehay, D., Ananthakrishnan, S., and Natarajan, P. (2013). Incremental
Topic-Based Translation Model Adaptation for Conversational Spoken Language Translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 697–701, Sofia, Bulgaria. Association for Computational Linguistics.
Hofmann, T. (1999). Probabilistic Latent Semantic Analysis. In Proceedings of the Fifteenth
conference on Uncertainty in artificial intelligence, pages 289–296. Morgan Kaufmann Publishers Inc.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., et al. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive posterand demonstration sessions, pages 177–180. Association for Computational Linguistics.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting on association
for computational linguistics, pages 311–318. Association for Computational Linguistics.
Picinini, S. (2014). Challenges of Machine Translation for User Generated Content: Queries
from Brazilian users. In Proceedings of the AMTA, volume 2: MT Users, pages 55–65.
Ruiz, N. and Federico, M. (2011). Topic Adaptation for Lecture Translation Through Bilingual Latent Semantic Models. In Proceedings of the Sixth Workshop on Statistical MachineTranslation, WMT ’11, pages 294–302, Stroudsburg, PA, USA. Association for Computational Linguistics.
Sanchez, J. and Badeka, T. (2014). Linguistic QA for MT of user-generated content at eBay. In
Proceedings of the AMTA, volume 2: MT Users, pages 1–24.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of association for machinetranslation in the Americas, pages 223–231.
Su, J., Wu, H., Wang, H., Chen, Y., Shi, X., Dong, H., and Liu, Q. (2012). Translation Model
Adaptation for Statistical Machine Translation with Monolingual Topic Information. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long
Papers - Volume 1, ACL ’12, pages 459–468, Stroudsburg, PA, USA. Association for Computational Linguistics.
Xiao, X., Xiong, D., Zhang, M., Liu, Q., and Lin, S. (2012). A Topic Similarity Model for
Hierarchical Phrase-based Translation. In Proceedings of the 50th Annual Meeting of the
Association for Computational Linguistics: Long Papers - Volume 1, ACL ’12, pages 750–
758, Stroudsburg, PA, USA. Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 282

Machine Translation with
Source-Predicted Target Morphology
Joachim Daiber
Khalil Sima’an

J.Daiber@uva.nl
K.Simaan@uva.nl

Institute for Logic, Language and Computation, University of Amsterdam
Science Park 107, 1098 XG Amsterdam, The Netherlands

Abstract
We propose a novel pipeline for translation into morphologically rich languages which consistsof two steps: initially, the source string is enriched with target morphological features and thenfed into a translation model which takes care of reordering and lexical choice that matches theprovided morphological features. As a proof of concept we first show improved translation
performance for a phrase-based model translating source strings enriched with morphologicalfeatures projected through the word alignments from target words to source words. Given thispotential, we present a model for predicting target morphological features on the source string
and its predicate-argument structure, and tackle two major technical challenges: (1) How to
fit the morphological feature set to training data? and (2) How to integrate the morphology
into the back-end phrase-based model such that it can also be trained on projected (rather than
predicted) features for a more efficient pipeline? For the first challenge we present a latentvariable model, and show that it learns a feature set with quality comparable to a manually
selected set for German. And for the second challenge we present results showing that it ispossible to bridge the gap between a model trained on a predicted and another model trainedon a projected morphologically enriched parallel corpus. Finally we exhibit final translation
results showing promising improvement over the baseline phrase-based system.

1 Introduction
Translation into a morphologically rich language poses a challenge for statistical machine translation systems. Rich morphology usually goes together with relatively freer word order of thetarget language, which makes it difficult to predict morphology and word order in tandem.
Technically speaking this difficulty could be due to data sparsity, but possibly also due to morphological agreement between words over long distances. In this paper we explore the idea ofcombating sparsity by conducting translation in a probabilistic pipeline (chain rule), whereby
morphological choice may precede lexical choice and reordering.
Whenever the predicate-argument structures of the source and target strings are similar,
we expect that the linguistic information required for determining the morphological inflectionof a plausible translation resides in the source sentence and its syntactic dependency structure.
Consequently, we explore target morphology as a source-side prediction task which aims atenriching the source sentence with useful target morphological information. Practically (seeFigure 1), after word aligning the sentence pairs, we project a subset of the target morphologicalattributes to the source side via the word alignments, and then train a model to predict these
attributes on predicate-argument aspects of source dependency trees (i.e., without the source
word order).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 283

s

s′m

1. align

3. project

s′m

t′

t

s

pred. model

pred. model

ŝ′m
MT

t
MT

t

2. tag(a) Morphology projection and pred. model training.

(b) Machine translation system training.

Figure 1: Overview of the training setup and morphology projection.

Our approach differs from other approaches to predict target morphology (e.g. Chahuneau
et al. (2013)) mainly in that we predict on the source side only. A related intuition underlies
source-side reordering schemes, which have seen a surge of successful applications recently(e.g. Collins et al. (2005) or Lerner and Petrov (2013)). While syntax-driven source-side reordering assumes that source and target syntax are similar, here we make a weaker assumption,
namely that the predicate-argument structures are similar.
We explore the prediction of target morphology on the source side because we see several benefits that could potentially be exploited for further improving machine translation intomorphologically rich languages. Source-side prediction models can capitalize on the much reduced complexity of having to represent and process only the input source sentence instead ofa large lattice of target hypotheses. Hence, morphological agreement can be enforced over long
distances by morphological predictions for the full source sentence. Furthermore, while notpursued in the present work, we hypothesize that the morphological information predicted byour model can be exploited in the word alignment process.
Our contributions in this paper are three. Firstly, we report experiments to support the
hypothesis that projecting morphology to the source side could be beneficial for translation, andthen present a model for learning to predict target morphology on the source side (Section 4).
Secondly, we address how to automatically learn the set of morphological attributes that fit withthe parallel training data (Section 5). Finally, we introduce methods for integrating this new
information into a machine translation system and evaluate on a translation task (Section 6).

2 Related work
Various approaches have been proposed to the problem of translating between languages ofvarying morphological complexity. Avramidis and Koehn (2008) enrich the morphologically
impoverished source side with syntactic information and translate via a factored machine translation model. In spirit, this paper is closely related to the present work; however, while theirdecorations are source-side syntactic information (e.g. the noun is the subject), we directly predict target morphology and learn to select the most relevant properties. A similar approach, inwhich source syntax is reduced to part-of-speech tags is used successfully for translation into
Turkish (Yeniterzi and Oflazer, 2010). Following the tradition of two step machine translation(Bojar and Kos, 2010), Fraser et al. (2012) translate morphologically underspecified tokens andadd inflections on the target side based on the predictions of discriminative classifiers.
Carpuat and Wu (2007), Jeong et al. (2010), Toutanova et al. (2008) and Chahuneau et al.
(2013) propose discriminative lexicon models that are able to take into account the larger context of the source sentence when making lexical choices on the target side. These proposals
differ mostly in the way that the additional morphological information is integrated into themachine translation process. Jeong et al. (2010) integrate their lexical selection model via features in the underlying treelet translation system (Quirk et al., 2005). Toutanova et al. (2008)
survey two basic methods of integration. In the first method, the inflection prediction model is2
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 284

Translation

Word order

Lexical choice

Tags

MTR

BLEU

Kendall’s τ

BLEU-1

None (baseline)

-

35.74

15.12

45.26

49.86

Projected manual set
Projected automatic set
Projected full set

77
225
846

36.34
36.50
36.67

15.86
15.73
15.96

45.79
46.45
46.27

51.30
51.24
51.52

Training and test decor.

All translation results statistically significant against baseline at p < 0.01

Table 1: Translation with various subsets of projected morphology.

allowed to change the inflections produced by the underlying MT system. The second method is
a two step method, where the MT system translates into target-language stems, which are theninflected by the inflection model. Chahuneau et al. (2013) create synthetic phrases, i.e. phrases
with inflections that have not been observed directly in the training corpus but have been createdby an inflection model. These synthetic phrases are then added to the training data of the MT
system and marked as such. This enables the MT system to learn how much to trust them.
Finally, Williams and Koehn (2011) add unification-based constraints to the target side of astring-to-tree model. The constraints are extracted heuristically from a treebank and violations
are then penalized during decoding.

3

Morphology projection hypothesis

A morphological attribute is a morphological property of a word. Each morphological attributecan assume any of a predetermined set of values, such as {nom, acc, dat, gen} for themorphological attribute case in the German language. Further, the morphological attributes
are refined based on a set of 9 atomic parts of speech, yielding a set of morphological attributesof the form noun:case, adj:case, verb:tense, etc.
In this paper, we are interested in the question whether target morphology can be addressed
directly on the source. We hypothesize that projecting target morphological attributes and learning to predict them on source side trees can be beneficial to machine translation. To test thishypothesis we initially perform translation experiments with a standard phrase-based MT setup
with and without projected morphological information.1 These experiments provide an indication for the potential of such an approach. They do, however, not answer the question to what
extent target morphology can realistically be predicted on the source side. This question will
be addressed in the next sections. We perform translation experiments with translation systems
decorated with projected morphological attributes. In these translation systems, the target sideof the test set was processed with a morphological tagger1 and subsets of the resulting morphological attributes were projected to the source side via alignments. These experiments provide aconservative indication of the potential of this approach. They are not oracle translation experiments, but simulate an optimal target morphology prediction model. The three systems listed
in Table 1 differ only in the subset of morphological attributes they use.
The experiment is documented in Table 1. We evaluate translation quality with METEOR
and BLEU (Denkowski and Lavie, 2011; Papineni et al., 2002), word order with Kendall’s Tau(Kendall, 1938) and lexical choice with unigram BLEU. Statistical significance tests are performed for the translation scores (METEOR and BLEU) using the bootstrap resampling method(Koehn, 2004). The results show that projecting target morphological attributes improves trans1 Details of the experimental setup are provided in Section 6.3.

3
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 285

Root
Sb

Peter
.

Adv
AuxP

escaped
.

AuxA

..
from

the
.

case=datPeter

entkamderpolice.
case=dat

PolizeiFigure 2: Morphology projection and a source dependency chain.
lation. Improvements result both from better lexical choice and sometimes also better word
order. Using the full set of attributes gives the best METEOR and BLEU scores, but it also contributes significantly to data sparsity. Surprisingly, including only a small, manually selected
subset of attributes gives comparable improvement while significantly decreasing the numberof tags. This manual subset is the set of attributes selected for prediction by Fraser et al. (2012),
who found that it is beneficial to make some morphological attributes part of the translated word
stem instead of predicting them on the target side. The automatic selection is a selection of features that an automatic learning procedure determined to be the most beneficial for representing
the language pair. This selection performed equally well in our experiments.
Hence, while better translation performance is achievable by including all attributes, theprediction task also becomes significantly harder; comparable translation performance can beachieved with a small, well-chosen set of attributes. The good performance of the manual
set shows that linguistic intuition can be a good starting point for selecting this set; however, amore empirically beneficial set may be selected by enriching the source side only with attributes
which help in selecting the correct target words. The fact that the automatic set produces a better
METEOR score than the manual set further supports this intuition.2 We highlight the METEOR
scores here, since for the language pair English-to-German, METEOR has higher correlation
with human judgments than BLEU (Machacek and Bojar, 2014). Now that we established thepotential of projecting target morphology on the source side, in the sequel we aim at capitalizingon this potential. In the next section, we present our model for predicting target morphology on
source trees based on source side dependency chains.

4 Modeling target-side morphology
Since the word order of the source and target language may differ significantly, predicting morphology in a sequential, word-by-word fashion could be inadequate. We think that source syntax
and the source predicate argument structure should be informative for predicting target morphology. Hence, we propose a source-side dependency chain model P (s′m | τ, s) to predict themorphologically enriched source string s′m given a lexical dependency tree τ of s.
4.1

Source-side dependency chains

A source-side dependency chain is any path from the root of the source dependency tree toany of its leaf nodes, such as escaped→from→police→the in Figure 2. Every node with a
1-to-1 alignment to a target node is decorated with the target node’s morphological attributes.
A standard morphological tagger, such as the n-th order linear chain CRF model (e.g. Mueller
et al., 2013), would predict the attribute–value vector for each word left-to-right with a historyof n − 1 tags. Modeling with source-side dependency chains instead, gives various advantages:
Besides providing access to the morphological tags assigned to the dependency tree parent andgrandparent nodes, it implicitly encourages morphological agreement between a node and its
n − 1 ancestor nodes. The model benefits from access to the node’s syntactic role, for example
2 The difference is statistically significant at p < 0.05.

4Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 286

ManualAutomatic

All

5

6

7

5

6

7

5

Strict

50k
100k
200k

68.50
67.08
67.40

70.13
67.38
67.40

68.86
67.01
68.55

70.84
69.33
69.58

69.73
71.15
69.82

70.97
69.52
70.06

58.33
58.65
57.99

Relaxed

50k
100k
200k

72.67
70.01
69.40

70.36
71.89
69.46

72.86
69.82
69.99

74.67
72.63
71.44

71.42
72.04
70.80

71.83
72.61
69.83

62.16
62.18
60.86

Best overall F1 score highlighted in bold.

Table 2: Impact of attribute selection and model parameters on prediction quality (F1 score).
to predict grammatical case. Finally, training data sparsity is alleviated because the dependency
chain formulation allows the extraction of chains from only partially aligned sentences.
4.2

Model estimation

We estimate the source dependency chain model using the general CRF framework. In a linearchain CRF model, the probability of a tag sequence y given a sentence x is:∑
exp t,i λi · ϕi (y, x, t)∑
P (y | x) = ∑t,i λi · ϕi (y, x, t)
y exp
where t is the index of a token, i is the index of a feature and λi is the weight corresponding to
the binary feature ϕi (y, x, t). To improve training and inference time, we use a coarse-to-finepruned CRF tagger (Mueller et al., 2013). The training procedure is identical to the linear-chain
case, except that we use dependency chains instead of left-to-right chains as training examples.
The dependency chain model’s feature set is based on the set used in the linear chain CRF for
morphological tagging (Mueller et al., 2013). Additionally to the features used by Mueller et al.
(2013), we add the following feature templates: the dependency label of the current token, thedependency label of the parent token, the number of children of the current token, the sourceside part-of-speech tag of the token, and the current token’s child tokens if they are a determiner(AuxA), auxiliary verb (AuxV), subject (Sb) or a preposition (AuxP).
4.3

Intrinsic evaluation

To evaluate the quality of the source dependency chain predictions, we perform experiments
on a heldout dataset. Models are trained on a subset of the parallel Europarl data. Evaluationis performed using the F1 score of the predictions compared to the projected morphologicalattributes obtained by automatic alignment of the source and target side of the evaluation set.
Impact of model parameters Table 2 shows prediction performance of the dependency chain
model in relation to a selection of model parameters. For each morphological attribute set,
we train models of order 5, 6 and 7. All models are trained on sets of 50k, 100k and 200k
dependency chains, which are randomly sampled from the training data. In strict training mode,
we require that target words and source words connected by alignment links agree in their coarse
part of speech tags. This restriction enforces a weak form of isomorphism between the source
and the target sentence and hence limits the training set to training instances of potentiallyhigher quality. In the relaxed setup, no such agreement is enforced.
Up to a certain point, higher order models perform better than models with shorter dependency histories; however, these models are also prone to the issues of data sparsity and overfit5
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 287

ting. The results show that strict training performs worse than the relaxed training regime. The
strict training regime could possibly produce cleaner training examples; however, since it also
enforces a potentially unrealistic isomorphism between the two sentences, those examples may
also be less helpful for the final prediction.
Impact of morphological attribute selection As shown in Section 3, it is possible to reduce
the set of morphological attributes without major losses in translation quality. For the dependency chain model, smaller attribute sets are preferable since they lead to less complex models
and faster training times. Individual attributes may be difficult to predict; hence, the exact
selection of attributes is also important for prediction quality.
ManualAutomaticAll

Training time, 50k
Training time, 100k
Training time, 200k

36m
58m
1h54m

45m
82m
3h5m

77m
2h51m
6h44m

Tags
Best F1

77
72.86

225
74.67

846
62.18

Table 3: Training times and best scores for the three attribute sets.
Table 3 summarizes training times and prediction performance of the three morphological attribute sets. Larger attribute sets and more training examples lead to longer training times.
Overall, the automatic set produces more accurate results than the manual selection. Our analysis shows that this is largely due to difficult to predict verb attributes, which are included in the
manual selection but are not part of the automatically learnt set. The finding that these attributes
are hard to predict is in line with Fraser et al. (2012), who equally dropped the prediction ofverb attributes in later work.

5 Learning salient morphological attributes
Decorating the source language with all morphological properties of the target language willlead to data sparsity and will complicate the prediction task. Therefore, it is necessary to reduce this set to only morphological attributes which are helpful for a given language pair. We
consider a morphological attribute to be salient if it enables the machine translation system toperform better lexical selection. It is computationally infeasible to test all possible combinations of morphological attributes in a full machine translation system; hence, we approximate
the machine translation system’s ability to perform lexical selection with a word-based translation system given by IBM model 1 (Brown et al., 1993). Based on this simplified translation
model, the set of salient features which improve the translation performance can be chosen by
a clustering procedure.
5.1

Learning procedure

Let (s, t) be a pair of parallel sentences in source and target language. IBM model 1 provides aniterative method for estimating the translation model P (t | s) from a set of parallel sentences.
We add the morphological decoration s′m to this model. The translation model now takes thefollowing form:
P (t | s) =∑P (s′m | s)P (t | s′m )s′m ∈Θm (s)6Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 288

where P (t | s′m ) is the standard IBM model 1 formulation applied to morphologicallydecorated source tokens. In this simple machine translation model, the morphological attributes are directly attached to the source words. For example, if the English token
police is decorated with grammatical case, gender and number, it would be replaced by
the string police/case=dat+gender=female+number=singular. We define thelog-likelihood of a set of parallel sentences X to be:∏

L(X) ≡ log∑

P (t | s)P (s) =(s,t)∈X

log P (t | s) + log P (s)(s,t)∈X

Let M0 be the initial set of all morphological attributes observed in the training corpus. Our(i)goal is to find the set Mn ⊆ M0 which maximizes the likelihood of a heldout dataset. By s′m
we denote the decorated source sentence containing only the morphological attributes in Mi .
We formulate the search for the set Mn as follows:∑
Mn = arg max
log P (t | s) + log P (s)Mi ⊂M0= arg max
Mi ⊂M0= arg max
Mi ⊂M0(s,t)∈X∑

log P (t | s)(s,t)∈X

∑

(
log

∑

(i)
P (s′m | s)P (t | s′m ))s′m ∈Θm (s)(s,t)∈X

We found the estimates for P (s′m | s) using the full set of attributes M0 to be reasonable,with sufficient probability mass assigned to the most likely path. Therefore, we approximate
this model by only using the first-best (Viterbi) assignment s′′m . The final, simplified search
objective is therefore:
)
(
∑
(i)
Mn = arg max
log P (s′′m | s)P (t | s′′m )Mi ⊂M0= arg max
Mi ⊂M0(s,t)∈X∑

log P (t | s′′m(i))(s,t)∈X

The optimal set of attributes can now be determined with a clustering procedure starting fromthe full set of morphological attributes M0 . This procedure is reminiscent of Petrov et al. (2006)
since as in their work, we can simulate the removal of a morphological attribute by merging thestatistics of each of its occurrences.3
1. Initialization:− Estimate the source dependency chain model P (s′m| s), apply it to decorate thetraining and heldout set, producing T0 and H0 (datasets T and H decorated with M0 ).
(0)− Estimate P (t | s′′m(0)): perform 5 iterations of IBM Model 1 training on T0 .

2. Start with i = 0.
(i)
3. Calculate P (t | s′′m ) for each sentence pair in the heldout set Hi .
3 For example, to simulate the removal of the attribute gender, we would merge the statistics of every occurrenceof the attribute (either gender=male or gender=female). The two tags case=nom+gender=female andcase=nom+gender=male would therefore be merged into one tag case=nom.

7
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 289

Noun
Manual†gendernumbercase

Adjective

Auto

Manualgendernumber
case

Verb

Auto

†

gendernumber‡case‡

gendernumbercasedeclension

synposdegreeManual‡*numberperson‡*tense*
mode*

Other
Auto

ManualAuto--part:negative
part:subpos
punc:type
num:type† Transferred with lemma. ‡ Propagated from noun. * Dropped in later work.

Table 4: Salient attributes for English–German.
4. Find the attribute m̂ ∈ Mi , such that:
( ∑
)
(i)
(i)\m′m̂ = arg minlog P (t | s′′m ) − log P (t | s′′m)m′∈Mi(s,t)∈Hi(i)\m′

where s′′mdenotes a sentence with the attributes in Mi minus attribute m′ .
5. Merge all values of m̂ in Ti and Hi , producing Ti+1 and Hi+1 .
): Merge the t-tables containing m̂ and perfom IBM Model 1
6. Estimate P (t | s′′m
iteration on Ti+1 .
7. Repeat from (3) with i = i + 1. Stop if no possible merge improves L(Hi ).
(i+1)

5.2

Intrinsic evaluation

The complexity of the clustering procedure is O(|M | × k × l2 ) for k sentences of length l.
In practice, the procedure runs for several hours on a standard machine. Table 4 shows theattributes determined by the learning procedure. The column Auto shows the procedure’s selection and the column Manual shows the manually determined set of morphological attributes forthe same language pair, as used by Fraser et al. (2012).
Quality of the selection From inspection of these attributes, we find that our method learnsa reasonable set of salient attributes. The manual and automatic selections differ mainly inthe verb attributes, which our learning procedure removed from the final set. Morphologicalattributes in the manual selection which are marked with ‡, are attributes that in the work of
Fraser et al. (2012) were transfered as part of the translated stem by their MT system. The
symbol ‡ marks morphological attributes that they propagated from the noun (for example, anadjective’s case is copied from the noun it modifies). Finally, the verb attributes, which aremarked with * are used by Fraser et al. (2012) but found to be problematic by Cap et al. (2014b)
and dropped in later work (Cap et al., 2014a). Likewise, inspection of our model showed thatverb attributes perform badly as they may be difficult to predict. Hence, our procedure successfully learnt not to model these attributes while retaining the beneficial noun and adjective
attributes.
Granularity of the morphological attributes When simulating the removal of a morphological attribute with this learning algorithm, all of its values are merged. In some language
pairs, however, it would be useful to merge the individual values of the attributes instead. For
example, from the spelling of German nouns it is usually not recognizable whether the noun iscase=nominative or case=accusative. Hence, the algorithm should ideally be able toalso merge individual values. Since this is a straight-forward extension of our current algorithm,
we plan to evaluate this aspect in future work.
8
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 290

6 Morphology-informed translation
To leverage the morphology predictions in a machine translation decoder, we integrate this additional information into the translation model. During training and tuning, the translation model
is decorated with morphological attributes either projected from the target side or predicted byour dependency chain model.
6.1

Integration of target morphology predictions

In practice, the predicted morphological attributes on the source side can be integrated into
the machine translation system as arbitrary features based on source morphology and target
strings. In our experiments, we opted for a feature representation in which this informationis encoded as source morphology-to-target affix features. We chose this simple representation
because it is generic enough to produce improvements on the one hand and it is not prone
to overfitting on the other hand. For each phrase candidate on the source side, sparse features fire for a given sequence of source-side morphology tags and target-side string affixes.
As an example, consider the sentence Peter entkam der Polizei (Peter escaped from the police) from Figure 2. In this case, the morphological attributes gender (female), number (singular) and grammatical case (dative) would have been projected from the target to the sourceside for the phrase the police/der Polizei. When translating the source segment the police, thefeature gender=fem+number=sing+case=dat X → -er X would fire based on thepredicted morphology. This hint would help the machine translation system choose the correct
German determiner der.4
6.2 Inference strategies
At test time, the morphological decoration of the source sentence needs to be selected. This decision should ideally take into account both the predictions of our source-side dependency chain
model and the content of the phrase table, which may be decorated with projected morphology.
We compare several inference strategies. The major distinction between these strategies iswhether the machine translation system is trained and tuned on projected morphology or predicted morphology. Training on predicted morphology has the benefit that it lets the MT system
learn how much it can trust the predictions made by the dependency chain model. However, thismethod is also more laborious in system development, since it requires retraining and tuning
the whole translation system for every change in the prediction model.
Training and decoding with Viterbi predictions In the first decoding setup, which is similar
to the most common setup used in preordering, we decorate both training and test set with theViterbi decorations extracted from the dependency chain model. Specifically, for each possible
dependency chain in the source dependency tree, we preform standard CRF Viterbi tagging
starting from the root of the tree. The full training and tuning set is decorated with these singlebest predicted decorations. System training and tuning is then performed on these sets. During
test time, only the single-best Viterbi prediction is considered by the MT system.
Training on projected morphology and decoding with Viterbi predictions The projectedtraining setup differs from the previous setup in that the morphological decorations on the training and tuning set are not predicted but projected from the target side via alignments. During test
time, the decorations are predicted using single-best Viterbi predictions as in the previous setup.
While this strategy is advantageous since it simplifies the system training, the main downside ofthis strategy is that it cannot take into account possible shortcomings of the prediction model.
4 This feature example is taken form the weights of the system trained with the automatic morphological attributeset and predicted training and test decoration.

9
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 291

Translation

Word order

Lexical choice

Morph. attributes

Training decor.

MTR

BLEU

Kendall’s τ

BLEU-1

No morphology

-

35.74

15.12

45.26

49.86

Manual selection

Predicted
Projected

35.85
34.63A

15.19
14.00A

45.43
44.07

50.01
48.75

Autom. selection

Predicted
Projected

35.99A C
35.98A C

15.23B
15.22C

45.88
45.89

50.27
50.27

A

Statistically significant against baseline at p < 0.05 B Statistically significant against baseline at p < 0.06
C
Statistically significant against Manual selection at p < 0.05

Table 5: Translation with predicted test decorations.

At training time, only projected decorations are observed, which might not be realistic when
taking into account the prediction model.
6.3

Evaluation

Having introduced and evaluated the attribute selection process and the prediction of target-side
morphological attributes based on source-side dependency chains, we now turn to the evaluationof the predicted morphological information within a full machine translation pipeline.
Experimental details We use a standard phrase-based machine translation system (Cer et al.,2010) with a 5-gram language model and distortion-based reordering (dl=5). Features basedon the source morphology predictions are learnt on either the projected morphology or thepredictions of the source dependency chain model. Experiments are conducted on English–
German. Source-side dependency trees are predicted based on the HamleDT treebank (Zemanet al., 2012) using TurboParser (Martins et al., 2010). The dependency parser is trained to
produce pseudo-projective dependency trees (Nivre and Nilsson, 2005).5 The system is trainedon the full parallel sections of Europarl (Koehn, 2005) and tuned and tested on the WMT 2009and WMT 2010 newstest sets respectively.
Monolingual morphological tagging is performed using the Marmot CRF-based tagger(Mueller et al., 2013). The tagger is trained on the English and German parts of the HamleDT
treebank. The morphological attributes of both languages follow the Interset standard (Zeman,2008), which contains 45 unique attribute vectors (tags) for English and 958 for German.
Discussion Table 5 shows the outcomes of using the inference strategies presented in Section 6.2. We evaluate translation quality with METEOR and BLEU (Denkowski and Lavie,
2011; Papineni et al., 2002), word order with Kendall’s Tau (Kendall, 1938) and lexical choicewith unigram BLEU. Statistical significance tests are performed for the translation scores (METEOR and BLEU) using the bootstrap resampling method (Koehn, 2004).
The results show that both attribute selections show improvements over the baseline whentraining and testing on predicted morphology. On the other hand, when training on projectedmorphology and performing Viterbi predictions, a visible gap between the manual set and theautomatic set can be observed. This gap indicates that with the automatic set, the predictions by
the dependency chain model are closer to the projected predictions so that the machine trans5 Projectivization was performed using MaltParser version 1.8; http://www.maltparser.org/.

10
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 292

lation system learns realistic weights for the prediction part. Additionally, the system basedon the automatic selection produces a significantly better METEOR score than the system using the manual selection. As in the experiments with projected morphology, the results of thisevaluation indicate that the improvements stem from both word order choices as well as better
lexical selection. In terms of time performance, we found that the additional information does
not significantly affect the speed of the translation system. The Viterbi algorithm for predicting the target morphology is efficient and as the information is passed to the MT system assparse features, no additional complexity is added. While we have focused on the language pair
English–German, the methods presented in this paper are applicable to many other language
pairs. We therefore aim to perform additional experiments for morphologically-rich target languages such as Turkish, Arabic and Czech.

7 Conclusion
In this paper, we have explored the novel approach of target morphology projection. After
testing the idea empirically, we have put forward three proposals to realize this idea: First, we
introduced the dependency chain model for predicting arbitrary target morphology attributes
based on source dependency trees. Second, we introduced a learning procedure to determinea language pair’s set of salient morphological attributes. And finally, we have introduced andcompared various strategies for integrating this new information into a machine translation system. The experiments we have performed have provided us with important insights. They havedemonstrated that projecting a small subset of morphological attributes to the source side canprovide major translation improvements, while reducing the complexity of prediction. Furthermore, the approach for learning the useful subset performs well based both on the intrinsic
evaluation and the empirical results during prediction and translation. Given that previous work
has found it rather difficult to achieve improvements in German morphology, we consider the
improvements in METEOR score and the modest improvements in BLEU score encouraging.
While the prediction performance of the dependency chain model leaves room for improvement, we submit that our experiments sufficiently demonstrate the potential of this approach.
We plan to further improve the prediction performance of the dependency chain model withextensions such as the use of (bilingual) word embeddings that could help resolve ambiguous
cases. In addition, to let the machine translation system better exploit this new knowledge,
deeper integration (e.g. into the language model) is necessary. Both ideas constitute the main
topics for extending the current work in the future.

Acknowledgements
We thank the three anonymous reviewers for their constructive comments and suggestions. The
first author is supported by the EXPERT (EXPloiting Empirical appRoaches to Translation)
Initial Training Network (ITN) of the European Union’s Seventh Framework Programme.

11
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 293

References
Avramidis, E. and Koehn, P. (2008). Enriching morphologically poor languages for statisticalmachine translation. In Proceedings of ACL-08: HLT, pages 763–770, Columbus, Ohio.
Association for Computational Linguistics.
Bojar, O. and Kos, K. (2010). 2010 failures in English-Czech phrase-based MT. In Proceedingsof the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, WMT ’10,
pages 60–66, Stroudsburg, PA, USA. Association for Computational Linguistics.
Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. (1993). The mathematics ofstatistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–
311.
Cap, F., Fraser, A., Weller, M., and Cahill, A. (2014a). How to produce unseen teddy bears:
Improved morphological processing of compounds in SMT. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages
579–587, Gothenburg, Sweden. Association for Computational Linguistics.
Cap, F., Weller, M., Ramm, A., and Fraser, A. (2014b). CimS – the CIS and IMS joint submission to WMT 2014 translating from English into German. In Proceedings of the Ninth
Workshop on Statistical Machine Translation, pages 71–78, Baltimore, Maryland, USA. Association for Computational Linguistics.
Carpuat, M. and Wu, D. (2007). Context-dependent phrasal translation lexicons for statisticalmachine translation. Proceedings of Machine Translation Summit XI, pages 73–80.
Cer, D., Galley, M., Jurafsky, D., and Manning, C. D. (2010). Phrasal: A statistical machinetranslation toolkit for exploring new model. Proceedings of the NAACL HLT 2010 Demonstration Session, pages 9–12.
Chahuneau, V., Schlinger, E., Smith, N. A., and Dyer, C. (2013). Translating into morphologically rich languages with synthetic phrases. In Proceedings of the 2013 Conference on
Empirical Methods in Natural Language Processing, pages 1677–1687, Seattle, Washington,
USA. Association for Computational Linguistics.
Collins, M., Koehn, P., and Kucerova, I. (2005). Clause restructuring for statistical machinetranslation. In Proceedings of the 43rd Annual Meeting of the Association for Computational
Linguistics (ACL’05), pages 531–540, Ann Arbor, Michigan. Association for Computational
Linguistics.
Denkowski, M. and Lavie, A. (2011). Meteor 1.3: Automatic metric for reliable optimizationand evaluation of machine translation systems. In Proceedings of the Sixth Workshop on
Statistical Machine Translation, pages 85–91, Edinburgh, Scotland. Association for Computational Linguistics.
Fraser, A., Weller, M., Cahill, A., and Cap, F. (2012). Modeling inflection and word-formationin SMT. In Proceedings of the 13th Conference of the European Chapter of the Association
for Computational Linguistics, pages 664–674, Avignon, France. Association for Computational Linguistics.
Jeong, M., Toutanova, K., Suzuki, H., and Quirk, C. (2010). A discriminative lexicon model forcomplex morphology. In Proceedings of the Ninth Conference of the Association for MachineTranslation in the Americas (AMTA 2010).
12
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 294

Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, pages 81–93.
Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages
388–395. Association for Computational Linguistics.
Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In Proceedingsof Machine Translation Summit X, volume 5, pages 79–86.
Lerner, U. and Petrov, S. (2013). Source-side classifier preordering for machine translation. In
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pages 513–523, Seattle, Washington, USA. Association for Computational Linguistics.
Machacek, M. and Bojar, O. (2014). Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore,
Maryland, USA. Association for Computational Linguistics.
Martins, A., Smith, N., Xing, E., Aguiar, P., and Figueiredo, M. (2010). Turbo parsers: Dependency parsing by approximate variational inference. In Proceedings of the 2010 Conferenceon Empirical Methods in Natural Language Processing, pages 34–44, Cambridge, MA. Association for Computational Linguistics.
Mueller, T., Schmid, H., and Schütze, H. (2013). Efficient higher-order CRFs for morphologicaltagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language
Processing, pages 322–332, Seattle, Washington, USA. Association for Computational Linguistics.
Nivre, J. and Nilsson, J. (2005). Pseudo-projective dependency parsing. In Proceedings ofthe 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages
99–106, Ann Arbor, Michigan. Association for Computational Linguistics.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting on association
for computational linguistics, pages 311–318. Association for Computational Linguistics.
Petrov, S., Barrett, L., Thibaux, R., and Klein, D. (2006). Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics,
pages 433–440, Sydney, Australia. Association for Computational Linguistics.
Quirk, C., Menezes, A., and Cherry, C. (2005). Dependency treelet translation: Syntacticallyinformed phrasal SMT. In Proceedings of the 43rd Annual Meeting of the Association for
Computational Linguistics (ACL’05), pages 271–279, Ann Arbor, Michigan. Association for
Computational Linguistics.
Toutanova, K., Suzuki, H., and Ruopp, A. (2008). Applying morphology generation models
to machine translation. In Proceedings of ACL-08: HLT, pages 514–522, Columbus, Ohio.
Association for Computational Linguistics.
Tran, K., Bisazza, A., and Monz, C. (2014). Word translation prediction for morphologicallyrich languages with bilingual neural networks. In Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
13
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 295

Williams, P. and Koehn, P. (2011). Agreement constraints for statistical machine translation
into German. In Proceedings of the Sixth Workshop on Statistical Machine Translation,
pages 217–226, Edinburgh, Scotland. Association for Computational Linguistics.
Yeniterzi, R. and Oflazer, K. (2010). Syntax-to-morphology mapping in factored phrase-based
statistical machine translation from English to Turkish. In Proceedings of the 48th Annual
Meeting of the Association for Computational Linguistics, pages 454–464, Uppsala, Sweden.
Association for Computational Linguistics.
Zeman, D. (2008). Reusable tagset conversion using tagset drivers. In Proceedings of the Sixth
International Conference on Language Resources and Evaluation (LREC’08).
Zeman, D., Mareček, D., Popel, M., Ramasamy, L., Štěpánek, J., Žabokrtský, Z., and Hajič,J. (2012). Hamledt: To parse or not to parse? In Proceedings of the Eight International
Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European
Language Resources Association (ELRA).

14
Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 296

Improved Beam Search with Constrained Softmax
for NMT
Xiaoguang Hu
Wei Li
Xiang Lan
Hua WuHaifeng Wang

huxiaoguang@baidu.com
liwei08@baidu.com
lanxiang@baidu.com
wu hua@baidu.com
wanghaifeng@baidu.com

Baidu Inc, Beijing, China

Abstract
We propose an improved beam search decoding algorithm with constrained softmax operations
for neural machine translation (NMT). NMT is a newly emerging approach to predict the besttranslation by building a neural network instead of a log-linear model. It has achieved comparable translation quality to the existing phrase-based statistical machine translation systems.
However, how to perform efficient decoding for NMT is still challenging, especially for commercial systems which provide real-time translation service. Unlike the standard beam searchalgorithm, we use a priority queue to choose the best hypothesis for the next search, which
drastically reduces search space. Another time consuming factor is the softmax operation in
the output layer because of the large target vocabulary size. To solve this problem, we introduce
a limited word set of translation candidates to greatly reduce the computation complexity. Our
experiments show that, under the GPU environment, our method achieves a speed about 3.5
times faster than the well optimized baseline system without sacrificing the translation quality.
Our method translates about 117 words per second, beating the real-time translation requirements for practical MT systems.

1

Introduction

Neural network models have become increasingly popular in recent machine translation tasks.
Initially, the neural networks were designed for language models, such as neural network language model (Bengio et al., 2003) and recurrent neural network language model (Mikolov
et al., 2010). Devlin et al. (2014) proposed a neural network joint model to improve translation by considering the source contexts jointly. The above work integrated the neural networkmodels into the traditional translation decoders as additional features in a log-linear framework(Och and Ney, 2002). Rather than applying a neural network as a part of the existing system,
Cho et al. (2014) proposed a whole new RNN Encoder-Decoder approach which generates thetarget translation with a neural network directly. Bahdanau et al. (2014) extended the above approach by allowing an RNN model to automatically (soft-)search for parts of a source sentence
to predict a target word, and achieved a translation performance comparable to the existingstate-of-the-art phrase-based systems (Koehn et al., 2003).
Although NMT shows high potential, its decoding efficiency is still challenging. Cho et al.
(2014) implemented a standard beam search decoding algorithm (Koehn, 2004) for an RNN

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 297

Encoder-Decoder system published as GroundHog 1 . The beam search algorithm successfully
reduces the search space from exponential size to polynomial size, and it is able to translate
about ten words per second. Even though the speed is acceptable for most research tasks, it is
not yet efficient enough to meet the requirement of commercial systems for providing real-timetranslation service. Huang and Chiang (2007) proposed the forest rescoring algorithm which
succeed to accelerate the decoding by using cube pruning. The main improvement comes fromthe reduction of language model calculation which is the most time-consuming part in bothphrase-based and syntax-based system. However, there is no separable language model feature
in NMT, which makes it difficult to use the cube pruning algorithm in NMT. Furthermore, we
found that one of the most time-consuming part in NMT decoding is the softmax operation over
the output layer because of the large vocabulary size. The hierarchical softmax (Mikolov et al.,2011) and Noise-contrastive estimation (Gutmann and Hyvarinen, 2010) succeed to accelerate softmax operation effectively both for training and predicting in the RNN language model.
However, they don’t work well for the decoding application. In the language model task, weonly need to compute the probability of the certain next word which we have known in advance. But in the decoding task, we have to compute all probabilities of candidates in the target
vocabulary to know which one is the best candidate.
In this work, in order to make the NMT system meet the real-time translation requirements,
we propose an improved beam search decoding algorithm with constrained softmax operation
over the output layer for the RNN Encoder-Decoder machine translation. We found that many
hypotheses can be pruned safely without affecting the final translation result, so that we apply a priority queue to choose the best hypothesis to be extended. Furthermore, we build alimited word set of translation candidates which is used to effectively reduce the computation
complexity in softmax operation. The experimental results show that, our optimized algorithm
achieves a speed about 3.5 times faster than well optimized baseline system at the same levelof translation quality. The improved beam search with priority queue helps to reduce about
37.6% hypothesis extension operations which results in 1.7 times speedup. The constrained
softmax strategy succeeds to reduce the output layer size from 30,000 to about 300 which leads
to another 2.1 times speedup.
In section 2, we introduce the standard beam search algorithm of RNN Encoder-Decoder
approach in detail. In section 3, we present our improved beam search algorithm with constrained softmax operation. We describe our experimental settings in section 4 and report theexperimental results in section 5. Finally, we summarize our conclusions and future work in
section 6.

2

Neural Machine Translation

The end-to-end neural machine translation has been newly proposed in recent years. We first
introduce the general RNN Encoder-Decoder translation framework (Bahdanau et al., 2014),
and then we elaborate the standard beam search decoding algorithm for NMT in detail.
2.1

Preliminary Definitions

In the neural machine translation framework, a source sentence is represented as a sequence of
1-of-K coded word vectors
x = (x1 , . . . , xTx ), xi ∈ RKx
and a target sentence is also represented as a sequence of 1-of-K coded word vectors
y = (y1 , . . . , yTy ), yi ∈ RKy
1 https://github.com/lisa-groundhog/GroundHog

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 298

where Kx and Ky are the vocabulary sizes of source and target languages respectively. Tx and
Ty are the lengths of source and target sentences respectively.
Given an input sentence x and y, we define some important concepts and symbols in
advance.
• encoder: RNN to convert the input sentence x into a sequence of hidden states h.
• decoder: RNN to predict the probability distribution over yi ∈ RKy from the given context
vector and hidden states.
• hidden state: a vector from RNN hidden layer at a certain step t. Where t is defined as thenumber of accepted or generated words right now. We use ht to represent the hidden state
for the encoder, and st for the decoder.
• context vector: a vector ct used to predict translation probability at step t generated fromh and st .
• hypothesis: a data structure used to store search states includes st−1 , ct , yt , and the translation probability at step t.
• extend: an operation to construct several new hypotheses from a hypothesis with translation candidates generated from the decoder.
2.2

Encoder-Decoder Framework

Figure 1: RNN Encoder-Decoder Framework

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 299

An RNN Encoder-Decoder translation system often contains an encoder and a decoder.
Figure 1 shows how this system works to translate a Chinese sentence into an English sentence.
Given a Chinese sentence x = ”beijing de chuzuche siji hen reqing” which means ”thetaxi driver in beijing is enthusiastic”, the encoder converts x into a series of hidden statesh = (h1 , . . . , h6 ) sequentially via equation (1).
f (xj , hj−1 ) j > 0
hj =
(1)
0
j=0
where hj ∈ Rn is a hidden state of the encoder after reading input word xj , n is the hidden
layer size of the encoder, and f is a nonlinear function which is sometimes complicated suchas in the long short term memory model (Hochreiter and Schmidhuber, 1997). Furthermore,a bidirectional RNN (Schuster and Paliwal, 1997), which consists of forward and backward
RNNs, is used here.
Given the encoder hidden states h generated from x, and all the previously predicted history words (y1 , . . . , yi−1 ), the decoder predicts the probability distribution over yi ∈ RKy . For
a translation candidate y, the translation probability is computed in equation (2).
p(y) =TYp(yi |y1 , . . . , yi−1 , h)

(2)

i=1

The translation history (y1 , . . . , yi−1 ) is represented by hidden state si−1 and yi−1 . The encoder
hidden states h is represented as ci . The output probability is computed in equation (3).
p(yi |y1 , . . . , yi−1 , h) = g(si−1 , yi−1 , ci )(3)where g is a non-linear function, of which we will give an example later in section 3.2.
2.3

Standard Beam Search

We describe the beam search algorithm implemented in GroundHog as in Algorithm 1.
b =Given the encoder, decoder and input sentence x, we try to find the best translation yargmaxy p(y|x). A group of stacks are used to store hypotheses during searching. Beam size
N is used to control the search space by extending only the top-N hypotheses in the current
stack. With the above settings, the translation y is generated word by word from left to right.
We define complete hypothesis as hypothesis which outputs EOS, where EOS is a special target
word indicating the end of sentence.
As compared to the standard decoding algorithm (Koehn, 2004) in phrase-based machinetranslation, there are some differences. Firstly, a hypothesis is always of less score than the
hypothesis from which it extended. Because after every extension in NMT, the total score will
always be multiplied with a probability which is less than 1. But in phrase-based system, thescore of a extension is not always less than 1 because the features such as word penalty scoreand phrase penalty score may be greater than 1 in the log-linear framework. Secondly, the stop
conditions are different. The phrase-based system will stop searching when all the words inthe source sentence are translated. In NMT, there is no exactly word alignment information totell which word is translated or not. It will stop searching when it has generated N complete
hypotheses.

3

Improved Beam Search

In this section, we introduce our improvement over the standard beam search algorithm for
NMT. We observed that not all the hypothesis extensions are necessary, and some extensions

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 300

Algorithm 1 Standard Beam Search
1: function S TANDARD S EARCH (enc, dec, x, y)2:
define stacks s3:define set c4:create initial hypo and put it into s[0]
5:
i←0
6:
N ← beam size
7:
while s[i] 6= ∅ do8:
for all h ∈ s[i] do
9:extend new hypos from h
10:
put new hypos into s[i + 1]
11:
end for12:
prune s[i + 1] to keep N − c.size hypos
13:
move complete hypo in s[i + 1] to c
14:
i←i+1
15:
end while16:y ← trace back from best h ∈ c
17: end functionof inferior hypotheses could be skipped safely without affecting the translation results. Furthermore, we found that the operation to compute the softmax score over the output layer is very
slow because the target vocabulary size Ky is often large. We can accelerate beam search by
reducing the vocabulary size in advance with translation knowledge learned from the phrasebased model.
3.1

Hypotheses Extension with Priority Queue

The standard beam search algorithm extends the hypotheses from left to right stack by stack
as shown in Figure 2 part (a). At certain step t, all the hypotheses in the stack before t haveto be extended. Actually, not all the extended hypotheses will succeed to generate a complete
hypothesis with EOS, because plenty of extended hypotheses will be pruned in the future.

Figure 2: (a) standard beam search (b) improved beam search;  extended hypothesis, 4 best
hypothesis to be extended, ◦ unextended hypothesis

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 301

In our improved beam search algorithm, we use a priority queue q to record the best hypotheses of each stack as shown in Figure 2 part (b). In a stack, there are at most N hypotheses,
which are ranked in descending order by their translation probabilities. If there is any unextended hypothesis in a stack, the one with the highest translation probability will be inserted
into queue q. With above settings, the improved beam search works as shown in Algorithm 2.
• Firstly, we define the stacks s and priority queue q to manage hypotheses, and we create
the initial hypothesis and insert it into s[0] and q.
• Secondly, we select the best hypothesis in the queue q which is the initial hypothesis by
now, and extend at most N new hypotheses from the initial hypothesis. We insert new
hypotheses into s[1] in descending order, then insert the best hypothesis in s[1] into queue
q. We update the queue q by eliminating the initial hypothesis.
• After that, we select the best hypothesis h in the queue q which is of stack index i, andextend at most N new hypotheses from h. We insert new hypotheses into stack s[i + 1] indescending order, and prune the inferior hypotheses to make sure the number of hypothesesis less than N . For stack s[i] and s[i+1], we update the hypotheses in queue q to make surethat q contains only the best unextended hypothesis of each stack. We adjust the queue q
to be in the priority order.
• We repeat the process until the best hypothesis h to be extended in the queue is a completeb by tracing backhypothesis. It’s safe to stop searching and output the best translation yfrom h, because h is associated with the highest translation probability among any otherunextended hypotheses by now, and no hypothesis in the searching space with beam size N
will get higher translation probability than h in the future since p(yi |y1 , . . . , yi−1 , h) ≤ 1.
Instead of extending the hypothesis stack by stack sequentially, our improved beam searchalgorithm only extend the best unextended hypothesis in all stacks. The standard algorithm
stops searching after generating N complete hypotheses, while our algorithm stops when acomplete hypothesis h is selected to be extended. Because h is better than any other hypotheses
in q, and the hypothesis in q is better than any other unextended hypotheses in the same stack,and the unextended hypotheses in the stack is better than any other hypotheses extended fromit in the future. Therefore our algorithm extends less hypotheses without changing translation
result. All the unextended hypotheses inferior to h will not be extended. Considering otherbreadth-first searching algorithms with a queue such as the classical Dijksta’s algorithm, thesize of the queue grows exponentially. In our algorithm, there is at most one hypothesis in thepriority queue for each stack, and it costs less memory and more efficient to manage the queue.
Furthermore, we find that it’s unfair to compare the translation probabilities of hypotheses
in the priority queue directly. Because the hypotheses with longer translation often tend to
get lower probability. We design a parameter α to alleviate the influence of translation length,
which is similar to the weight of word penalty feature in the phrase based system.
1pe(y|x) = p(y|x) 1+α×(i−1)(4)

Where i ≥ 1 is the stack index of current hypothesis. If α = 0, then pe = p; if α = 1 the pe isthe geometry average of p over i. The lower the α is, the less hypothesis will be pruned. Afternormalization, the translation result will be changed.
3.2

Constrained Softmax Over Output Layer

For a hypothesis extension, the decoder predicts the next word probability p(yi |si−1 , yi−1 , ci )by executing RNN feed forward operation. This procedure comprises some matrix multiplication operations and a final softmax operation over the output layer as shown in Figure 3. Here,

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 302

Algorithm 2 Improved Beam Search
1: function I MPROVED S EARCH (enc, dec, x, y)2:
define stacks s3:define priority queue q
4:create initial hypo and put it into s[0], q
5:N ← beam size
6:h ← find the best hypo in q
7:
while h is not complete hypo do
8:i ← get stack index of h9:extend new hypos from h
10:
put new hypos into s[i + 1]
11:
prune s[i + 1] to keep N hypos
12:
U PDATE Q UEUE(q, s[i])13:
U PDATE Q UEUE(q, s[i + 1])
14:h ← find the best hypo in q
15:
end while16:y ← trace back from h
17: end function
18: function U PDATE Q UEUE (q,s)
19:
h←q∩s
20:
if ∃ best unextended hypo uh ∈ s then21:
replace h with uh in q
22:
else
23:
delete h from q
24:
end if25:
adjust q to be a priority queue
26: end function. hypo both in q and s. no unextended hypo in s

we compute word probability using the same g function proposed by Bahdanau et al. (2014) as
follows.
p(yi |si−1 , yi−1 , ci ) ∝ exp (yi| Wo ui )
(5)

Ky ×l

ui = [max (eui,2j−1 , uei,2j )]|j=1,...,l(6)uei = Uo si−1 + Vo Eyi−1 + Co ci

(7)

2l×n

2l×m

2l×2n

Where Wo ∈ R, Uo ∈ R, Vo ∈ R
and Co ∈ R
are weight matrices. And lis the size of maxout hidden layer ti ; n is the size of hidden layer si ; m is the word embedding
dimensionality of Eyi−1 .
Actually, we find that the matrix multiplication to compute the softmax over the output layer is often the most time-consuming operation. For most translation tasks, the sizeof output layer (Ky ≥ 10, 000) is usually over ten times larger than sizes of other layers(l, n, m ≤ 1, 000). The computation time complexity is close to O(l × Ky ). If we can reduce Ky , the hypothesis extension will become faster. RNN with output layer factorized byclass layer (Mikolov et al., 2011) and Noise-contrastive estimation (Gutmann and Hyvarinen, 2010) is able to accelerate softmax operation effectively both for training and predicting
in RNNLM application. However, unlike estimating the translation probability for only one
certain target word yi , we have to estimate the probability distribution for all target words to get
the best candidate.
Given an input source sentence x, we note that the number of possible translation candidates Tx is much smaller compared to the target vocabulary size Ky . Instead of calculating theProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 303

Figure 3: Deep Output Layer of RNN Decodertranslation probability for every word in the target vocabulary, we only calculate the probability
for a limited set of target candidates which are relevant to the given x. The problem is how
to get such a candidate set. Intuitively, we can make use of the word alignment information
generated with IBM model (Brown et al., 1993). And we found that there are often too many
candidates especially for the common words, which makes the improvement limited.
Actually, we can get more accurate translation candidates from the phrase pairs of thephrase-based translation model (Koehn et al., 2003). Firstly, we train a phrase-based translation
model with the word aligned bilingual sentence pairs. Then we segment the input sentence x
into a number of continuous source phrases. Next, we search the corresponding target phrases
for all source phrases from the phrase-based translation model, namely phrase table. After that,
we build a candidate word set by enumerating all the words in the target phrases. Finally, weconstrain the softmax operation with the candidate word set for all the hypothesis extension ofx.
One disadvantage of above approach is that it is too costly to load the whole phrase table
into memory. Fortunately, we can reduce the memory cost by pruning phrase table carefully.
The phrase table filtering techniques have been widely used in machine translation. Instead of
considering both the phrase coverage and translation features in the filtering technique, our system only considers the word coverage, which is much easier to implement. We investigated thehistogram pruning (Zens et al., 2012) and length-based pruning method to reduce the memory
used in our system. For histogram pruning, we preserves the X target phrases with highest
probability for each source phrase. For length-based pruning, we prune the phrase pairs whichcontain more than X words in source or target side. Where X is the threshold. We found thateven the basic length-based pruning method works well enough in our task.

4

Experimental Settings

In this section, we evaluate the improved beam search algorithm on the task of Chinese-toEnglish translation.
4.1

DatasetIn this paper, we use Chinese-to-English bilingual corpora from LDC which is a subset of
corpus for NIST08 task, containing 67M Chinese words and 74M English words, to train ourmodels. No monolingual corpus are used to help the model training. The NIST MT03 Chineseto-English test set is used as the development set, and NIST MT08 Chinese-to-English test set
is used to evaluate our translation results. The development set is used to choose the best RNN

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 304

model in history, because the performance of RNN model will fluctuate during training.
4.2

Toolkits

The open source SMT system Moses 2 (Koehn et al., 2007) is used to train a phrase-based
machine translation system. The phrase table trained with Moses will be used to constrain
the softmax translation. We use the open source RNN Encoder-Decoder toolkits GroundHog
which is implemented with Theano (Bergstra et al., 2010; Bastien et al., 2012) to train a neuralmachine translation model. It is re-implemented with C++ in the GPU Environment and named
NetTrans, which is well optimized and faster than GroundHog which is implemented in Python.
Given the same model, there is no translation differences between GroundHog and NetTrans.
We will report our experimental results on NetTrans instead of GroundHog.
4.3

Settings

We train a phrase-based model from the bilingual corpus with Moses using ”grow-diagfinal” word alignments. We train two RNN encoder-decoder models (set Kx =Ky =30,000 and
Kx =Ky =60,000 separately) with GroundHog toolkit written with Theano by using the default
settings proposed by Bahdanau et al. (2014). A multilayer network with a single maxout(Goodfellow et al., 2013) hidden layer is used to compute the conditional probability of eachtarget word. The size of a hidden layer n is 1000, the word embedding dimensionality m is 620
and the size of maxout hidden layer in the deep output l is 500. We use a minibatch stochastic
gradient descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train all the RNN
models. Each SGD update direction is computed using a mini-batch of 50 sentences. The
training time for each model is approximately 22 days. We set beam size N = 10 for beam
search decoding. We run both the training and decoding programs on a machine with GPU
cards (NVIDIA Tesla K10).

5Results and Analysis

5.1

Baseline

In this section, we report the translation results of Moses, GroundHog and NetTrans. Our experimental results in table 1 shows that the NetTrans runs about 3 times faster than GroundHog
with the same translation results. GroundHog is implemented with Theano, which is a very
flexible toolkit for general deep learning tasks. NetTrans is well designed especially for the
RNN Encoder-Decoder framework on the GPU architecture with pure C++. In NetTrans, we
execute the operations between the same matrices continuously to avoid additional GPU initialization and we use memory pool to avoid allocating memory frequently and store the matrix inmemory continuously.
SystemMoses
GroundHog
NetTrans

BLEU
24.19
24.50
24.50

Speed (words/sec)
11.2
33.4

Table 1: Baseline Systems
5.2

Improved Beam Search with Constrained Softmax

We describe our experimental settings as follows. Baseline is the standard beam search algorithm implemented in NetTrans. Priority represents the improved beam search algorithm with
2 http://www.statmt.org/moses/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 305

priority queue, and the parameter α (0 ≤ α ≤ 1) is set as in Equation (4). CandX represents the histogram pruning with threshold X. Our experimental results under different Alpha(0 ≤ α ≤ 1) are shown in Figure 4. Curve Priority tells the results of improved beam searchwith the priority queue. Curves Cand50 3 , Cand20 4 report the results of constrained softmax with different CandX. As shown in Figure 4, both improved beam search and constrained
softmax can accelerate the decoding speed effectively.

(a) BLEU Score with Different α(b) Translation Speed (words/second) with Different α

Figure 4: Results of Improved Beam Search
As shown in Table 2, our method achieves a speed more than 3.5 times faster than the baseline at the same level of translation quality with α=0.2. The improved beam search with priority queue achieves in 1.7 times speedup by reducing 37.6% hypothesis extension operations.
The hypothesis extension times both for our method and the baseline method (exto and extb )are counted respectively. The pruning ratio is computed with equation (extb − exto )/extb .
The results show that the speed acceleration with different α is roughly consistent with thepruning ratio. The constrained softmax strategy leads to another 2.1 times speedup by reducing the output layer size from 30,000 to about 300. Theoretically, the speedup should be
Ky × n/max{n × n, Ky × n} and thus 30 times faster with hidden layer dimension n = 1000.
Actually, the standard softmax costs about half of the decoding time under the GPU environment. After constraining it with a candidate word set, it makes the constrained softmax no3 We use the default settings in Moses that performs well for phrase-based MT with CandX=50.
4 We use a reduced candidate set to examine how different candidate sizes affects translation quality

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 306

longer the bottleneck of translation. At the same level of translation speed, our method performs more than 1 BLEU scores better than the method to reduce the beam size N from 10 to 3
directly.
SystemBaseline
α=0
α=0.2
α=0.2,Cand50
α=0.2,Cand20
Beam Size=3

BLEU
24.50
24.50
24.29
24.29
24.44
23.37

Speed
33.4
41.11
55.4
111.6
117.2
104.4

Ratio
14.6%
37.6%
42.8%
45.1%
66.6%

Table 2: Optimized Search with Pruning Ratio

5.3

Constrained Candidates Selection

We define PhraseX as the length-based pruning with threshold X. Cands is the average word
size of translation candidates set for a given sentence. Coverage represents the recall ratio of thereal candidates generated by the baseline RNN system (α=0.2). Memory means the additional
memory used to store the whole phrase table for candidates selection.
As shown in Table 3, the Coverage decreases when reducing CandX, while Coverage almost remains the same when reducing PhraseX. For example, when we only select those phrase
pairs whose length is less than or equal to 2 (PhraseX=2), the coverage is almost the same asthat when PhraseX=7. We find that our method works well with very limited memory (170MB)
in setting Cand50 and Phrase2. This result is consistent with that for phrase-based SMT system, where phrase pairs with 2 or 3 words are frequently used and the target candidates of thesource phrase is set to 50 by default. Moreover, with such setting, the translation quality is notdegraded in BLEU score metrics. This result shows that our method for constrained candidate
selection is effective to speed up the decoder without degrading of translation quality and withlimited memory.
Selection
Cand5,Phrase7
Cand10,Phrase7
Cand20,Phrase7
Cand50,Phrase7
Cand50,Phrase3
Cand50,Phrase2
Cand50,Phrase1

Cands
83.8
126.6
195.4
352.5
350.9
337.6
276.3

Coverage
87.3%
91.4%
94.0%
96.2%
96.2%
96.1%
95.3%

Memory2.26GB
2.29GB
2.31GB
2.33GB
571MB
170MB
12MB

Table 3: Constrained Candidates Selection

5.4

The Effect of Vocabulary Size

We train a RNN Encoder-Decoder model with the target word size set to Ky =60,000, to examine
how our method affects the decoding speed under different settings. As shown in Table 4, as
the vocabulary size becomes larger, our approach performs even better, achieving a speed about
5.6 times faster than the baseline. The result is consistent with our intuition.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 307

SystemBaseline
α=0
α=0.2
Cand50
Cand20

BLEU
24.35
24.35
24.19
24.05
24.41

Speed (words/sec)
21.3
27.1
38.6
113.8
119.8

Table 4: Results of Vocabulary Size = 60,000

6Conclusion and Future WorkIn this paper, we propose an improved beam search decoding algorithm with constrained softmax operation for the neural network translation. It accelerates the translation speed more than
3.5 times compared to the standard beam search algorithm as in our well optimized baseline
while keeping the same translation quality. Our experiments show that the improved beam
search algorithm and the constrained softmax operation over the output layer are very effective
for neural network translation.
In the future, we will try to accelerate the training for NMT using constrained softmax
strategy. Furthermore, we will try to reduce the searching errors by selecting more accurate
bilingual translation candidates.

Acknowledgments
This work is supported by the National Basic Research Program of China (973 program No.
2014CB340505).

References
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align andtranslate. In arXiv:1409.0473 [cs.CL].
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., VardeFarley, D., and Bengio, Y. (2012). Theano: new features and speed improvements. In Deep Learningand Unsupervised Feature Learning NIPS 2012 Workshop.
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model.
Journal of Machine Learning Research 3, pages 1137–1155.
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley,D., and Bengio, Y. (2010). Theano: a cpu and gpu math expression compiler in python. In Proceedingsof the Python for Scientific Computing Conference (SciPy).
Brown, P. F., Pietra, S. A. D., Pietra, V. J. D., and Mercer, R. L. (1993). The mathematics of statisticalmachine translation: Parameter estimation. Computational Linguistics, 19(2):263–311.
Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machinetranslation: Encoderdecoder approaches. In Proceedings of Eighth Workshop on Syntax, Semantics andStructure in Statistical Translation (SSST-8), pages 103–111, Doha, Qatar.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust neuralnetwork joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of
the Association for Computational Linguistics, pages 1370–1380, Baltimore, Maryland, USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 308

Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks.
In Proceedings of The 30th International Conference on Machine Learning, pages 1319–1327.
Gutmann, M. and Hyvarinen, A. (2010). Noise-contrastive estimation: A new estimation principle forunnormalized statistical models. In Proceedings of the 13th International Conference on Artificial
Intelligence and Statistics (AISTATS).
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, pages 9(8),1735–1780.
Huang, L. and Chiang, D. (2007). Forest rescoring: Faster decoding with integrated language models. In
Proceedings of ACL.
Koehn, P. (2004). Pharaoh: a beam search decoder for phrase-based statistical machine translation models.
In Proceedings of the Sixth Conference of the Association for Machine Translation in the Americas,
pages 115–124.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W.,
Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source
toolkit for statistical machine translation. In ACL 2007 demonstration session.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of
HLT-NAACL 2003, pages 127–133.
Mikolov, T., Karafiat, M., Burget, L., Cernocky, J. H., and Khudanpur, S. (2010). Recurrent neural network
based language model. In Proceedings of Interspeech.
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J. H., and Khudanpur, S. (2011). Extensions of recurrent
neural network language model. In Proceedings of ICASSP.
Och, F. J. and Ney, H. (2002). Discriminative training and maximum entropy models for statistical machinetranslation. In Proceedings of ACL.
Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on
Signal Processing, pages 45(11), 2673–2681.
Zeiler, M. D. (2012). Adadelta: An adaptive learning rate method. In arXiv:1212.5701 [cs.LG].
Zens, R., Stanton, D., and Xu, P. (2012). A systematic comparison of phrase table pruning techniques.
page 972983.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 309

Bilingual Distributed Phrase Representations for
Statistical Machine Translation
Peyman Passban
Chris Hokamp
Qun Liu

ppassban@computing.dcu.ie
chokamp@computing.dcu.ie
qliu@computing.dcu.ie
ADAPT Centre, School of Computing, Dublin City University, Ireland

Abstract
Phrase–based machine translation (PBMT) relies upon the phrase-table as the main resourcefor bilingual knowledge at decoding time. A phrase table in its basic form includes aligned
phrases along with four probabilities indicating aspects of the co-occurrence statistics for eachphrase pair. In this paper we add a new semantic similarity score as a statistical feature toenrich the phrase table. The new feature is inferred from a bilingual corpus by a neural network (NN), and estimates the semantic similarity of each source and target phrase pair. We
observe a significant increase in system performance with the addition of the new feature. We
evaluated our model on the English–French (En–Fr) and English–Farsi (En–Fa) language pairs.
Experimental results show improvements for all translation directions of En↔Fr and En↔Fa.

1Introduction

Phrase–based machine translation begins by segmenting a source sentence into phrases andlooking up candidate translations for the sub-sentential phrases in a phrase table. The goal of
PBMT is to assemble these translation candidates into the optimal target sequence. Finding
the best source segmentation, translation candidates, and phrase ordering is a search problem
which is typically formulated as a log-linear model using both dynamic and static features,
whose weights are optimized via a search heuristic on held-out development data. The inverse
phrase translation probability ϕ(f |e), inverse lexical weighting lex(f |e), direct phrase translation probability ϕ(e|f ) and direct lexical weighting lex(e|f ) are four of the standard staticfeatures used in phrase tables. These features are computed directly from the co-occurrenceof aligned phrases in the training corpora. However, co-occurrence information alone cannot
capture semantic information about phrases, especially when they are taken out of context.
Therefore, many techniques have been proposed to enrich the feature list by including features
which contain syntactic and/or semantic information (Banchs and Costa-jussà, 2011).
Most work evaluating the inclusion of semantic information into SMT decoders has focused upon adding dynamic features (those which must be computed at decoding time). However, dynamic features require the implementation of a new feature function which depends
upon the hypothesis data structure of the particular decoder. The implementation of dynamic
features typically requires significantly more engineering effort than simply augmenting thephrase table. In this paper, we add a new static feature and show how a good static featurealone can significantly boost translation quality. The basic idea behind our work is to use thevector representation, or semantic embedding of phrases, which is generated by an NN. The
semantic features of the source and target languages are projected into a shared bilingual space,

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 310

which preserves both semantic and syntactic information about the phrases. The supervised
approach to generating embeddings from neural networks optimizes the network to produce
vectors which are good with respect to a specific objective. As an example, if the goal is to do
the sentiment analysis or sentence clustering, a vector should reflect the polarity of a sentence —whether is positive or negative — or its closeness to a specific distribution (Kalchbrenner et al.,2014). In tasks like translation, a good vector should reflect semantic and syntactic information
about original constituent (sentence, phrase or word) in addition to contextual knowledge fromits surrounding words, and ideally some information which will make it easier to map into thetarget language.
Methods like word2vec1 (Mikolov et al., 2013a) or that of (Le and Mikolov, 2014) produce
general-purpose vector representations which can be leveraged for a variety of downstream
applications. As the word and sentence vectors encode syntactic and semantic information, they
are potentially useful for translation tasks. In pure neural MT engines, the word embeddings
are trained as parameters of the model, which generally attempts to maximize the likelihoods(Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014), and then used directly to perform
translation. Another line of work has tried to make use of distributed representations within theclassic MT pipeline (Gao et al., 2013; Devlin et al., 2014).
In this work, we prepare an enhanced bilingual corpus which includes the source and target
phrases extracted by the alignment model, along with the original source and target sentences,
and a set of word pairs which are direct translations of one another. An NN is used to generate
embeddings for the each of the phrases. The generated vectors reflect the similarity of phrases inthe same language as well as their relevancy to the phrases of the other language. In the training
data, monolingual distributional information for each language is contained in the sentences,while bilingual information is conveyed by the bilingual phrase pairs and word pairs.
Our contributions in this paper are twofold. a) We extract a novel static feature from abilingual corpus which boosts the translation quality. Most previous work has added new dynamic features which significantly increase computational overhead — our simple static feature
can achieve comparable improvements with less effort. b) Our network extracts the vectors
from a bilingual corpus. To the best of our knowledge, related research has modeled the source
and target vectors separately in isolated spaces and focused upon finding a means to transformthe source & target representations into comparable forms. We extracted vectors in the joint
space and tried to capture information of the both target and source sides in a single vector.
Inferred feature caused considerable improvements and showed that, a good static feature canperform as efficiently as a dynamic one. The structure of paper is as follows. Section 2 gives an
overview of related work, and tries to show why word, phrase, and sentence vectors are usefulfor MT purposes. Section 3 explains the network architecture in detail. In Section 4, experimental results are reported for two language pairs: En–Fr and En–Fa. Finally, in the last section
we present our conclusions and discuss some avenues for future work.

2

Background

Using vector representations for textual data (word, phrase, sentence, etc. . . ) is not a newidea. The concept of continuous, distributed representations for text tokens was introducedby the Vector Space Model of Salton et al. (1975) and expanded by techniques such as like
Latent Semantic Analysis (Deerwester et al., 1990), Latent Dirichlet Allocation (Blei et al.,2003), and Random Indexing (Sahlgren, 2005). Real valued, continuous representations are
straightforward to use within Machine Learning models, and have contributed to the current
state-of-the-art in many NLP tasks. However, a disadvantage of distributed representations is
1 http://code.google.com/p/word2vec/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 311

that they are typically not decomposable — i.e. columns often do not correspond to intuitive
features, thus models must be evaluated on a task-by-task basis.
The techniques mentioned in the previous paragraph build vectors from bag-of-words(BOW) representations, discarding structural dependencies and word order, so they cannot notreflect semantic information which depends upon syntactic structure. To compensate for these
deficiencies, a more recent line-of-work uses neural networks to generate vectors which can
encode local distributional information, as well as syntactic structure and word order in somecases. Hinton (1986) used NN for text modelling for the first time. Recently, neural networks
have achieved state-of-the-art performance in many areas of NLP due both to the developmentof new learning algorithms and to the availability of computational resources such as GPUs.
Word2vec (Mikolov et al., 2013a), and word2vec inspired works (Wolf et al., 2014) have beensuccessfully applied to a wide variety of NLP tasks. The following paragraph focuses uponwork that leverages these vectors for MT purposes.
A basic but successful application of NN-based word vectors for MT was reported in(Mikolov et al., 2013b). They project words of the source language into vectors and do thesame with words of target language. Then try to find a transformation function which maps thesource semantic space into the target semantic space using a small set of word pairs known to behigh-quality translations. The model significantly reduces the volume of bilingual data required
to train such systems and this is the main advantage of their approach. The model works onmonolingual data and only needs a small number of parallel words to make the bridge between
languages. The cross-lingual transformation allows an MT system to search for translations
for OOV (out-of-vocabulary) words by consulting a monolingual index which contains words
that were not observed in the parallel training data for the MT system. Garcia and Tiedemann(2014) and Dinu and Baroni (2014) are other examples of approaches which leverage NN-based
word vectors for translation tasks. They focus upon exploiting similarities at the word level, but
MT encompasses more than just word-level translation. To extend the application of text embeddings beyond single words, Gao et al. (2013) proposed learning embeddings for source andtarget phrases by training a network to maximize the sentence-level BLEU score. The outcome
is a set of vectors for phrases, and the similarity between each phrase pair vectors is used as adynamic feature function in the log–linear model at decoding time. In another work Costa-jussa
et al. (2014) tried to find the similarities among source sentences and incorporate source side
contextual information into the decoding. Some other models try to re-score the phrase table orinfer new phrase pairs to address the OOV word problem in order to improve translation quality(Alkhouli et al., 2014; Costa-Jussà and Banchs, 2011). Zhao et al. (2015) did the same using
monolingual datasets and extended the phrase table.

3

Learning Embeddings for Phrases

Our model extends the document vectors of Le and Mikolov (2014) to bilingual texts. By
including both monolingual and bilingual ‘documents’ into the training data, we learn a distributed representation for both languages simultaneously. In the method proposed by Le and
Mikolov (2014), documents are treated as atomic units in order to learn an embedding with thesame dimensionality as the vectors for the individual words in the model. We adapt this idea
to sentences and phrases, where phrases are presented both as monolingual and as bilingual
documents.
Le and Mikolov (2014) create a new vector for each document, which in our case may be
a monolingual sentence, a monolingual phrase, a bilingual phrase pair, or a bilingual word pair.
During training, the document vector is concatenated with the vectors for individual words topredict the surrounding words in the given unit of text. Intuitively, we expect document vectors
to be representative of the semantic content of the entire unit of text, while word vectors are

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 312

“ on ”+“ the cat sat on …”“ the““ cat““ sat“Figure 1: Network architecture for jointly learning sentence and word vectors
representative of all of the contexts where the word occurs. During training, word vectors andsentence/phrase vectors are updated until the cost is minimized. The model learns a semantic
space where constituents with similar distributional tendencies tend to have similar vectors.
More formally, given a sequence of tokens c1 , c2 , ..., cn the objective is to maximize the averagelog probability of a word given its context:n−l1X
log p(ci |ci−l , ..., ci+l )ni=lUsing the standard terminology for NN models, the training objective can be explained asfollows. The cost function is defined according to the average log probability. Values for the
probabilities can be calculated using a multiclass classifier such as softmax:
eywt
p(ci |ci−l , ..., ci+l ) = P yj
jewhere yj is the output for the input word wj in which:y = b + W h(ci−l , ..., ci+l )W and b are parameters for the Softmax function, and h represents the output of one or more
hidden layers. The network is trained by stochastic gradient decent and back-propagation(Rumelhart et al., 1988) to obtain the set of constituent vectors C, and network parameters,
W and b.
In our case, the network includes one hidden layer with 100 nodes which is fed by abilingual corpus. The Corpus includes a) source and target sentences which are those we used
to train our SMT engine, b) phrase pairs and c) bilingual lexicon. Both the phrase and lexicon
sets are extracted by Moses (Koehn et al., 2007). Specifically, each line of corpus contains asentence in one of the languages, a phrase pair, or a tuple of words.
As it has been shown in (Huang et al., 2012), word vectors can be affected by the word’ssurroundings as well as by the global structure of a text. Accordingly, by using a training corpus with some bilingual examples we expect to learn a semantic space which contains bothlanguages. Each unique word has a specific vector representation. Clearly similar words in thesame language would have similar vectors (Mikolov et al., 2013a). Words that are direct translations of each other (same meaning with different languages) should also have similar vectors.
As the corpus contains the tuples of < wordL1 , wordL2 >, equal words are connected together.
By the same logic, phrasal units are also connected together. During decoding, the sentences
in a good translation pair should be built from similar sub-units, indicating the semantic compatibility of the constituent phrases. Table 1 shows the most similar phrases and words for two

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 313

examples. The items that were originally in Farsi have been translated into English, and are
indicated with italics.
Query
1
2
3
4
5
6
7
8
9
10
Query
1
2
3
4
5
6
7
8
9
10

we can’t let them win
you could everwhat the bloody hell is that .
as you know it .
let her go .
he is lying . no i am not .
to be
he wonyou !?
you got that .
they are aheadsadness<apprehension, nervous>
emotion<ill , sick>pain<money , money>
benignity<may he was punished , punished harshly>is really gonna hurti know toms dying<bitter , angry >Table 1: The top 10 most similar vectors for two queries: an English phrase, and a Farsi word.
Recall that the index includes vectors for words in both languages, phrases in both languages,
sentences in both languages, bilingual phrase pairs, and bilingual lexical pairs. Farsi words areindicated in italics.
The pipeline for adding our semantic feature to the phrase table is very straightforward.
We have a set of vectors for the phrases of both languages. Each phrase is modelled with a 100dimensional vector. The phrase table is scanned sequentially, and for each phrase pair, related
vectors are fetched, then their similarity is estimated. To measure the similarities we use thecosine metric, which is:v1 .v2similarity(v1 , v2 ) =||v1 || ∗ ||v2 ||Values from the cosine similarity are in the range [-1,1].We map the similarity scores to therange [0,1] before adding them to the phrase table. After including the new feature, the weights
for the log-linear model are learned by using held-out data.

4

Experimental Results

To study the impact of our new feature, we selected two datasets. One is the TEP++ corpus(Passban et al., 2015) which is a collection of 600K parallel En–Fa sentences. Farsi is a particularly interesting language for new MT research because it is both low resource and morphologically rich. The state-of-the-art MT quality for Farsi is not advanced relative to languages with

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 314

more training data available. The other dataset is the WMT2 Fr-En corpus. The WMT corpora
are frequently used as the de facto standard test datasets for SMT systems.
In all of the experiments Moses (Koehn et al., 2007) is used to build the SMT engines.
BLEU (Papineni et al., 2002) is the evaluation metric and the feature weights are tuned by
MERT (Och, 2003). All language models are built using SRILM (Stolcke et al., 2002). The
improvements in translation quality are statistically significant according to the results of paired
bootstrap re-sampling (Koehn, 2004).3 The experimental setup is shown in the Table 2 andresults from the baseline and extended systems are in Table 3.
Language pair
DatasetLanguage model
MT engine

En–Fr
Europarl corpus (Koehn, 2005) version 7 is used as a bilingual training set. As a dev set 2000 sentences of news-test2014 were used
and the test set is the test set of the
WMT2015 shared translation task.

En–Fa
The training, test and dev
sets are subsets of the
TEP++ corpus consistingof 575191, 2000 and 1000
sentences respectively. All
the sentences have been
selected randomly.
5-gram language models trained on the monolingual part of the
training sets.
statistical phrase based engine with default Moses configuration
Table 2: Experimental setup

As the results show, the new feature leads to improvements in all directions. We anticipatedthese improvements, as the positive impact of distributed phrase representation has already beenshown by Gao et al. (2013). However, our work consider two new aspects of the problem. We
train the vectors in a shared bilingual space, and show that proposed model can generate similar
vectors for similar/equivalent constituents, even for languages pairs such as En-Fa, which are
not typologically similar. We also show that a simple but efficient static feature can improvetranslation quality.

Baseline
Extended
Improvement

En–Fr
29.91
30.62
+0.71

Fr–En
27.31
27.95
+0.64

Fa–En
29.21
29.72
+0.51

En–Fa
21.03
21.44
+0.41

Table 3: Results for base-line and extended systems
The new semantic similarity feature causes, on average, +0.56 enhancement in terms of
BLEU for all of the directions of En↔Fr and En↔Fa, and as the size of training data increases
the method provides even better performance. Improvements for the En–Fr pair demonstratethat achievements of the model are valid for large datasets, and improvements for the En–
Fa pair show that the model can be used to translate distant language pairs. The word orderand structure of the Farsi and English languages are quite different from each other, and Farsi
is a morphologically rich language, making translation more difficult than for closely related
language pairs such as En-Es.
2 http://www.statmt.org/wmt15/translation-task.html
3 We used ARK research group codes for statitical significance testing for 1000 samples with parameter of 0.05,
http://www.ark.cs.cmu.edu/MT/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 315

5Conclusion and Future work

In this work we presented a new bilingual semantic similarity feature obtained from a neuralnetwork that is trained on a bilingual corpus, and computes the distributed representation ofphrases in a shared semantic space. Each phrase is projected into a vector and the similarity ofthe vectors for each phrase pair is estimated. The similarity score for the phrase pair is added
as a new phrase table feature, and the MT engine is tuned according to the default featuresin addition to new one. This augmentation of the information in the phrase table provides
improvements in translation quality.
The method is quite straightforward and does not impose any significant overhead to thebaseline SMT pipeline. Distributed vector representations preserve the semantic information of
the constituents as well as their order and structural dependencies. The bilingual examples in the
training data create dependencies between the equivalent constituents from different languages.
As the model connects the phrases of two different languages to each other, it implicitly includes
contextual information about the phrase pair into the MT process. Our next goal is to incorporate
information from the source and target sides at decoding time. Although our model provides a
global measure of the quality of a phrase pair, we cannot use the current framework to do tasks
like disambiguation, because our features are static. We hope to incorporate the knowledge
from paragraphs and text segments that the source and target phrases are extracted from, andcompare this information to the context of the phrase at decoding time in order to provide adynamic means of computing cross-lingual similarity.

Acknowledgement
We would like to thank the three anonymous reviewers and Rasul Kaljahi for their valuablecomments. This research is supported by Science Foundation Ireland through the CNGL Programme (Grant 12/CE/I2267) in the ADAPT Centre (www.adaptcentre.ie) at Dublin City University.

References
Alkhouli, T., Guta, A., and Ney, H. (2014). Vector space models for phrase-based machinetranslation. Syntax, Semantics and Structure in Statistical Translation, page 1.
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning toalign and translate. In Proceedings of International Conference on Learning Representations,
San Diego, USA.
Banchs, R. E. and Costa-jussà, M. R. (2011). A semantic feature for statistical machine translation. In Proceedings of the fifth workshop on syntax, semantics and structure in statisticaltranslation, pages 126–134, Portland, Oregon, USA.
Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent dirichlet allocation. the Journal ofmachine Learning research, 3:993–1022.
Costa-jussa, M., Gupta, P., Rosso, P., and Banchs, R. (2014). English-to-hindi system description for wmt 2014: Deep sourcecontext features for moses. In Proceedings of the Ninth
Workshop on Statistical Machine Translation. Association for Computational Linguistics,
Baltimore, Maryland, USA.
Costa-Jussà, M. R. and Banchs, R. E. (2011). A vector-space dynamic feature for phrase-based
statistical machine translation. Journal of Intelligent Information Systems, 37(2):139–154.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 316

Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., and Harshman, R. A. (1990).
Indexing by latent semantic analysis. JAsIs, 41(6):391–407.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast androbust neural network joint models for statistical machine translation. In Proceedings of
the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages
1370–1380, Baltimore, USA.
Dinu, G. and Baroni, M. (2014). Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568.
Gao, J., He, X., Yih, W.-t., and Deng, L. (2013). Learning semantic representations for thephrase translation model. arXiv preprint arXiv:1312.0482.
Garcia, E. M. and Tiedemann, J. (2014). Words vector representations meet machine translation.
Syntax, Semantics and Structure in Statistical Translation, page 132.
Hinton, G. E. (1986). Learning distributed representations of concepts. In Proceedings of theEighth Annual Conference of the Cognitive Science Society, Amherst, USA.
Huang, E. H., Socher, R., Manning, C. D., and Ng, A. Y. (2012). Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual
Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages
873–882. Association for Computational Linguistics.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In EMNLP,
pages 1700–1709.
Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (2014). A convolutional neural network
for modelling sentences. In Proceedings of Association for Computational Linguistics (ACL2014), Baltimore, USA.
Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In EMNLP,
pages 388–395.
Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In MT summit,
volume 5, pages 79–86.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., et al. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive posterand demonstration sessions, pages 177–180. Association for Computational Linguistics.
Le, Q. V. and Mikolov, T. (2014). Distributed representations of sentences and documents. In
Proceedings of ICML, Beijing, China.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). Efficient estimation of word representations in vector space. In ICLR Workshop, Scottsdale, AZ, USA.
Mikolov, T., Le, Q. V., and Sutskever, I. (2013b). Exploiting similarities among languages formachine translation.
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedingsof the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages
160–167, Sapporo, Japan.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 317

Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting on association
for computational linguistics, pages 311–318. Association for Computational Linguistics.
Passban, P., Way, A., and Liu, Q. (2015). Benchmarking smt performance for Farsi unisng the
TEP++ corpus. In Proceedings of the 18th annual conference of the European Association
for Machine Translation (EAMT), pages 82–88, Antalya, Turkey.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1988). Learning representations by backpropagating errors. Cognitive modeling, 5:3.
Sahlgren, M. (2005). An introduction to random indexing. In In Methods and Applications of
Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering, TKE 2005.
Salton, G., Wong, A., and Yang, C.-S. (1975). A vector space model for automatic indexing.
Communications of the ACM, 18(11):613–620.
Stolcke, A. et al. (2002). SRILM-an extensible language modeling toolkit. In INTERSPEECH.
Wolf, L., Hanani, Y., Bar, K., and Dershowitz, N. (2014). Joint word2vec networks for bilingual
semantic representations. International Journal of Computational Linguistics and Applications, 5(1):27–44.
Zhao, K., Hassan, H., and Auli, M. (2015). Learning translation models from monolingual
continuous representations. In Proceedings of the Ninth Workshop on Statistical MachineTranslation. Association for Computational Linguistics, Baltimore, Maryland, USA.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 318

Improved Search Strategy for Interactive
Predictions in Computer-Assisted Translation
Fatemeh Azadi
Department of Computer Engineering,Amirkabir University of Technology, Tehran, Iran
Shahram Khadivi1
eBay Inc, Aachen, Germany

ft.azadi@aut.ac.ir
skhadivi@ebay.com

Abstract
The statistical machine translation outputs are not error-free and in a high quality yet. So in
the cases that we need high quality translations we definitely need the human intervention.
An interactive-predictive machine translation is a framework, which enables the collaboration of the human and the translation system. Here, we address the problem of searching thebest suffix to propose to the user in the phrase-based interactive prediction scenario. By
adding the jump operation to the common edit distance based search, we try to overcome
the lack of some of the reorderings in the search graph which might be desired by the user.
The experiments results shows that this method improves the base method by 1.35% in
KSMR2, and if we combine the edit error in the proposed method with the translation scores
given by the statistical models to select the offered suffix, we could gain the KSMR improvement of about 1.63% compared to the base search method.

1. Introduction
Although the significant improvements achieved in the field of statistical machine translation,
the current models and therefore the systems which can be built from them are still far fromperfect. So, in order to achieve good or even acceptable translations, manual post editing isneeded. An alternative to this approach is given by the interactive predictive machine translation paradigm (Barrachina et al., 2009). Under this paradigm, translation is considered as an
iterative process where the human translator and the computer collaborates to generate thefinal translation, and in this way both the human and the computer can help each other.
In the interactive predictive MT scenario, the system initially offers a translation for asource sentence. From this translation, the user could mark a prefix as correct and begins totype the rest of the target sentence as it’s desirable for him. By typing the single next character (or maybe the full next word, according to the system settings) the interactive MT system
would suggest a new suffix that completes the user’s confirmed prefix with the new characterthe user has just typed. This procedure continues iteratively until the user accepts the translation to be complete and correct.
One of the important issues in the development of interactive predictive MT systems isthe strategy to search and offer the best suffix that completes the prefix confirmed by the user.
One of the most common ways for phrase-based translation systems is to use the search graphthat is obtained once during the translation process, and complete each user’s confirmed pre12This work has been done when the author was with Amirkabir University of Technology.
Keystroke and mouse-action ratio

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 319

fix by searching the best path which has a prefix with highest matching to the user’s prefix,
and offering the rest of it to the user. The most important advantage of using the pre-built
search graph in each interaction is that it leads to an efficient response time for each interaction.
The problem in searching the best system offer arises when the prefix confirmed by theuser is not producible by the statistical models of the translation system. That is it could not
be produced by any of the paths in the search graph, so that the interactive predictive MT system could not offer any completion to the user. A common solution to this problem is thatinstead of searching a path in the graph whose prefix matches exactly with the user’s prefix,
we search more freely, for a path which has the most similarity with the user’s prefix. Usually, the edit distance similarity measure is used for this purpose.
The search graph may not contain all of the possible reorderings for a certain set ofwords, due to the weakness of translation and reordering models or the pruning done during
the generation of the search graph. So, the user’s desired translation might have a reordering
which doesn’t exist in any of the search graph paths and the search based on edit distancecould not perform well in these cases, since it aligns the user’s prefix and a prefix of each pathin the search graph monotonically and with no reordering.
In this paper we try to solve this problem for phrase-based interactive MT systems, byadding the jump operation in computing edit distance, which allows jumping over some partsin each path while aligning with the user’s prefix, and then offering those parts as the suffix tothe user; i.e. shifting some parts to a position after user’s prefix, so that it changes the reordering of the system’s translation.
Also, we compare and combine this method with another improvement done to the edit
distance based search in (Vakil and Khadivi, 2012), which uses the weighted sum of the edit
distance and the translation score for each path for selecting the best suffix.
In section 2 the related works done in the computer assisted translation field are reviewed. In section 3, we briefly explain the statistical framework for the interactive predictive
MT. Then in section 4, the searching algorithms for finding the best suffix are discussed andthe proposed method in this paper is presented. Finally, the experiments done for the evaluation of our method is presented in section 5.

2. Related WorkInteractivity in CAT3 has been explored for a long time. Systems have been designed to interact with human translators in order to solve different types of (lexical, syntactic, or semantic)
ambiguities (Slocum 1985; Whitelock et al. 1986).
A major change in the interactive CAT technology was done within the TransTypeproject in (Foster et al. ,1997; Langlais et al., 2000). A new approach called Target-Text
Mediation was proposed, in which the focus of the interaction was shifted from the source
text to the production of the target text directly. This project with TransType2 (TT2) (Bender
et al., 2005; Tomás and Casacuberta 2006; Barrachina et al., 2009) proposed to embed an MT
system in an interactive scenario. This way, the human translator can ensure a high quality
output while the MT system ensures the significant gain of productivity.
Particularly the interactive-predictive machine translation paradigm was proposed in(Barrachina et al., 2009). In this paradigm a statistical MT system uses the source sentence
and a prefix of it’s translation which is confirmed by the user to propose a suitable
continuation. And the text would be translated in an iterative process of interactions between
3Computer Assisted Translation

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 320

the user and the interactive MT system. The results showed that interactive MT can save asignificant amount of human effort. Also different translation models where tested in(Barrachina et al., 2009), and the results showed that the phrased based models have a better
performance than the other ones.
After TT2 some sporadic works were done on developing CAT systems (OrtizMartinez et al., 2009; Huang et al., 2012), but the more famous one was the Caitra system(Koehn 2009; Koehn and Haddow, 2009). It was a web-based system, which could beaccessed online by the users. The search method used as the base method in this paper, which
will be discussed in section 4.1, is the method proposed and implemented in this system.
Recently two other CAT projects, CASMACAT4 and MATECAT5 were done, whichwere committed to develop an open source workbench targeted both at researchers toinvestigate novel and enhanced types of assistance and at professional translators for actualuse (Federico et al., 2014; Alabau et al., 2014). One of the goals of these projects was to gain
insight into the cognitive processes involved in human translation, by using eye tracking and
key logging. The cognitive analysis will determine what types of assistance are offered to thetranslator, what information should be displayed on the screen, and what information should
be hidden as it would be distracting. Also they try to add the capability of the automatic
adaptation to the translated content to the CAT tools, which is not related to the subject of thispaper. Also the interactive-predictive CAT scenario when the user pronounce the correcttranslation instead of typing it has been studied in (Vakil and Khadivi, 2012b).
The other work done to improve the searching methods for the interactive prediction is(Vakil and Khadivi, 2012a), which tries to consider both the edit distance and the translation
score of each path in the search graph instead of just using the edit distance. This approach,which is called the weighted edit distance, is discussed more in section 4.2.
Also in (Ortiz-Martínez, 2011) and (González-Rubio et al., 2013) a search approach is
proposed, which is somewhat alike the weighted edit distance approach. But they introduce a
unified statistical framework, which integrates all of the statistical models used in thetranslation machine with the edit distance error, which is called error correction model here.
Also the proposed approach, models the edit distance as a Bernoulli process where eachcharacter of the candidate string has a probability of being erroneous.
The jump operation that we used here is somehow like the shift operation used inTranslation Error Rate (Snover et al., 2006), which is an evaluation metrics for machinetranslation and measures the number of edits required to change a system output into one of
the references.

3. Interactive-Predictive Machine Translation
In the interactive predictive machine translation paradigm, we want to produce a suffix whichis the best according to the user’s prefix and the source sentence. So in the statistical approach, we should maximize the probability of the suffix given the source sentence and theconfirmed prefix, as follows:tˆs  arg max Pr(ts | s, t p )ts

Where t s is the suffix, t p is the user’s confirmed prefix, and s is the source sentence.
We can write the above equation as follows:
45

http://www.casmacat.eu/index.php
http://www.matecat.com/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 321

tˆs  arg max Pr(t p , ts | s)  arg max Pr(t p , ts ).Pr( s | t p , ts )tstsNoting that t p ts  t , this equation would be very similar to the following, which is thestatistical machine translation’s equation.

tˆ  arg max Pr(t ).Pr( s | t )t

The main difference between these two is that in the interactive MT the search process
is restricted to those target sentences that contains t p as prefix. This implies that we can use
the same MT models if the search procedures are adequately modified (Och et al., 2003).
One of the most common ways, that we used here, for searching the best suffix in theinteractive MT is to use the search graph that is obtained once during the translation process,and complete each user’s confirmed prefix by searching the best path which has a prefix
which matches to the user’s prefix, and offering the rest of it to the user. The most important
advantage of using the pre-built search graph in each interaction is that it leads to an efficient
response time for each interaction. However, there’s not always a path in the search graphwhose prefix matches exactly with the user’s prefix, so some mechanisms should be chosen
the search for a path which has the most similarity with the user’s prefix. A common approachis to use the edit distance measure which will be discussed in the next section.

4. Searching Algorithms for the Interactive Prediction
In this section, we first introduce the searching strategy based on the edit distance, which ispresented in (Koehn, 2009), and is used as our base search algorithm. Then we try to improve
this algorithm in two ways; first by adding the edit distance error as a weighted model to thetranslation scores computed for each path by the decoder, and using this score for selecting
the best path, as in (Vakil and Khadivi 2012). And in the second method we add a jump operation to the computation of the edit distance, to consider the reorderings which are not in the
search graph but might match with the user’s prefix.
4.1.

Searching Based on Edit DistanceAccording to (Koehn, 2009), for giving a completed translation offer to the user, we shouldfind a hypothesis (or node) in the graph whose generated partial translation has the minimum
edit distance with the confirmed prefix; This approach is called edit distance-based search.
The edit distance between two strings is defined as the minimum number of edits needed totransform one string into the other, while the allowable edit operations are insertion, deletion,or substitution (Levenshtein, 1966). Although these operations could be done at character orword level, the word level distance makes more sense for our purpose, as performing one ormore character level operations on a word might change the whole meaning of that word, andthe number of operations done is not matter.
In this method the prediction is the optimal completion path that matches the user’s
prefix with minimal edit distance, and if there were more than one such paths, the one withhighest translation score will be chosen. This approach is based on the assumption that a hypothesis with minimum edit distance with the confirmed prefix has the continuation which ismore likely to be consistent with the remained part of the user’s desired translation.
This search method has shortcomings, as the search graph may not contain all of thereordering possibilities of words due to the pruning that is applied during the generation of thesearch graph. Since the edit distance only includes the insertion, deletion and substitution op-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 322

erations, it could not be able to consider the difference in reorderings between the user prefix
and any of the translation hypotheses in the search graph. Thus, it’s only able to find thosehypotheses which have a similar reordering of words to the user’s prefix. In order to solve thisproblem we discuss two approaches in the next sections.
4.2.

Searching Based on Weighted Edit DistanceThis approach which is presented in (Vakil and Khadivi, 2012), is based on the fact that thepaths in the search graph that have higher translation scores according to the translation andlanguage models, are more probable to be better translations in terms of both semantics andsyntax. However, these better translation hypotheses might not sometimes have lowest edit
distance with the user’s prefix due to the different reordering of words they have. Thus, using
the edit distance alone for selecting the best path, regardless of its translation score, would
select another path in the graph that might be an unfavourable translation.
To clarify this, consider the example shown in Table 1. We have the user’s desiredtranslation and two translation hypotheses samples, extracted from the search graph with theirtranslation scores, for a specific sentence. Assume that the prefix confirmed up to now by theuser is «Newton is one of the greatest scientists w», and the system should now offer a continuation for it.
Table 1. Example 1; the user's desired translation and system’s translation hypotheses
User’s desired translation :Translation hypothesis 1 :Translation hypothesis 2 :Newton is one of the greatest scientists who
discovered gravity
one of the greatest scientists is Newton who
discovered gravity
Newton gravity one of the greatest discovered
which is the greatest

Score: -5.25
Score:-10.23

For offering the rest of the translation by the method based on the edit distance, thetranslation hypothesis whose prefix has less edit distance with the confirmed prefix, should bechosen.
The least edit distance for each of the hypotheses is shown in Table 2. The first hypothesis needs 2 insert operations and 2 delete operations, and the second hypothesis needs
just 2 substitutions to match the user’s prefix. So the second hypothesis which is not a goodtranslation at all would be chosen, and the completion offered to the user would be «hich is
the greatest», which is obviously not desired. Instead, if we consider the translation scoresalong with the edit distances the first hypothesis could be chosen that has a much better offer
for the continuation.
Table 2. The least edit distance between each hypothesis and user's prefix in example 1
1
2

Newton is one of the greatest scientists is Newton w|ho discovered gravity
Newton gravityis one of the greatest discoveredscientists w|hich is the greatest# of edits
42

As the previous example shows, using only the edit distance to find the best path maylead to a hypothesis which has low quality according to the translation and language models.
To overcome this problem, in this approach the total score for each path that a prefix of itmatches with the user’s prefix, would be a weighted summation of the edit distance with theProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 323

translation scores obtained by the statistical models, and in the search process we look for ahypothesis with best total score.
4.3.

Edit Distance with Jump Operation

The weighted edit distance approach somehow overcomes the problem of lack of some reorderings in the search graph indirectly in some cases. In order to targeting this problem andresolving it directly, in this paper we propose another method that gives more freedom to theedit distance based searching by adding another operation, called jump operation.
To better explain the aim of adding jump operation and the effect that it could have on
the system’s offers, consider the example shown in Table 3. In this example, alike the previous one we have the user’s desired translation and two system’s translation hypotheses, except that here both hypotheses are good enough translations.
Table 3. Example 2; the user's desired translation and system’s translation hypotheses
desired translation :Translation hypothesis 1:
Translation hypothesis 2:

Newton who discovered gravity is one of thegreatest scientists
one of the greatest scientists is Newton who
discovered gravity
Newton is one of the greatest scientists who
discovered gravity

Score:-5.25
Score:-5.62

Assume that the user’s confirmed prefix is «Newton who discovered », and the systemshould now give an offer. For this purpose the minimum edit distance between the confirmed
prefix and any prefix of each of the translation hypotheses should be computed, which isshown in Table 4.
Table 4. The least edit distance between each hypothesis and user's prefix in example 2
1 oneNewton ofwho thediscovered |greatest scientists is Newton who discovered gravity

edits
3

2 Newton iswho onediscovered |of the greatest scientists who discovered gravity

2

According to Table 4, if we use only the edit distance, the second hypothesis will bechosen and the system’s offer would be “of the greatest scientists who discovered gravity”.
Also if we use the weighted edit distance method, the first hypothesis might be selected due toits better translation score, and the system might offer “greatest scientists is Newton who discovered gravity”. But obviously none of these offers are desirable by the user.
This problem caused by the fact that we consider the alignment between the user’s prefix and the prefix of the hypothesis is monotone and with no reordering, while due to theweakness of translation and reordering models and also the pruning done during the generation of the search graph, many of the reordering possibilities might not be produced or might
get low scores and be pruned. So in some cases, the part that the user translates at the beginning is at the end or at the middle of the system’s translations. In these cases, if we could shift
the parts of the hypothesis that match with the user’s prefix to the beginning of the sentence
and arranged the remained parts in a good order (as more than one segments might be remained, that needs concatenating to make a single segment), we could propose a better offer.
For instance, in the hypothesis 2 of the above example, if we could shift “who discovered

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 324

gravity“ after the word “Newton” at the beginning of it, we could get the user’s desired translation.
For this purpose, we define jump operation in the edit distance. With this operation we
can jump from some parts of each translation hypothesis and after matching the user’s prefix
completely, we collect the parts that we jumped over with the last remained part and arrange
them in a good order to generate the system’s offer.
As using the jump operation without any limitations enlarges the search space exponentially, we just allow one jump with variable length over multiple edges in each path in the
search graph. This is equivalent to allowing just one jump over some contiguous phrases ineach system’s translations. Also the jump operation could add an error to the edit distancevalue. Different amounts of errors considered for the jump operation, which will be discussed
in the experiments, but the best was adding one unit of error for each jump with any length.
In the above example we can match “Newton” in user’s prefix and hypothesis 2, thenby jumping from “is one of the greatest scientists”, we match the part “who discovered” withthe rest of the user’s prefix, and the part “gravity” will remain at the end of the hypothesis. So,with just one jump operation we could match the user’s prefix and the hypothesis.
The subject which is left is how to arrange the jumped over and remained parts to generate the final offer. As we could have at most one jumped over part and one ending part, weconsider the two permutations of attaching these two parts; i.e. first the jumped over partcomes and then the end part, or first the end part comes and then the jumped over part. Thenfor selecting between these two states we use language model, and the one whose language
model score according to the user’s prefix is more will be chosen.
For the above example, the two sentences “Newton who discovered is one of thegreatest scientists gravity” and “Newton who discovered gravity is one of the greatest scientists” will be scored with the language model, and as the second one is more well-formed thefinal offer to the user would be “gravity is one of the greatest scientists”.
The pseudocode of the search algorithm based on weighted edit distance with jump
operation is shown in Figure 1. The base of this implementation is like the edit distance approach presented in (Koehn, 2009). This algorithm associates backpointers with each hypothesis (or node) in the graph that point back to the lowest error path with which it can bereached. Each backpointer saves the number of edit distance errors, the translation score andthe position in the user’s prefix that the edit distance is computed up to there till the backpointer’s associated hypothesis. As each hypothesis may match the confirmed prefix at different positions, there are multiple backpointers for each hypothesis.
For adding the jump operation a variable named jumpState added to backpointers,
which shows the state of the jump operation till that backpointer. If no jump had been occurred until this point of the path, this variable is zero, if in the previous step of the backpointer’s path a jump happened, this variable should be 1 (means the jump could continue inthe next step or terminate here), and if a jump had been done and finished before this backpointer in its path, the jumpState value would be 2.
The algorithm starts from the first node (empty hypothesis) and iterates over the nodes
in the search graph in the topological order. When examining each node’s backpointers, allforward transitions (edges of the search graph) are examined using a string edit distance between the remaining prefix and the words in the transition target phrase (line 10). This may
consume the remaining prefix, and possibly lead to a new best path whose total score is better
than the best total score found so far (lines 11-18). Otherwise, new backpointers for the forward states are created (lines 19-21).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 325

Input: user prefix (u), search graph (g), weight of edit distance error (α)
Output: best completion offer
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.

bestTotalScore   , bestPath  {}Add backpointer (Score=0.0, Error=0, prefixMatched=0, jumpState=0) to start state
for all state s  G in topologically increasing order dofor all backpointer b of state s do
if bestTotalScore  b. Score  s. forwardScore   * b. Error thenfor all transition t from state s do
if b. jumpState  0 || b. jumpState 1 thenProcessJump(S, b, t)
end ifCompute edit distance matrix for end part of u after b.prefixMatched and t.phrase
for all matches m in matrix that consumed all of u do
new score c = b.Score + t.Score + t.toState.forwardScore
new error e = b.Error + m.Error
if c   * e  bestTotalScore then
set this as bestPath
bestTotalScore  c   * e

end ifend forfor all matches m in matrix that consumed all of t.phrase do
ProcessMatch(S, b, t, m)
end forend forend ifend forend forreturn the best ordering for jumped part and the end part in bestPath using LM

Figure 1. The Pseudocode for searching based on weighted edit distance with jump operation
Also, for each forward transition of a specific backpointer, the jumpState of that backpointer will be checked and if it is 0 or 1 it means that a jump could be done over that forward
transition. Therefore, this operation performs and the new backpointer for this case is also
created and added to the forward state’s backpointers (lines 7-9).
The ProcessJump function, shown in Figure 2, will make the new backpointer for performing the jump operation at each state. The error which is considered for this case is one if
it is the beginning of a jump, and zero otherwise; i.e. for each jump with any length, 1 unit oferror is considered (line 5).
Also, the ProcessMatch function will make the new backpointers for each case ofmatching a part of user’s prefix with the translation phrase produced at the processing transition, at each stage. The jumpState for the new backpointer will remain zero if the previous
backpointer’s jumpState was zero (means that the jump operation could be performed later inthe path), and it would be 2 otherwise (which means the jump operation was performed before, its finished and could no longer be used in the rest of this path) (line 6).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 326

ProcessJump(state S, backpointer b, transition t)

1.
2.
3.
4.
5.
6.

reached state s  t . toState
create new backpointer b b. BackState  S
b. Score  b. Score  t. Score (the translation score till here)b. Error  b. Error   b. jumpState  0 ?1 : 0  (number of errors till state s’)b. prefixMatched  b. prefixMatched(the index in the user’s prefix that the edit distance error is computed till there)

8.

b. jumpState  1b  backpointer of state swith prefixMatched equals b. prefixMatched and jumpState equal to 1

9.

if b not defined or b. Score   * b. Error  b. Score   * b. Error )

7.

10.
11.

set b newbackpointer for state s  instead of b
end ifFigure 2. The Pseudocode for adding a jump operation
ProcessMatch(state S, backpointer b, transition t, match m)

2.
3.
4.

reached state s  t .toState
create new backpointer b b. BackState  S
b. Score  b. Score  t. Score , b. Error  b. Error  m. Error

5.

b. prefixMatched  m. prefixMatched1.

6.

b. jumpState   b. jumpState  0 ? 0 : 2 b  backpointer for state s with prefixMatched equals b. prefixMatched

7.
8.
9.
10.

and jumpState equal to b. jumpStateif b not defined b. Score   * b. Error  b. Score   * b. Error

set b newbackpointer for state s  instead of b

end ifFigure 3. The Pseudocode for adding a match case

5. Experiments
In this section, we discuss the experiments done to evaluate the searching algorithms presented in the previous section for interactive prediction.
5.1.

Evaluation Metrics

The common way for evaluating the performance of the interactive predictive machine translation systems automatically is to estimate the effort of a human translator to produce correcttranslations using that system. For this purpose, the translating process of the user is simulatedusing a set of reference translations as the desired translations in the user’s mind.
In this simulation, for each given source sentence in the test set, the translation is produced by the interactive MT system at the first step. Then it’s compared with the reference

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 327

given for that source sentence, and the character-level longest common prefix of them will becomputed. Afterwards, the first non-matching character is replaced by the corresponding reference character and then the system offers a new completion for the given prefix. This process is iterated until a full match with the reference is obtained. Each time a part of the new
system’s offer is accepted by the user and it expands the prefix, a mouse action is done, andeach time the new character is added to the prefix from the reference a keystroke is considered to be done by the user.
The measure used here for evaluation is KSMR 6, which is the overall number of interactions, i.e. keystrokes and mouse-actions, done to translate the whole test set divided by the
number of running characters in the reference translations.
5.2.

Translation System Description

In this work, we use a Farsi to English phrase-based translation system to evaluate our proposed method. We use Giza++ (Och and Ney, 2003), to train the translation models and Moses Toolkit7 (Koehn et al., 2007) as the MT decoder, as well as the SRILM 8 (Stolcke, 2002)
tools to build the language model. The interactive system is implemented as a server using
Moses decoder, which gets a new source sentence or the user’s confirmed prefix each timeand gives the continuation offer to the client, while the client simulates the user’s interactions.
The statistics of the corpora used for training the translation system (Jabbari et al.,2012) and the test set is shown in Table 5.
Table 5. The statistics of the training and testing corpora
Corpora# of Sentences

Farsi
Train

# of Lexicons

37356049

39469736770830

3143722164408
English
Farsi

Test

4524
200

English5.3.

# of Running Words

4431

Results

In our experiments, at first we evaluate each of the search methods discussed in section 4, tocompare their performance, and to study the effect of each improvements done to the baseline. For the weighted edit distance method, the weight of the edit distance error is tuned on adevelopment set and is finally set to -0.5. Also for the jump operation error, three different
cases are considered; number of words we jumped over, number of phrases we jumped overand 1. The results of the KSMR and the average response time for each interaction are given
in Table 6 for each of the methods.

6

Keystroke and mouse-action ratio
https://github.com/moses-smt/mosesdecoder8
http://www.speech.sri.com/projects/srilm/
7

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 328

Table 6. Evaluation results of different search methods
Searching Method

KSMR (%)

Average Response Time (s)Edit Distance Method

31.78

0.21

Jump Error = # of
Words

31.62

0.93

Jump Error = # ofPhrases

31.23

1.09

Jump Error = 1

30.43

1.67

Weighted Edit Distance Method

31.36

0.27

Jump Error = # of
Words

30.31

1.1

Jump Error = # ofPhrases

30.2

1.42

Jump Error = 1

30.15

1.65

Adding Jump to
Edit Distance
MethodAdding Jump to
Weighted Edit
Distance Method

As the results shows, using the weighted sum of edit distance error and translation
scores makes 0.42% improvements in KSMR and no significant growth in response time,compared to the base edit distance method.
Also, adding the jump operation to the edit distance method shows that it could improves its KSMR up to 1.35%, when the jump error is set to one regardless of the jumplength. The results of the other settings for jump error are not as good as considering it to beone, which shows assuming it to be the number of words or phrases that we jumped over is ahigh error, which prevents some long jumps that might be useful.
Furthermore, when we combine weighted edit distance method with jump operation,
we get 1.21% improvements in KSMR, in comparison to the weighted method itself, and1.63% improvements compared to the base edit distance method. This shows that these two
improving methods do not have much overlap in the cases that they improve the baseline system.
Despite the improvements in KSMR, the response time while we add the jump operation increases significantly. This increase can be expected as the number of backpointers created in the search process grows with adding jump operation, and the search space becomesmuch greater. For solving this issue we limit the jump operation more, by putting a thresholdfor the maximum amount of phrases (edges in the graph) that we could jump over. In the next
experiment, the effect of this threshold is studied, and the results are shown in Table 7.
As the results in Table 7 shows, if we assume that the response time below 1 second isacceptable, then by selecting the threshold equal to 5, we can gain about 0.82% improvementin KSMR from the weighted edit distance method without jump and 1.24% improvementcompared to the base edit distance method.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 329

Table 7. Results of the effect of limiting the jump length
Searching Method

Adding Jump to
Weighted Edit
Distance Method

Maximum
Jump Length

KSMR (%)

Average Response Time (s)

1

31.16

0.79

2

30.92

0.8

3

30.71

0.9

4

30.64

0.93

5

30.54

0.96

6

30.41

1.04

8

30.27

1.13

10

30.23

1.23

∞

30.15

1.65

6. Conclusions and Future WorkIn this paper we aimed to improve a common search method used for the interactive prediction in the computer assisted translation systems, which searches for a path in the searchgraph, whose prefix has least edit distance with the user’s confirmed prefix.
As the reordering of words which is desired by the user might not be in the searchgraph, the simple edit distance method has the problem of not considering the differences inreorderings between the system’s translation and the user’s prefix. In order to solve this problem, we propose adding a jump operation to the edit distance, which allows jumping over acontiguous sequence of phrases in a system’s translation while matching the user’s prefix withthat translation, and then add the part that it jumped over to the continuation offer of the system.
The results show that adding jump to the edit distance improved the KSMR measureby 1.35%. Also, we show that this method does not have much overlap with another method
which has been proposed for this purpose and used the weighted sum of edit distance andtranslation scores to select the best offer. Combining these two methods improved the baseedit distance method by 1.63% in KSMR.
However, adding jump operation increases the average response time of the system ineach interaction. This is because of the significant increase in the number of backpointersgenerated in the search process. We limit the length of the jumps to reduce the response time.
Instead, the mechanisms could be presented to prune the inappropriate backpointers, and prevent them from further processing, so that we could increase the response speed with less reduction in KSMR

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 330

References
Alabau, V., Buck, C., Carl, M., Casacuberta, F., Garcıa-Martınez, M., Germann, U., et. al. (2014).
Casmacat: A computer-assisted translation workbench. In Proceedings of the 14th Conference of
the European Chapter of the Association for Computational Linguistics. 25-28.
Barrachina, S., Bender, O., Casacuberta, F., Civera, J., Cubel, E., Khadivi, S., Lagarda, A., Ney, H.,
Tomás, J., Vidal, E. and Vilar, J. M. (2009). Statistical approaches to computer-assisted translation. Computational Linguistics 35, no. 1: 3-28.
Bender, O., Hasan, S., Vilar, D., Zens, R., and Ney, H. (2005). Comparison of generation strategies forinteractive machine translation. Proceedings of the 10th Annual Conference of the European Association for Machine Translation (EAMT 05). 33-40.
Federico, M., Bertoldi, N., Cettolo, M., Negri, M., Turchi, M., Trombetti, M., et. al. (2014). The MateCat tool. In Proceedings of COLING. 129-132.
Foster, G., Isabelle, P., and Plamondon, P. (1997). Target-Text Mediated Interactive MachineTranslation. Machine Translation 12, no. 1-2: 175-194.
González-Rubio, J., Ortiz-Martínez, D., Benedí, J., and Casacuberta. F. (2013) Interactive MachineTranslation using Hierarchical Translation Models. EMNLP. 244-254.
Huang, C., Yang, P., Chen, K., and Chang, J.S. (2012). TransAhead: a computer-assisted translation andwriting tool. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 352-356.
Jabbari, F., Bakhshaei, S., Mohammadzadeh Ziabary, S. M., and Khadivi, S. (2012). Developing an
Opendomain English-Farsi Translation System Using AFEC: Amirkabir Bilingual Farsi-English
Corpus. Association for MachineTranslation in the Americas (AMTA).
Koehn, P. (2009). A process study of computer-aided translation. Machine Translation 23, no. 4: 241263.
Koehn, P., and Haddow, B. (2009). Interactive assistance to human translators using statistical machinetranslation methods. MT Summit XII.
Koehn, P., et al. (2007) Moses: Open source toolkit for statistical machine translation. Proceedings ofthe 45th annual meeting of the ACL on interactive poster and demonstration sessions. Association
for Computational Linguistics. 177-180.
Langlais, P., Foster, G., and Lapalme, G. (2000). TransType: a computer-aided translation typing system. Proceedings of the 2000 NAACL-ANLP Workshop on Embedded machine translation systems-Volume 5. Association for Computational Linguistic. 46-51.
Levenshtein, V. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet
physics doklady 10, no. 8: 707-710.
Och, F.J., and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational linguistics 29, no. 1: 19-51.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 331

Och, F.J., Zens, R., and Ney, H. (2003). Efficient search for interactive statistical machine translation.
Proceedings of the tenth conference on European chapter of the Association for Computational
Linguistics-Volume 1. Association for Computational Linguistics. 387-393.
Ortiz-Martínez, D. (2011). Advances in Fully-Automatic and Interactive Phrase-Based Statistical Machine Translation. PhD thesis, Universitat Politècnica de València.
Ortiz-Martınez, D., Garcıa-Varea, I., and Casacuberta, F. (2009). Interactive machine translation basedon partial statistical phrase-based alignments. RECENT ADVANCES IN NATURAL LANGUAGE
PROCESSING. 330-336.
Slocum, J. (1985). A survey of machine translation: its history, current status, and future prospects.
Computational linguistics 11, no. 1 (1985): 1-17.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A Study of Translation Edit
Rate with Targeted Human Annotation. Proceedings of Association for Machine Translation in theAmericas.
Stolcke, A. (2002). SRILM-an extensible language modeling toolkit. Proc. Intl. Conf. on Spoken
Language Processing, vol. 2, pp. 901-904, Denver.
Tomás, J., and Casacuberta, F. (2006). Statistical phrase-based models for interactive computer-assistedtranslation. Proceedings of the COLING/ACL on Main conference poster sessions. Association for
Computational Linguistics. 835-841.
Vakil, Z., and Khadivi, S. (2012a). A New Search Approach for Interactive-Predictive ComputerAssisted Translation. COLING (Posters). 1261-1270.
Vakil, Z., and Khadivi, S. (2012b). Interactive-predictive speech-enabled computer-assisted translation.
International Workshop on Spoken Language Translation (IWSLT). 237-243.
Whitelock, P. J., Wood M. M., Chandler, B. J., Holden, N., and Horsfall, H. J. (1986) Strategies forinteractive machine translation: the experience and implications of the UMIST Japanese project.
Proceedings of the 11th coference on Computational linguistics. Association for Computational
Linguistics. 329-334.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 332

Improving Semantic SMT via Soft Semantic Role
Label Constraints on ITG Alignments
Meriem BELOUCIF
mbeloucif@cs.ust.hk
Markus SAERS
masaers@cs.ust.hk
Dekai WU
dekai@cs.ust.hk
Human Language Technology Center
Hong Kong University of Science and Technology, Hong Kong

Abstract
We show that applying semantic role label constraints to bracketing ITG alignment to train
MT systems improves the quality of MT output in comparison to the conventional BITG and
GIZA alignments. Moreover, we show that applying soft constraints to SRL-constrained BITG
alignment leads to a better translation system compared to using hard constraints which appear too harsh to produce meaningful biparses. We leverage previous work demonstrating that
BITG alignments are able to fully cover cross-lingual semantic frame alternations, by using
semantic role labeling to further narrow BITG constraints, in a soft fashion that avoids losing
relevant portions of the search space. SRL-based evaluation metrics like MEANT have shownthat tuning towards preserving the shallow semantic structure across translations, robustly improves translation performance. Our approach brings the same intuition into the training phase.
We show that our new alignment outperforms both conventional Moses and BITG alignment
baselines in terms of the adequacy-oriented MEANT scores, while still producing comparable
results in terms of edit distance metrics.

1

Introduction

The quality of machine translation output relies heavily on word alignment. However,
the most widespread approach to word alignment is the ad hoc method of training IBM
models (Brown et al., 1990) in both directions and combining their results using various heuristics. Word alignments based on inversion transduction grammars or ITGs
(Wu, 1997), on the other hand, provide a more structured model leading to efficient andoptimal bidirectional alignments.
In this paper we introduce an improved word aligner based on applying soft semantic role label constraints to ITG alignment. We show that both translation adequacy andfluency can be improved by replacing the conventional GIZA++ based alignment (Ochand Ney, 2000) with more semantically motivated alignments obtained through training

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 333

ITGs (Saers and Wu, 2009) under soft SRL constraints. The new approach is motivatedby Addanki et al. (2012) who demonstrated empirically that the semantic role reorderings found in cross-lingual SRL frames is essentially 100% covered by ITG constraints,
which suggests that it should be possible to use ITG constraints as a starting point underwhich to align semantic frames as we do in this paper.
Our approach is further motivated by the fact that including semantic role labelingin the SMT pipeline in a different way has already been shown to increase translation
quality. The semantic frame based evaluation metric MEANT, which was shown tocorrelate better with human adequacy judgment than conventional surface based evaluation metrics (Lo et al., 2012), can be used as an objective function for tuning SMT.
Tuning to MEANT, which attempts to optimize the degree to which a sentence’s semantic frames can be preserved across translation, was shown to improve translation
quality across many metrics (Lo et al., 2013b; Beloucif et al., 2014). We show in thispaper that including soft constraints based on semantic role labeling into the alignment
training step yields both higher adequacy-oriented MEANT and, while still producing
comparable scores on surface based and edit distance metrics.
2

Related work

2.1

Alignment

For most recent automatic machine translation systems, learning a good word alignment is paramount for producing meaningful translation. Unfortunately, conventionalalignment algorithms such as IBM models (Brown et al., 1990) and the HMM-alignment
model (Vogel et al., 1996) are flat and directed, meaning that (a) they allow unstructured
movement of words leading to weak word alignment, (b) translations in one direction
are considered in isolation, and (c) two separate alignments are needed to form a single
bidirectional alignment. The harmonization of two directed alignments is typically done
heuristically, which means that there is no model that considers the final bidirectionalalignment that the translation system is trained on to be optimal. Transduction grammars, on the other hand, do provide a model that (a) is inherently structurally compositional, and (b) can provide optimal bidirectional alignments. Although this structured
optimality comes at a higher cost in terms of time complexity, it allows for preexisting
structured information to be incorporated into the model, and for models to be comparedin a meaningful way.
There are different classes of transduction grammars, ranging from finite-statetransduction grammar, via linear transduction grammar (Saers et al., 2010) and inversion transduction grammar (Wu, 1997; Saers and Wu, 2009; Saers et al., 2009),
to syntax-directed transduction grammar (Lewis and Stearns, 1968; Aho and Ullman,
1972) and many ways to formulate the model over them: Wu (1995); Zhang and Gildea(2005); Chiang (2007); Cherry and Lin (2007); Blunsom et al. (2009); Haghighi et al.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 334

(2009); Saers et al. (2010); Neubig et al. (2011). In this paper, we introduce a semantically biased version of inversion transduction grammars (Wu, 1997) that is biased
towards constituents that conform to monolingual semantic parses on the input and/or
output languages, and compare their performance against (a) ITGs without such a bias,and (b) the conventional heuristics.
2.2

Semantic role labeling in MT

Our alignment method is fully compatible with the principle that a good translation is
one where a human can successfully understand the main meaning of the output sentence as captured by the basic event structure: “who did what to whom, when, where andwhy” (Pradhan et al., 2004; Lo and Wu, 2011, 2012; Lo et al., 2012). The MEANT family of metrics are semantic evaluation MT evaluation metrics that correlate with humanadequacy judgements more closely than most commonly used surface based metrics (Loand Wu, 2011, 2012; Lo et al., 2012; Lo and Wu, 2013b; Macháček and Bojar, 2013).
MEANT compares the MT output sentence against provided reference translations, andproduce a score measuring the degree of similarity between their semantic frame structures. Our new approach is encouraged by the fact that many previous studies haveempirically shown that integrating semantic role labeling into the training pipeline bytuning against MEANT improves the translation adequacy (Lo et al., 2013a; Lo and Wu,
2013a; Lo et al., 2013b; Beloucif et al., 2014). We show here, that soft incorporationof SRL constraints much earlier in the pipeline, at the word alignment stage of SMT
training, can further improve translation adequacy.
2.3

Inversion transduction grammars

A transduction represents a set of bi-sentences that define the relation between an input
language L0 and an output language L1 . Accordingly, a transduction grammar generates
a transduction or a set of bi-sentences, translates between sentences in L0 and sentencesin L1 , and accepts the sentence pairs in the transduction. Inversion transductions area subset of syntax-directed transductions which are generated and parsed by inversion
transduction grammars or ITGs (Wu, 1997). An ITG can always be written in 2-normal
form and is represented by a tuple ⟨N, V0 , V1 , R, S⟩ where N is a set of non-terminals,
V0 and V1 are the vocabularies of L0 and L1 respectively, R is a set of transduction rules
and S ∈ N is the start symbol. In the 2-normal form, each inversion transduction must
be on one of the following forms:
S→A
A → [BC]A → ⟨BC⟩A → e/ϵ
A → ϵ/f

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 335

Input%    
Gloss% this%one%in%Japan%yet%haven’t%sell%.%
Reference% They%do%not%sell%this%in%Japan%,%yet%.%GIZA++% this%in%Japan%yet%on%sale%.%
ITG% this%in%Japan%has%not%on%sale%.%
ITG%with%hard%SRL%constraints% %ITG%with%soA%SRL%constraints% this%is%not%sold%in%Japan%yet%.%Figure 1: MT output of the three different systems for a given Chinese sentenceplease%
tell%
me%
the%
fastest%
way%
to%
go%
to%
San%
Francisco%.%
%   % %
% % 
%

%

%
%

Figure 2: An alignment of a bisentence produced by GIZA++ alignment, with growdiag-final-and as a heuristic

A → e/f

ITGs allow straight and inverted rules such that straight transduction rules use
square brackets and take the form A → [BC] and inverted rules use inverted brackets and take the form A → ⟨BC⟩. Straight transduction rules generate transductions
with the same order in L0 and L1 which means that, in the parse tree, the children instantiated by straight rules are read in the same order. Inverted transduction rules on
the other hand, generate transductions with inverted order in L0 and L1 , so the children
instantiated by inverted rules are read left-to-right in L0 and right-to-left in L1 . In thispaper we show that an ITG-based SMT system is able to perform better on semantic
metrics when biased towards respecting semantic role labeling.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 336

please%
tell%
me%
the%
fastest%
way%
to%
go%
to%
San%
Francisco%.%
%   % %
% % 
%

%

%
%

Figure 3: An alignment of a bisentence produced by ITG alignment
please%
tell%
me%
the%
fastest%
way%
to%
go%
to%
San%
Francisco%.%
%   % %
% % 
%

%

%
%

Figure 4: An alignment of a bisentence produced by ITG alignment using automaticshallow semantic parsing constraints. The input is parsed by a Chinese automatic shallow semantic parser.
3

SRL-constrained ITGs

The model we propose in this paper introduces soft constraints to bias a bracketinginversion transduction grammar (BITG) towards preferring bilingual constituents that
conform to automatically generated monolingual semantic role labels on both sides.
Because of the structural differences between monolingual SRLs and the bilingual BITG
parses, we implement this as a penalty for BITG constituents violating the semantic role

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 337

labels in either language.
The semantic roles and their fillers in a sentence sometimes span multiple syntactic
units, or in technical terms: the semantic trees are (a) not necessarily consistent with thesyntactic trees, and (b) not necessarily projective. Since BITG trees are defined to be
projective, applying even a single monolingual SRL parse as a hard constraint would
rule out all possible BITG trees, and all possible alignments for that sentence pair, sinceno BITG parse can conform to a non projective constraint. Even when the monolingual
SRL trees are projective in both languages, there is a risk of overly constraining thesearch, as the only way for the BITG parser to satisfy incompatible SRL constraints
is to sacrifice the lexical correspondences; the only way to conform to (a) the input
language SRL, (b) the output language SRL, and (c) the ITG constraint may be to delete
a constituent in the input language and insert it in the output language, even when there
is a good translation between them, because translating them would violate at least oneof the two constraints. The ITG constraint is what allows us to do this processing in
polynomial time, so it is non-negotiable. As the lexical relation is what defines the wordalignment, which is what we are interested in, we opt to soften the SRL constraints. In
practice, the automatic SRL parses are fairly noisy, an engineering reason to soften them,but even with perfect SRL parsers, soft constraints are theoretically necessary.
The soft constraints takes the form of a fixed penalty that is paid whenever the
BITG parser wants to introduce a bi-constituent that crosses a semantic constituent (thestring a predicate or one of its role fillers span). No penalty is paid for bi-constituentscompletely covering a semantic constituent or by bi-constituents that are completely
covered by semantic constituents. To allow for some degrees of freedom, we allows fortwo separate penalties, one for crossing an input language semantic constituent, λ1 , andone for crossing an output language semantic constituent, λ0 . These hyper parametersneed to be set manually.

4

4.1

Experiment Setup

Data

For this paper, we tested our systems on Chinese to English translations. We usedIWSLT 2007 data set for this experiment. The training set contains 39,953 sentences.
The dev and test set contain 1512 sentences and test 489 sentences, respectively. Boththe English and Chinese corpora were normalized for puntuation, tokenized and truecased. We also used our own Chinese named entity recognition, and dedicated proper
name translator.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 338

4.2

Word alignment

We compare the performance of our SRL soft-constrained model to the SRL hardconstrained system and to the conventional unconstrained ITG baseline. We perform agrid search over soft-constraints hyper parameters to find the optimal settings. We thencompared the performance of our proposed alignments to the conventional GIZA++
baseline with grow-diag-final-and to harmonize the two alignment directions.
Our ITG baseline is a token-based BITG system. We initialize it with uniform structural probabilities, setting aside half of the probability mass for lexical rules. This probability mass is distributed among the lexical rules according to co-occurrence counts
from the training data, assuming each sentence to contain one empty token to accountfor singletons. These initial probabilities are refined with 10 iterations of expectation
maximization where the expectation step is calculated using beam pruned parsing (Saers
et al., 2009) with a beam width of 100. On the last iteration, we extract the alignments
imposed by the Viterbi parses as the word alignments outputted by the system.
The new SRL-constrained ITG approach adds the crossing penalty based on automatic SRL parses discussed in Section 3 to the ITG baseline discussed above. The
shallow semantic parses of the training data were produced using ASSERT (Pradhan etal., 2004) and C-ASSERT (Wu et al., 2006) for English and Chinese respectively.
We show how applying soft SRL constrained ITG alignment outperforms alignment both (a) without constraints and (b) with hard SRL constraints. The hyper parameterλi = 0 represents the hard constraints, λi = 1 represents the case with no constraints
and λi between 0 and 1 are the soft constraints. We run some prior experiments andobserved that applying hard SRL constraints did not lead to any alignment at all: theconstraints were too harsh and did not permit any biparses. Soft SRL constrained ITGs,
on the other hand, outperformed the unbiased BITG model in term of both adequacyoriented MEANT scores. We noticed that λ0 = 1 and λ1 = 0.5 correspond to thebest combination. The SRL constraints were only used during training of the probabilities of the ITG, and not when extracting the Viterbi parses and the corresponding wordalignments.

4.3

SMT pipeline

To test the different alignments described in this paper, we used the standard Moses
toolkit (Koehn et al., 2007), with a 6-gram language model learned with the SRI language model toolkit (Stolcke, 2002), to train our baselines. We tested our approach
using Moses hierarchical models. For tuning, we used ZMERT (Zaidan, 2009), a standard implementation of minimum error rate training or MERT (Och, 2003), we run eachtuning task ten times for each system, then we decoded with both DEV and TEST set,
we then chose the results according to what performed the best on the know develop-

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 339

ment data. We also present the highest score achieved by each system among all runs
(an upper bound on the score that can be achieved). We compared an edit-distance based
metrics, BLEU (Papineni et al., 2002) and TER (Snover et al., 2006), and a semantic
evaluation metric MEANT (Lo et al., 2012) as the tuning objective.

5

Results

We compared the the performance of our soft SRL-constrained ITG alignment to (a) the
GIZA++ baseline and (b) the unbiased BITG, for both BLEU, TER and MEANT tuned
systems. We evaluated our MT output using the semantic metric MEANT (Lo et al.,2012). Tables 1, 2 and 3 show the improvment in terms of MEANT scores for the SRL
ITG aligned system in comparison to conventional ITG alignment and GIZA++ alignment for BLEU, TER and MEANT tuned systems respectively. The MEANT score for
ITG based systems is considerably higher than the MEANT score for GIZA++ aligned
model. We also observe that MEANT score for ITG with SRL constraints is better thanthe conventional ITG model. We believe that a better SRL-parser would yield a bettersystem still. Tables 4, 5 and 6 give an upper bound on the results for the ten runs, we
also see here that new semantically biased ITG model outperforms the baseline and ITG
based alignment.
In addition, we evaluated the performance of our model using surface-based metrics
such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), CDER
(Leusch et al., 2006), WER (Nießen et al., 2000), and TER (Snover et al., 2006), andobserved that both the unbiased ITG and semantically biased ITG based systems yieldcomparable results that are high in comparison to conventional GIZA++ alignment.
Figure 1 shows an interesting example extracted from the test data. The Chinese
input sentence has been pre-segmented into eight word-like units, and a word-for-word
gloss reads approximately ”this one in Japan yet haven’t sell .” The translator who produce the reference translation took some liberties and introduced the actor they, who no
doubt was present in the context, but is not needed when the sentence is considered inisolation. The conventional GIZA++ system fails completely, not only is the translation
bad English, but the meaning it conveys is the polar opposite of the original sentence
due to a dropped negation. Although these kind of errors are disastrous for the consumer of the translations, they are common with surface based systems, and not likely
to be addressed with surface based tuning objectives. The unbiased ITG system produces a sentence that is understandable but far from good, comparable to something asecond-language learner would write early on. The produced sentence does, however,convey the correct meaning, although some nuance is missing due to the dropped yet.
The ITG system with hard SRL constraints outputs nothing, because the constraints
prevented it from learning from most of the training examples. The ITG system with
soft SRL constraints produces a sentence that conveys the correct meaning in good English. Considered in isolation, the produced sentence is even better than the man-madeProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 340

Table 1: The optimal results given the known development set for the BLEU tuned
systems
cased
SystemMEANT BLEU METEOR TER WER PER
CDER
Giza alignment
49.42
25.07 0.451
59.95 60.96 54.91 59.32
ITG alignment
50.02
24.13 0.460
8.49
59.55 53.98 58.01
SRL ITG alignment 50.92
23.67 0.460
58.78 59.79 53.61 58.25
Table 2: The optimal results given the known development set for the TER tuned systems
cased
SystemMEANT BLEU METEOR TER WER PER
CDER
Giza alignment
48.70
23.02 0.407
59.93 60.83 55.76 59.55
ITG alignment
49.76
21.72 0.431
57.83 58.76 53.90 57.38
SRL ITG alignment 50.02
21.92 0.433
58.78 59.55 53.34 58.01
Table 3: The optimal results given the known development set for the MEANT tuned
systems
cased
SystemMEANT BLEU METEOR TER WER PER
CDER
Giza alignment
48.88
21.18 0.455
61.49 62.90 55.76 60.91
ITG alignment
49.12
23.47 0.465
60.30 61.20 55.36 59.53
SRL ITG alignment 50.66
22.83 0.446
59.48 60.32 54.30 58.84

reference translation, as it is closer to the original.
Figures 2-4 shows a sentence pair from the training data, and how the different
systems end up aligning the words. The pre-segmented Chinese words correspond toplease, tell, I go, San Francisco, most, fast, of, way, and full stop. The harmonized
GIZA++ alignments (Figure 2) are good, except that to go to San Francisco and thefastest are grouped into atomic units, which means that this is not an example of go,
San Francisco, or fastest being translated. The unconstrained ITG alignment (Figure3) makes two mistakes: it insists on aligning 我去’I go’ with me instead of go, and italigns the superlative marker 最 with Francisco. The SRL constraints (Figure 4) areable to fix the latter, but not the former.

6

Conclusion

In this paper we showed that incorporating SRL constraints into the training of bracketing ITGs for early stage word alignment in SMT training leads to improved semantic

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 341

Table 4: The upper bound score among the eleven runs for BLEU tuned systems foreach pipeline.
cased
SystemMEANT BLEU METEOR TER WER PER
CDER
Giza alignment
50.91
25.07 0.453
59.53 60.42 54.43 59.02
ITG alignment
50.75
24.41 0.469
58.47 59.55 53.90 57.80
SRL ITG alignment 51.28
24.67 0.556
58.78 59.69 53.61 58.25
Table 5: The upper bound score among the eleven runs for TER tuned systems for eachpipeline.
cased
SystemMEANT BLEU METEOR TER WER PER
CDER
Giza alignment
49.94
23.02 0.408
59.40 60.52 55.58 59.14
ITG alignment
50.75
22.11
0.434
57.16 58.57 53.55 57.30
SRL ITG alignment 50.92
24.70 0.441
57.93 58.86 53.34 57.43
Table 6: The upper bound score among the eleven runs for MEANT tuned systems foreach pipeline.
cased
SystemMEANT BLEU METEOR TER WER PER
CDER
Giza alignment
49.15
21.25 0.455
61.49 62.90 55.76 60.75
ITG alignment
49.94
23.94 0.466
60.30 61.20 55.36 59.53
SRL ITG alignment 50.66
22.83 0.457
59.48 60.32 54.30 58.81

translation adequacy. Moreover, we showed that applying soft SRL constraints in oneof the languages produces better performance on the semantically oriented MEANT
metric, in comparison to not applying any constraints. As automatically produced semantic parses for both languages are incompatible much of the time, we showed thatthe increased flexibility of soft constraints helps improve the word alignment quality.
Finally, we observed that applying SRL constraints to BITG alignment using soft constraints not only improves MEANT scores but also retains the performance using surface
oriented metrics like metrics like BLEU, METEOR, TER, WER, PER and CDER.
7

Acknowledgment

This material is based upon work supported in part by the Defense Advanced Research
Projects Agency (DARPA) under BOLT contract no. HR0011-12-C-0016, and GALE
contract nos. HR0011-06-C-0022 and HR0011-06-C-0023; by the European Union under the FP7 grant agreement no. 287658; and by the Hong Kong Research Grants Council (RGC) research grants GRF620811, GRF621008, and GRF612806. Any opinions,

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 342

findings and conclusions or recommendations expressed in this material are those of
the authors and do not necessarily reflect the views of DARPA, the EU, or RGC. We are
grateful to Pascale Fung, Yongsheng Yang and Zhaojun Wu for sharing the maximum
entropy Chinese segmenter and C-ASSERT, the Chinese semantic parser, with us.
References
Karteek Addanki, Chi-kiu Lo, Markus Saers, and Dekai Wu. LTG vs. ITG coverage of cross-lingual verb
frame alternations. In EAMT-2012, Trento, Italy, May 2012.
Alfred V. Aho and Jeffrey D. Ullman. The Theory of Parsing, Translation, and Compiling. Prentice-Halll,Englewood Cliffs, NJ, 1972.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved
correlation with human judgments. In Workshop on Intrinsic and Extrinsic Evaluation Measures for
Machine Translation and/or Summarization, Ann Arbor, MI, Jun 2005.
Meriem Beloucif, Chi kiu Lo, and Dekai Wu. Improving meant based semantically tuned smt. In IWSLT
2014, 2014.
Phil Blunsom, Trevor Cohn, Chris Dyer, and Miles Osborne. A Gibbs sampler for phrasal synchronous
grammar induction. In ACL-IJCNLP 2009, pages 782–790, Suntec, Singapore, Aug 2009.
Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Frederik Jelinek, John D.
Lafferty, Robert L. Mercer, and Paul S. Roossin. A statistical approach to machine translation. Computational Linguistics, 16(2):79–85, 1990.
Colin Cherry and Dekang Lin. Inversion transduction grammar for joint phrasal translation modeling. In
SSST, pages 17–24, Rochester, NY, Apr 2007.
David Chiang. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228, 2007.
Aria Haghighi, John Blitzer, John DeNero, and Dan Klein. Better word alignments with supervised ITG
models. In ACL-IJCNLP 2009, pages 923–931, Suntec, Singapore, Aug 2009.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In ACL 2007,
pages 177–180, Prague, Czech Republic, Jun 2007.
Gregor Leusch, Nicola Ueffing, and Hermann Ney. CDer: Efficient MT evaluation using block movements. In EACL-2006, 2006.
Philip M. Lewis and Richard E. Stearns. Syntax-directed transduction. Journal of the Association for
Computing Machinery, 15(3):465–488, 1968.
Chi-kiu Lo and Dekai Wu. MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluatingtranslation utility based on semantic roles. In ACL HLT 2011, 2011.
Chi-kiu Lo and Dekai Wu. Unsupervised vs. supervised weight estimation for semantic MT evaluation
metrics. In SSST-6, 2012.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 343

Chi-kiu Lo and Dekai Wu. Can informal genres be better translated by tuning on automatic semantic
metrics? In MT Summit XIV, 2013.
Chi-kiu Lo and Dekai Wu. MEANT at WMT 2013: A tunable, accurate yet inexpensive semantic frame
based mt evaluation metric. In WMT 2013, 2013.
Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. Fully automatic semantic MT evaluation. In WMT2012, 2012.
Chi-kiu Lo, Karteek Addanki, Markus Saers, and Dekai Wu. Improving machine translation by training
against an automatic semantic frame based evaluation metric. In ACL 2013, 2013.
Chi-kiu Lo, Meriem Beloucif, and Dekai Wu. Improving machine translation into Chinese by tuning
against Chinese MEANT. In IWSLT 2013, 2013.
Matouš Macháček and Ondřej Bojar. Results of the WMT13 metrics shared task. In WMT 2013, Sofia,
Bulgaria, Aug 2013.
Graham Neubig, Taro Watanabe, Eiichiro Sumita, Shinsuke Mori, and Tatsuya Kawahara. An unsupervised
model for joint phrase alignment and extraction. In ACL HLT 2011, pages 632–641, Portland, OR, Jun
2011.
Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. A evaluation tool for machine translation: Fast evaluation for MT research. In LREC 2000, 2000.
Franz Josef Och and Hermann Ney. Improved statistical alignment models. In ACL 2000, pages 440–447,
Hong Kong, Oct 2000.
Franz Josef Och. Minimum error rate training in statistical machine translation. In ACL-2003, pages
160–167, Sapporo, Japan, Jul 2003.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluationof machine translation. In ACL-02, pages 311–318, Philadelphia, PA, Jul 2002.
Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James H. Martin, and Dan Jurafsky. Shallow semantic
parsing using support vector machines. In HLT-NAACL 2004, 2004.
Markus Saers and Dekai Wu. Improving phrase-based translation via word alignments from stochastic
inversion transduction grammars. In SSST-3, pages 28–36, Boulder, CO, Jun 2009.
Markus Saers, Joakim Nivre, and Dekai Wu. Learning stochastic bracketing inversion transduction grammars with a cubic time biparsing algorithm. In IWPT’09, pages 29–32, Paris, France, Oct 2009.
Markus Saers, Joakim Nivre, and Dekai Wu. Word alignment with stochastic bracketing linear inversion
transduction grammar. In NAACL HLT 2010, pages 341–344, Los Angeles, CA, Jun 2010.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In AMTA 2006, pages 223–231, Cambridge, MA, Aug
2006.
Andreas Stolcke. SRILM – an extensible language modeling toolkit. In ICSLP2002 - INTERSPEECH
2002, pages 901–904, Denver, CO, Sep 2002.
Stephan Vogel, Hermann Ney, and Christoph Tillmann. HMM-based word alignment in statistical translation. In COLING-96, volume 2, pages 836–841, 1996.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 344

Zhaojun Wu, Yongsheng Yang, and Pascale Fung. C-ASSERT: Chinese shallow semantic parser, 2006.
Dekai Wu. Trainable coarse bilingual grammars for parallel text bracketing. In WVLC-3, pages 69–81,
Cambridge, MA, Jun 1995.
Dekai Wu. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3):377–403, 1997.
Omar F. Zaidan. Z-MERT: A fully configurable open source tool for minimum error rate training ofmachine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88, 2009.
Hao Zhang and Daniel Gildea. Stochastic lexicalized inversion transduction grammar for alignment. In
ACL-05, pages 475–482, Ann Arbor, MI, Jun 2005.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 345

Wiktionary-Based Word Embeddings
Gerard de Melo

gdm@demelo.org

IIIS, Tsinghua University, Beijing, China

Abstract
Vectorial representations of words have grown remarkably popular in natural language processing and machine translation. The recent surge in deep learning-inspired methods for producing
distributed representations has been widely noted even outside these fields. Existing representations are typically trained on large monolingual corpora using context-based prediction
models. In this paper, we propose extending pre-existing word representations by exploiting
Wiktionary. This process results in a substantial extension of the original word vector representations, yielding a large multilingual dictionary of word embeddings. We believe that thisresource can enable numerous monolingual and cross-lingual applications, as evidenced in aseries of monolingual and cross-lingual semantic experiments that we have conducted.

1

Introduction

Vectorial representations of words have grown to play an important role in natural language
processing and machine translation. Especially for the latter, deep learning and representation
learning-based approaches have recently proven remarkably effective (Sutskever et al., 2014;
Luong et al., 2015). Vector-based encodings of meaning are a central ingredient in many ofthese recent neural machine translation systems, although they can also be beneficial in ordinaryphrase-based machine translation (Mikolov et al., 2013b).
In this work, we focus on the task of creating vector representations of multilingual words(as well as lexicalized phrases). Previous work in this area has relied on multilingual corpora
to train bilingual word vectors. We investigate to what extent external large-scale resources canbe used to create much more multilingual word representation data. In particular, we rely on
Wiktionary, a sister project of Wikipedia that for many years now has been creating a large, collaboratively edited online dictionary. Due to its rich multilingual data, now with over 4 million
entries in over 1,000 languages, Wiktionary has been used extensively in natural language processing, e.g. for part-of-speech tagging (Li et al., 2012) and named entity recognition (Richman
and Schone, 2008), for cross-language image search (Etzioni et al., 2007) and text classification(Nastase and Strapparava, 2013), and for producing language registries (de Melo, 2015) andetymological databases (de Melo, 2014). Wiktionary has also made it possible to translate lexical knowledge bases such as WordNet to hundreds of languages (de Melo and Weikum, 2009)
and to translate thesauri (Borin et al., 2014). Finally, it has been used as an extra ingredient inregular machine translation systems (Göhring, 2014).
Relying on Wiktionary instead of on other training data has playfully been called Wikily supervision (Li et al., 2012). Our work constitutes a form of Wiktionary-based supervisionThis research was supported by China 973 Program Grants 2011CBA00300, 2011CBA00301, and NSFC Grants
61033001, 61361136003, 61450110088.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 346

for multilingual word representation learning. More specifically, our method starts with existing word representations such as the widely available ones trained on large English corpora(Mikolov et al., 2013a; Pennington et al., 2014). It then uses Wiktionary to decide how to place
new words into the same vector space.

2

MethodFor obtaining the new word representations, we adopt the following framework. We assume,
we are given vectors uw for words w ∈ V0 , where V0 is some initial vocabulary of words. Suchvectors may come from any of the popular methods for training word vectors. We later use thewell-known vectors from the word2vec (Mikolov et al., 2013a) and GloVe (Pennington et al.,2014) projects. Our goal is to create new vectors vw for all words w in a substantially larger
vocabulary V , which typically will contain words from many different languages.
Note that the words w are tagged with language codes and are distinguished accordingly.
For example, the Czech word tuna refers to a ton (the weight unit), and the Spanish word tunameans prickly pear/nopal. Neither of these bear any relationship with the fish meaning of the
English word tuna. Thus, we treat words with different language tags as distinct entities withseparate vectors. This, of course, does not preclude connections in the data from encouraging
a high degree of proximity between different vectors. For example, the method will encourage
the English word sushi to have similar vectors to those of the French and Breton sushi, whichhave the same form and meaning.
The vectors vw should reflect the semantics of the words so as to be useful in downstream
applications. While in the past, word vectors were often chosen such that individual dimensions
have some interpretable meaning, current state-of-the-art vector space word embeddings do nothave this property. Instead, we allow for words to be assigned arbitrary vectors as long as vector
similarities and distances reflect corresponding word similarities and distances.
In order to achieve this, we draw on Wiktionary in order to obtain a large set W of semantic
triples taking the form
(w1 , r, w2 )where w1 , w2 are words, and r is a relation that holds between them. The most frequent relation that we obtain from Wiktionary data is the translation relation. Other examples include
synonymy and derivational relationships. Based on the triples in W , we then define the following objective:XX
fW (w1 , w2 ) vtw1 vw2
w1 w2

subject to
kvw k2 ≤ 1 ∀ w,
where fW (w1 , w2 ) should quantify the connection strength (and polarity) between words. Thus,
words are encouraged to have similarities that correspond to their relatedness, measured in termsof their dot products.
In practice, we maximize this objective function iteratively using stochastic gradient ascent. Initially, we set(
uw w ∈ V0
vw =
(1)
0
otherwise
We then repeatedly make local updates for individual triples in order to optimize the vectors in
the direction of the objective. We use two different learning rates α1 , α2 with α1 ≥ α2 for this.
The first one, α1 , is the greater of the two and is the learning rate used for new words, whilst
the second, α2 , is the learning rate used for words that were already in V0 , i.e., the vocabulary

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 347

Figure 1: Wiktionary example, showing an excerpt of the page for the word pulse.
of the input word vectors. Since the original words have already been optimized in some priorlearning process, this severely tempers the extent to which they may be negatively affectedby noise, especially towards the beginning, when the vectors for the new words have not yet
stabilized.
In our experiments, we simply usefW (w1 , w2 ) = |{t ∈ W | ∃r : t = (w1 , r, w2 )}|to quantify relation strengths. While this function is non-negative, the fact that we start off withexisting high-quality word vectors, that we constrain L2 norms to not grow indefinitely, and that
we choose slow learning rates allow us to end up with high-quality word vectors.

3

Wiktionary Parsing

The word relationship triples in W are taken from Wiktionary. Unfortunately, Wiktionary’s
data is created and maintained using a rather informal semi-structured wiki markup form that isdifficult to parse and not very standardized at all. For example, Figure 1 shows just a small partof the page for the word pulse. We rely on a custom information extraction system to produce a
conversion of Wiktionary to structured data, as required for W . This is a rule-based system thatpartitions the raw wiki markup into different parts looking for sections and other subdivisions.
It extracts translations both from the translation sections and from the gloss text, as these are arich resource as well. In the gloss text, we sometimes have short translations, and sometimes
we may also find inflectional or derivational links, as, for instance, in Figure 2.
Table 1 provides details about the extracted data, obtained by applying our parser on an
XML dump of the English edition of Wiktionary (2013-12-17 version). Note that the links
counts refer to the total numbers of directed links after adding inverses and removing any duplicates. We see that Wiktionary provides several million translation links as well as significant
numbers of other relationships, including inflectional and derivational ones.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 348

Figure 2: Simpler Wiktionary example showing the page for the French word déjeûna, which
is listed as an inflected form of déjeûner (to have lunch).
Item

Count

Translational equivalence links
Derivational/Inflectional links
Related term links
Synonymy links
Orthographic/other variant links

3,598,807
2,455,781
580,631490,130
17,357

Unique words

3,968,843

Table 1: Wiktionary input statistics, where link counts refer to directed links after adding inverses and removing duplicates.

4

Experiments

4.1

Training

For the original input vectors, we rely on two well-known sources. The first are the pretrained
word2vec vectors (Mikolov et al., 2013a) released by Google1 . This dataset provides vector
representations for words and multi-word phrases trained on a Google News dataset consistingof about 100B word tokens using word2vec. The vocabulary size is 3,000,000. However, outof these 3,000,000, actually 2,070,978 terms contain a space, most of which are multi-word
expressions or named entities. Thus, the number of genuine words is much smaller.
As a second vector dataset, we experiment with the pre-trained vectors from the GloVe
project (Pennington et al., 2014), which they obtained by applying their algorithm to data froma CommonCrawl corpus consisting of 840B word tokens. The vocabulary size is 2,195,960, outof which none contain a space. While the corpus is larger, it should be noted that CommonCrawl
contains a lot of rather noisy Web data.
We train our model using a starting learning rate of α1 = 0.1 for new words and α2 =0.001 for original words. The vectors stabilize fairly quickly, so we run the algorithm for only
10 epochs.
4.2

Coverage

As a result of this training process using Wiktionary data, the original word2vec representations
are modified from covering 3 million tokens in just a single language to covering nearly 6
1 https://code.google.com/p/word2vec/

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 349

Language# Words (word2vec)# Words (GloVe)

EnglishItalian
Latin
SpanishFrenchGermanFinnishPortuguese
Russian
Serbo-Croatian
Mandarin Chinese
Japanese
PolishDutch
SwedishHungarianDanish...

3,228,842
400,881335,722
242,097
237,189
113,827
110,325
101,077
67,70955,65250,56347,94044,37742,80840,74037,64432,971...

2,417,077
405,544336,668
242,412238,744
114,246
110,613
10,1253
67,86355,77850,51348,02544,54142,90040,81437,70532,988All

5,934,987

5,133,925

Table 2: Top languages after training, using the Google word2vec and GloVe vectors as input,
respectively.
million words in over 500 languages. Similarly, the GloVe vectors are extended from 2,195,960
word vectors in one language to around 5 million vectors, again in over 500 languages. In
Table 2, we list the languages with the greatest coverage on the extended word2vec dataset.
Remarkably, even the coverage of English increases quite substantially, by over 200,000
entries. Although this number might appear small in comparison with the original vocabulary
size of 3,000,000, there is a marked difference in quality between the two. Apart from theroughly 2 million multi-word expressions among these 3 million vocabulary items, the original
data also contains vast amounts of tokens that are not genuine lexical items but simply various
sorts of names, codes, misspellings, file names, and so on (e.g. Krakowiak, SBSA, www.flu.gov,
reccomend, WILLOW). In contrast, the added vocabulary items are mostly genuine word forms,
contributed by Wiktionary’s editors.
Table 3 summarizes the total number of languages covered by the vectors trained on Wiktionary. A lot of rare minority languages are covered to some extent. While the vocabulary
size for them tends to be small, the coverage often focuses on the most important words, suchas those found in Swadesh lists and of interest in linguistic and anthropological studies. 38
languages are covered with a vocabulary size of at least 10,000. For these languages, the coverage should suffice for certain NLP tasks, including cross-lingual ones. This is what we shallinvestigate next.
4.3

Semantic Relatedness

Semantic relatedness studies have a long history in computational lexical semantics. Given a setof word pairs and corresponding scores quantifying how strongly human assessors deem the two
respective words in a word pair semantically related, the goal is automatically produce similarassessment scores. The evaluation is normally carried out in terms of correlation coefficients.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 350

Count (word2vec)

Count (GloVe)

11
38
62
123
267
360

11
38
62
123
266
360

No. of languages with ≥ 50000 words
No. of languages with ≥ 10000 words
No. of languages with ≥ 5000 words
No. of languages with ≥ 1000 words
No. of languages with ≥ 100 words
No. of languages with ≥ 10 words
Table 3: Number of languages.

UKP30

Chandar A P et al. (2014) En-De Vectors
Ours (word2vec)
Ours (GloVe)

0.212 @ 34.5%
0.752 @ 96.6%
0.777 @ 96.6%

GUR65

Chandar A P et al. (2014) En-De Vectors
Faruqui et al. (2015)
Ours (word2vec)
Ours (GloVe)−0.319 @ 26.2%
0.603 @ N/A
0.717 @ 96.9%
0.768 @ 96.9%

GUR350Chandar A P et al. (2014) En-De Vectors
Ours (word2vec)
Ours (GloVe)

0.558 @ 51.7%
0.605 @ 68.3%
0.680 @ 68.0%

ZG222

Chandar A P et al. (2014) En-De Vectors
Ours (word2vec)
Ours (GloVe)

0.111 @ 38.3%
0.161 @ 54.1%
0.306 @ 54.1%

Table 4: German semantic relatedness results, evaluated in terms of Spearman’s rank correlation
coefficient and coverage.

RG65

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.629 @ 55.4%
0.805 @ 100.0%
0.844 @ 100.0%

MC30

Chandar A P et al. (2014) En-Es Vectors
Faruqui et al. (2015)
Ours (word2vec)
Ours (GloVe)

0.430 @ 60.0%
0.591 @ N/A
0.830 @ 76.7%
0.853 @ 76.7%

WS353

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.256 @ 65.1%
0.538 @ 65.6%
0.596 @ 65.6%

Table 5: Spanish semantic relatedness results, evaluated in terms of Spearman’s rank correlation
coefficient and coverage.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 351

JI65

Chandar A P et al. (2014) En-Fr Vectors
Faruqui et al. (2015)
Ours (word2vec)
Ours (GloVe)

0.586 @ 49.2%
0.606 @ N/A
0.822 @ 96.9%
0.827 @ 96.9%

Table 6: French semantic relatedness results, evaluated in terms of Spearman’s rank correlation
coefficient and coverage.
English–German RG65

Chandar A P et al. (2014) En-De Vectors
Ours (word2vec)
Ours (GloVe)

0.441 @ 38.4%
0.812 @ 97.6%
0.828 @ 97.6%

English–Spanish RG65

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.588 @ 59.5%
0.869 @ 100.0%
0.863 @ 100.0%

English–French RG65

Chandar A P et al. (2014) En-Fr Vectors
Ours (word2vec)
Ours (GloVe)

0.598 @ 52.0%
0.864 @ 100.0%
0.855 @ 100.0%

English–Spanish MC30

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.351 @ 70.0%
0.745 @ 90.0%
0.797 @ 90.0%

Spanish–English MC30

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.645 @ 56.7%
0.713 @ 83.3%
0.721 @ 83.3%

English–Spanish WS353

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.303 @ 75.9%
0.582 @ 79.8%
0.641 @ 79.8%

Spanish–English WS353

Chandar A P et al. (2014) En-Es Vectors
Ours (word2vec)
Ours (GloVe)

0.299 @ 73.3%
0.550 @ 78.7%
0.612 @ 78.7%

Table 7: Cross-lingual semantic relatedness results

These quantify to what degree the word pairs turn out to be in a similar order when sortingwith respect to the two kinds of relatedness scores – ground-truth human-provided ones vs.
automatically generated ones.
Although semantic relatedness assessment is not an end-user task in itself, it is an importantbuilding block in numerous NLP systems. For instance, measures of semantic relatedness can beused in search query expansion, text classification, and schema and ontology matching, among
many others.
Following Pennington et al. (2014), we use cosine similarity over normalized vectors. The
word vectors we generate are case-sensitive, distinguishing Reading, which often refers to thecity, from reading, which often refers to the process of reading. However, some of the datasets
do not preserve case and so we also consider any possible capitalized version of the input word.
Whenever at least one word has multiple candidate vectors, we take the maximum similarity
over all pairs.
We evaluate this method using Spearman’s rank correlation coefficients over all covered

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 352

word pairs, while reporting the respective coverage percentage. In computing Spearman’s ρ, we
follow the recommended procedure of using average ranks for tied positions.
While most semantic relatedness datasets focus on English, there are a few non-English
ones as well, which we can use to evaluate our system. Unfortunately, their number is rather
small, so we are limited in the number of languages that we can readily consider in this sortof evaluation. We use several publicly released datasets that were often based on pre-existing
English datasets2 . In Table 4 we provide evaluation results on German-language datasets, while
Tables 5 and 6 provide similar results on Spanish and French datasets. For comparison, we list
all published results known to us that are also based on vectors as well as results on all other
non-English word vectors we could obtain and evaluate directly. In all cases, we see that ourvectors fare significantly better than the competitors.
4.4

Cross-Lingual Semantic Relatedness

Semantic relatedness can also be evaluated across languages. We adopt the same methodology
as earlier but rely on the Spanish-English evaluation data from Hassan and Mihalcea (2009),
which we can use to compare our vectors with those of Chandar A P et al. (2014). Further,
we consider the new cross-lingual semantic relatedness evaluation data released by CamachoCollados et al. (2015), which is based on the RG65 dataset. Our results on these cross-lingual
datasets are listed in Table 7. Again, our Wiktionary-based representations compare favorablywith other available results.
4.5

Word Choice Problems

Word choice problems consist of a target word and a selection of possible words or phrases
describing it. Consider the following three examples.
gourmeta) enjoys cookingb) has indigestionc) has an expert appreciation of foodd) is hungry

dalea) plainb) retreatc) shelterd) valley

bracea) to screamb) prepare for dangerc) hold your breathd) close your eyes

Here, the correct answers are c) for gourmet, d) for dale, and b) for brace. For English, werely on a well-known dataset used by Mohammad et al. (2007). We also use a large Germanlanguage collection of similar quiz questions3 . The latter consists of 984 problem instancescollected from 2001 to 2005 editions of the German version of Reader’s Digest Magazine,
where they appear as “Word Power” problems.
We compute cosine similarities between the target word and the candidate answers. Someanswers are individual words or expressions already covered in our data, in which case this issimple. If a candidate answer, however, consists of multiple words that are not covered in ourdata as a multi-word expression, we simply use the maximum cosine similarity between any of
the words in the answer phrase and the target word.
We assess the accuracy as the sum of scores over all problem instances divided by the
number of problem instances. Following the convention from previous work (Mohammad et al.,2007), the score is 1 if the correct answer is ranked highest among the candidates, 0 if it is not
2 For more information on these datasets, please refer to https://www.ukp.tu-darmstadt.de/data/
semantic-relatedness/german-relatedness-datasets/ as well as Hassan and Mihalcea (2009) andCamacho-Collados et al. (2015).
3 https://www.ukp.tu-darmstadt.de/data/semantic-relatedness/
german-word-choice-problems/Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 353

Dataset

Vectors

Accuracy

EnglishChandar A P et al. (2014) En-De Vectors
Original word2vec input vectors
Ours (word2vec)
Original GloVe input vectors
Ours (GloVe)

27.42%
65.31%
76.49%
68.54%
74.42%

GermanChandar A P et al. (2014) En-De Vectors
Ours (word2vec)
Ours (GloVe)

27.35%
40.91%
40.85%

Table 8: Accuracy results on English and German word choice problems
ranked highest, and n1 if our method’s top ranked answers form a tie of n answers with the same
similarity score.
The results are provided in Table 8. Although our method for handling phrases is very
simplistic, we obtain reasonably good results. Somewhat surprisingly, we quite significantly
improve over the original input vectors for English. The contribution could come from the English lexical information in Wiktionary as well as from the cross-lingual relationships extracted
from Wiktionary.
For German, the results are not as good as for English, which, however, is mainly due
to the morphological complexity of the phrases in German. Better results could easily be obtained by improving the linguistic analysis of candidate answers, for instance by performing
lemmatization, stop word removal or interpretation, and compound splitting, which, of course,
is particularly helpful for German with its notoriously long compound nouns. After that, one
could then use our vectors to obtain more reliable similarity scores.
4.6

Word and Entity Analogies

Mikolov et al. (2013c) showed that distributed word vectors trained on large corpora using
prediction approaches may exhibit intriguing semantic and linguistic regularities, even if these
are not in any way directly part of their training objective. For instance, in their results, thevectors for king and queen stand roughly in the same relationship to each other as the vectors
for uncle and aunt, or man and woman. This works to the extent that simple vector arithmetic
often produces a vector whose nearest known word vector is the correct answer.
In order to create a non-English analogy dataset, we took the semantic analogies datasetof Mikolov et al. (2013c) and filtered out the parts focusing on geography, as these are to alarge extent language-independent names like Portland or Alaska. This left us with the family
and male/female related analogies. From these, we randomly selected 50 examples and createdthe corresponding French-language analogies. When multiple different translations appeared
reasonable, we first restricted the choice based on the register (maman for mom but mère formother) and then used the most popular form in the few cases where more than one option remained, e.g. belle-mère for stepmother rather than marâtre, which typically has the connotationof implying an evil stepmother.
For each analogy entry, the first two words demonstrate the analogy, and for the second
pair of words, only the first is given as input to the system. The goal is predict the second
one. We follow Mikolov et al. (2013c) in computing the target vector using simple vector
arithmetic. We then find words near that target vector by choosing the nearest neighbors interms of the Euclidean distance, considering only French words so as to obtain an answer in thecorrect target language. On our dataset of 50 French analogies, we obtain the results shown inProceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 354

Vectors
Chandar A P et al. (2014) En-Fr Vectors
Ours (word2vec)
Ours (GloVe)

Accuracy
2.0%
30.0%
35.0%

Table 9: Accuracy results on the French word analogy task

Table 9. Note that the French dataset is somewhat more challenging than the English original,
because some translations are polysemous and no longer retain the sense distinctions of the
English originals. For instance, both girl and daughter correspond to fille in French. While theapproach by Chandar A P et al. (2014) does extremely poorly, our vectors achieve reasonable
results. In those cases where they return the wrong answer, the correct one is often among the
top 3.

5

Background and Related Work

Distributed representations in neural networks go back to at least Rumelhart et al. (1986), who
described, for their well-known family tree case study, how weights distributed across different
input units can be used to describe people. Importantly, their representation allowed two different people from separate families to share most weights if their other attributes were similar.
The lineage of the distinct idea of distributional semantics can be traced to use theories of
linguistic meaning, which, roughly speaking, hold that language use in context determines themeaning of a word. This view fits well with the idea of empiricist corpus linguistics and thecomputational goals of discerning aspects of meaning using data-driven methods. Thus, distributional methods have received considerable attention in natural language processing (Schütze,1993). Over time, it became apparent that one of the challenges with many distributional methods is the sparsity of observed word co-occurrences in a corpus in comparison with the overall
distribution of likely word co-occurrences. Since many distributional approaches use numerical vectors to represent the contexts, this sparsity often is manifested in the form of sparse
vectors with many zeros. Smoothing techniques and algorithms such as Latent Semantic Analysis (Deerwester et al., 1990) were proposed to alleviate some of these problems.
More recently, distributed and distributional methods have grown together in the form ofneural network-inspired architectures that learn distributed representations from large corpora
by accounting for word co-occurrences (Collobert et al., 2011; Turian et al., 2010). The resulting representations are still vectorial and based on corpus co-occurrences, but much lowerdimensional than in traditional distributional approaches and thus significantly less sparse. The
massive attention on deep learning in recent years, paired with fast training methods as in theword2vec method by Mikolov et al. (2013a), which actually forgoes deep learning, has propelled these methods to the forefront of NLP, to the point that they are known well beyond thecore natural language processing community.
Subsequently, a number of improvements to the learning algorithms have been proposed.
Our objective function is related to those of other models that aim to exploit similarities between words. Chen et al. (2015) extend the word2vec CBOW objective function in order to pay
special attention to contexts that reveal more explicit semantic relationships, rather than treating all contexts as equal. The semantic relationships are obtained using information extraction
methods, e.g. from lists and definitions. Yu and Dredze (2014) and Faruqui et al. (2015) propose to optimize monolingual word embeddings so as to match information from pre-existinglexical resources. Hill et al. (2015) used dictionary glosses from several resources (including
Wiktionary) in order to train neural networks to produce vectors for multi-word phrases.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 355

While most word representation learning research has been monolingual aiming at English,
recently there has been some interest in multilingual aspects of it.
Some works take pre-existing vectors for different languages and connect them. Mikolov
et al. (2013b) develop a method to learn projections between two monolingual word embedding
spaces. Lazaridou et al. (2015) investigate means to improve such projections. Faruqui andDyer (2014) propose using canonical correlation analysis (CCA), while Lu et al. (2015) suggest
using Deep CCA instead. Our method, in contrast, does not assume that we have already created
non-English word vectors. We only rely on English word vectors, which are readily available
from numerous sources.
A number of studies have focused on using multilingual corpora, often parallel corpora,
to produce bilingual word vector spaces (Kalchbrenner and Blunsom, 2013; Kočiský et al.,2014). Utt and Padó (2014) investigate using syntax for bilingual vector space models. Hermann and Blunsom (2014) create bilingual word representations without word alignment. Hillet al. (2014) showed that word embeddings obtained from translations better reflect the ontological status of words than regular neural embeddings. One advantage of corpus-based approaches
is the potential to have a substantial coverage, given sufficiently large corpora. However, as theamount of available parallel text is somewhat limited, in practice, this advantage may only apply to methods that do not require parallel corpora. Our work is complementary to this lineof research on corpus-driven approaches. We exploit the availability of high-quality word vectors for English trained on very large Web-scale data, leading to word vector spaces that reflect
word analogies well. We further draw on the availability of multilingual lexical resources suchas Wiktionary, covering hundreds of languages, including lesser-resourced ones, for which corpora may be difficult to obtain.
More generally, our work differs from previous work by going beyond bilingual vector
spaces in order to place millions of word forms from different languages into a single sharedvector space.

6

Conclusion

We have presented the first study to produce large amounts of word vectors from Wiktionary
in many languages. Unlike previous work on bilingual word embedding spaces, our approachproduces a single significantly multilingual word vector space rather than just bilingual ones.
Our experiments show that our vectors reflect semantic properties and that they are useful bothin monolingual and in cross-lingual settings.
In the future, we would like to investigate the potential of these multilingual vectors formachine translation of text. While deep recurrent neural network architectures have recently
achieved state-of-the-art results in several machine translation settings, they still suffer fromsignificant problems with out-of-vocabulary items (Sutskever et al., 2014; Luong et al., 2015).
Rather than only addressing these with custom ad hoc techniques as in the approach taken by
Luong et al. (2015), it would be helpful to investigate to what extent we can incorporate themwithin the same vector space.

References
Borin, L., Allwood, J., and de Melo, G. (2014). Bring vs. MTRoget: Evaluating automatic thesaurus
translation. In Proceedings of the 9th Language Resources and Evaluation Conference (LREC 2014),
Paris, France. ELRA.
Camacho-Collados, J., Pilehvar, M. T., and Navigli, R. (2015). A framework for the construction ofmonolingual and cross-lingual word similarity datasets. In Proceedings of the 53rd Annual Meeting of
the Association for Computational Linguistics (ACL 2015).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 356

Chandar A P, S., Lauly, S., Larochelle, H., Khapra, M., Ravindran, B., Raykar, V. C., and Saha, A.
(2014). An autoencoder approach to learning bilingual word representations. In Advances in Neural
Information Processing Systems 27 (NIPS 2014), pages 1853–1861. Curran Associates, Inc.
Chen, J., Tandon, N., and Gerard de Melo (2015). Neural word representations from large-scale commonsense knowledge. In Proceedings of the IEEE/WIC/ACM Web Intelligence Conference and theIEEE/WIC/ACM Intelligent Agent Technology Conference 2015 (WI/IAT 2015).
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011). Natural language
processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.
de Melo, G. (2014). Etymological Wordnet: Tracing the history of words. In Proceedings of the 9th
Language Resources and Evaluation Conference (LREC 2014), Paris, France. ELRA.
de Melo, G. (2015). Lexvo.org: Language-related information for the linguistic linked data cloud. Semantic Web, 6(4).
de Melo, G. and Weikum, G. (2009). Towards a Universal Wordnet by learning from combined evidence. In Proceedings of the 18th ACM Conference on Information and Knowledge Management(CIKM 2009).
Deerwester, S., Dumais, S., Furnas, G., Landauer, T., and Harshman, R. (1990). Indexing by latentsemantic analysis. Journal of the American Society for Information Science, 41(6):391–407.
Etzioni, O., Reiter, K., Soderland, S., and Sammer, M. (2007). Lexical translation with application toimage search on the web. In Proceedings of Machine Translation Summit XI.
Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., and Smith, N. A. (2015). Retrofitting word
vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies (NAACL 2015).
Faruqui, M. and Dyer, C. (2014). Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2014).
Göhring, A. (2014). Building a spanish-german dictionary for hybrid mt. In Proceedings of the 3rdWorkshop on Hybrid Approaches to Translation (HyTra) at EACL 2014, pages 30–35. Association for
Computational Linguistics.
Hassan, S. and Mihalcea, R. (2009). Cross-lingual semantic relatedness using encyclopedic knowledge. In
Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP
2009), Singapore.
Hermann, K. M. and Blunsom, P. (2014). Multilingual Distributed Representations without Word Alignment. In Proceedings of the International Conference on Learning Representations (ICLR) 2014.
Hill, F., Cho, K., Jean, S., Devin, C., and Bengio, Y. (2014). Not all neural embeddings are born equal.
CoRR – Computing Research Repository – arXiv, abs/1410.0718.
Hill, F., Cho, K., Korhonen, A., and Bengio, Y. (2015). Learning to understand phrases by embedding thedictionary. CoRR – Computing Research Repository – arXiv, abs/1504.00548.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings ofthe 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), Seattle.
Association for Computational Linguistics.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 357

Kočiský, T., Hermann, K. M., and Blunsom, P. (2014). Learning Bilingual Word Representations by
Marginalizing Alignments. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2014), Volume 2: Short Papers.
Lazaridou, A., Dinu, G., and Baroni, M. (2015). Hubness and pollution: Delving into cross-space mapping
for zero-shot learning. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics (ACL 2015).
Li, S., Graça, J. a. V., and Taskar, B. (2012). Wiki-ly supervised part-of-speech tagging. In Proceedingsof the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 1389–1398, Stroudsburg, PA, USA.
Association for Computational Linguistics.
Lu, A., Wang, W., Bansal, M., Gimpel, K., and Livescu, K. (2015). Deep multilingual correlation forimproved word embeddings. In Proceedings of the 2015 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies (NAACL 2015), pages
250–256, Denver, Colorado. Association for Computational Linguistics.
Luong, T., Sutskever, I., Le, Q., Vinyals, O., and Zaremba, W. (2015). Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for
Computational Linguistics (ACL 2015).
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). Efficient estimation of word representations in
vector space. CoRR – Computing Research Repository – arXiv, abs/1301.3781.
Mikolov, T., Le, Q. V., and Sutskever, I. (2013b). Exploiting similarities among languages for machinetranslation. CoRR – Computing Research Repository – arXiv, abs/1309.4168.
Mikolov, T., tau Yih, W., and Zweig, G. (2013c). Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies (NAACL-HLT-2013). Association for
Computational Linguistics.
Mohammad, S., Gurevych, I., Hirst, G., and Zesch, T. (2007). Cross-lingual distributional profiles ofconcepts for measuring semantic distance. In 2007 Joint Conference on Empirical Methods in Natural
Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007).
Nastase, V. and Strapparava, C. (2013). Bridging languages through etymology: The case of cross language text categorization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), pages 651–659. The Association for Computer Linguistics.
Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP
2014), pages 1532–1543.
Richman, A. and Schone, P. (2008). Mining Wiki Resources for Multilingual Named Entity Recognition.
In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human
Language Technologies (ACL-08: HLT), pages 1–9.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating
errors. Nature, 323:533–536.
Schütze, H. (1993). Word space. In Advances in Neural Information Processing Systems 5 (NIPS 1992).

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 358

Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In
Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal, CA.
Turian, J., Ratinov, L., and Bengio, Y. (2010). Word representations: A simple and general method forsemi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), pages 384–394.
Utt, J. and Padó, S. (2014). Crosslingual and multilingual construction of syntax-based vector spacemodels. Transactions of the Association of Computational Linguistics, 2:245–258.
Yu, M. and Dredze, M. (2014). Improving lexical embeddings with semantic knowledge. In Proceedingsof the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2014), Volume 2:
Short Papers.

Proceedings of MT Summit XV, vol.1: MT Researchers' Track

Miami, Oct 30 - Nov 3, 2015 | p. 359