NAIST-IS-DD1661017

Doctoral Dissertation

Improving Low-Resource Machine Translation
through Syntactic and Contextual Information

Akiva Miura

March 8, 2018

Graduate School of Information Science
Nara Institute of Science and Technology

A Doctoral Dissertation
submitted to Graduate School of Information Science,
Nara Institute of Science and Technology
in partial fulfillment of the requirements for the degree ofDoctor of ENGINEERING
Akiva Miura
Thesis Committee:
Professor Satoshi Nakamura
Professor Yuji Matsumoto
Associate Professor Katsuhito Sudoh
Assistant Professor Graham Neubig
Doctor Masao Utiyama(Supervisor)(Co-supervisor)(Co-supervisor)(Carnegie Mellon University)(NICT)

Improving Low-Resource Machine Translation
through Syntactic and Contextual Information∗
Akiva Miura

Abstract
Translation is an essential tool to communicate with foreign language speakers.
However, it requires specialized knowledge, and thus expectations are heightening
toward machine translation (MT), which has potential to perform manual translation tasks in an automated fashion. Nowadays many practical applications ofMT translate from English into other widely spoken languages and vice versa.
On the other hand, MT quality has not yet reached a practical level in many
language pairs that do not include English.
The current mainstream MT frameworks are statistical MT and neural MT,
which are characterized by the ability to learn to translate automatically throughmachine learning techniques. It has been observed that translation with models
trained on larger parallel corpora can achieve higher accuracy, and usually millions of sentence pairs are required in order to produce a high quality translation.
Unfortunately, readily available parallel corpora are limited for most language
pairs, particularly those that do not include English, having few sentence pairs,
or none at all. Moreover, the cost of manually producing a high quality parallel
corpus is estimated to be in the millions of dollars.
In this thesis, we focus on improving MT quality with two types of approaches
for coping with the scarceness of bilingual corpora (1) pivot translation and (2)active learning for MT.
Pivot translation is a useful method for translating between languages withlittle or no parallel data by utilizing parallel data in an intermediate (pivot) language such as English. Although various methods using pivot languages have∗

Doctoral Dissertation, Graduate School of Information Science,
Nara Institute of Science and Technology, NAIST-IS-DD1661017, March 8, 2018.

ibeen proposed, ambiguity due to expressions in the pivot language often causes
incorrect selection of translation rules and harms translation quality. Therefore,
pivot-side disambiguation is a key issue in pivot translation. In the first part of
the thesis, we propose two new pivot translation methods to solve the two types ofambiguity respectively. The first method is proposed to solve semantic ambiguity,and lets MT models remember the information of the pivot phrase. This information can help to select appropriate translation rules considering pivot-side context
with pivot language models. The second method is proposed to solve syntactic
ambiguity, and introduces an explicitly syntax-aware matching condition to findcorrect correspondence of source-pivot and pivot-target translation rules, and can
produce more reliable models. Experimental results on multilingual translation
show a significant improvement in all the tested language pairs.
Active learning is a framework that makes it possible to eﬃciently train statistical models by selecting informative examples from a pool of unlabeled data. Previous work has found this framework eﬀective for MT, making it possible to train
better MT models with less eﬀort, particularly when annotators translate shortphrases instead of full sentences. However, previous methods for phrase-based
active learning for MT fail to consider whether the selected units are coherent
and easy for human translators, and also have problems with selecting redundant phrases with similar content. In this part, we propose two new methods
for selecting more syntactically coherent and less redundant segments in active
learning for MT. Experiments using both simulation and extensive manual translation by professional translators find the proposed method eﬀective, achieving
both greater gain of translation score for the same number of translated words,
and allowing translators to be more confident in their translations.
Our experiments demonstrate that MT quality can significantly benefit fromsyntactic and contextual information when faced with limited training data.

Keywords:
pivot translation, active learning, machine translation, parallel corpus, low-resource
language pairsiiAcknowledgementsὁ Θεὸς γάρ ἐστιν ὁ ἐνεργῶν ἐν ὑμῖν καὶ τὸ θέλειν
καὶ τὸ ἐνεργεῖν ὑπὲρ τῆς εὐδοκίας.
(ΠΡΟΣ ΦΙΛΙΠΠΗΣΙΟΥΣ 2:13)『神は御意を成さんために汝らの衷にはたらき、汝等をして志望をたて、業を⾏はしめ給へばなり。』(ピリピ⼈への書 2 章 13 節 ⼤正改訳)

私のうちに志と使命を与え，今⽇まで導いてくださった主イエス・キリストの御名を讃えます．本当に多くの⽅々に助けられ，⽀えられながら博⼠論⽂の執筆に⾄ることができました．
本学情報科学研究科の中村哲教授には，博⼠前期・後期課程を通じての 4
年間，指導教官として常に熱⼼なご教鞭をいただきました．⼊学時から希望していた研究テーマに挑戦させていただき，なかなか成果が出ない時にも⾟抱強くご指導いただきました．また，⾝に余るほどのご⽀援をいくつも承りました．⼼より感謝いたします．
本学情報科学研究科の松本裕治教授には，副指導教官として⼤変貴重なご指導とご助⾔をいただきました．⼼より感謝いたします．
本学情報科学研究科の須藤克仁准教授 (前 NTT コミュニケーション科学基礎研究所) には，副指導教官として熱⼼なご教⽰とご助⾔をいただきました．
NTT の研究所にご在籍されていた頃より関⻄機械翻訳勉強会をはじめ，様々な会合でお会いしては研究者⼈⽣として興味深いお話を聞かせていただきました．特に本学に異動して来られてからは多くのご助⾔をいただき，前進することができました．⼼より感謝いたします．
Carnegie Melon University の Graham Neubig 助教 (前 奈良先端科学技術⼤学院⼤学 助教) には，副指導教官として熱⼼なご教⽰とご助⾔をいただきました．⼊学時より研究テーマに全⾯的に熱意をもってご指導いただき，全くiii

未経験であった論⽂執筆や研究発表に対しても丁寧にご助⾔いただきました．
本学で執筆したすべての論⽂を最初に添削してくださり，とりわけ，国際会
議に複数の英語論⽂を投稿し, 採択されたのは先⽣のご指導無しには成し得ませんでした．私の在学中に，本学より Carnegie Melon University へ異動されたため，4 年間のうち約 2 年間は，遠隔地からご指導いただきましたが，毎週のように個別でミーティングを⾏っていただき，⼿厚いご指導を承りました．⼼より感謝いたします．
情報通信研究機構の内⼭将夫様には，副審査委員として⼤変貴重なご指導とご助⾔をいただきました．⼼より感謝いたします．
名古屋⼤学情報科学研究科の⼾⽥智基教授 (前 奈良先端科学技術⼤学院⼤学 准教授) には，本学勤務時において，研究全般にわたり貴重なご助⾔をいただきました．⼼より感謝いたします。
本学情報科学研究科の鈴⽊優特任准教授，Sakriani Sakti 特任准助教，吉野幸⼀郎助教，⽥中宏季助教には，研究室ゼミナールや研究班ミーティング等
において貴重なご助⾔をいただきました．⼼より感謝いたします．
知能コミュニケーション研究室秘書の松⽥真奈美様には，研究室の活動を通して様々な⾯でお世話になりました．⼼より感謝いたします．
株式会社 ATR-Trek の Michael Paul 様，中澤聡様，中村明様には，機械翻訳のための能動学習⼿法に関する研究において，活発な議論の場を設け数々の有益なご助⾔をいただいたことで，研究を深めることができました．⼼より感謝いたします．
⾸都⼤学東京の⼩町守准教授には，私が機械翻訳の研究を願って進路に悩んでいた際に，メールで相談に乗ってくださり，本学への⼊学を勧めてくださりました．これが⼤きなきっかけとなり，本学の博⼠前期課程へ⼊学し，後に博⼠後期課程へと進学しましたが，お聞きしていた情報の通り，本当に恵
まれた環境で学業に邁進することができました．情報処理学会の⾃然⾔語処理研究会で発表を⾏った際には，毎回質問してくださったり，研究室にお呼びいただき学⽣達との議論の場を⽤意してくださったりもしました．⼼より感謝いたします．
情報通信研究機構の藤⽥篤様には，私の研究に関⼼を寄せていただき，学会や勉強会でお会いしては声をかけてくださったり，様々な⾯でお世話になる機会がありました．洞察が鋭く，的確なご意⾒をいただく度に，新しい着眼点を得ることができ，また，研究への取り組みも正されました．⼼より感iv

謝いたします．
最後になりますが，私が 18 歳で実家を離れてからも暖かく⽀え続けてくれた⺟・恵⼦と，修⼠論⽂執筆直前に急逝した⽗・純⼆には，返しきれない恩を感じつつ，⼼より感謝いたします．妻・彩華には，博⼠後期課程の 2 年間，
苦しい⽇々が続く中，常に⽀え続けてくれました．⼼より感謝いたします．

v

Contents
Acknowledgementsiii

List of Figuresix

List of Tablesxi

1 Introduction
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Known Issues in Multilingual Machine Translation . . . . . . . . .
1.3 Approaches toward Low-Resource Machine Translation . . . . . .
1.4 Thesis Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.1 Better Pivot Translation by Remembering the Pivot . . . .
1.4.2 Syntactic Matching Methods in Pivot Translation . . . . .
1.4.3 Syntactic and Non-Redundant Segment Selection for Active
Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Document Structure . . . . . . . . . . . . . . . . . . . . . . . . .

1
1
1
2
3
4
5

2 Machine Translation Frameworks
2.1 Statistical Machine Translation . . . . . . . . . . . . . . . . . . .
2.1.1 Translation Models . . . . . . . . . . . . . . . . . . . . . .
2.1.2 Language Models . . . . . . . . . . . . . . . . . . . . . . .
2.1.3 Phrase-Based SMT . . . . . . . . . . . . . . . . . . . . . .
2.1.4 Synchronous Context-Free Grammars . . . . . . . . . . . .
2.1.5 Hierarchical Rules . . . . . . . . . . . . . . . . . . . . . . .
2.1.6 Explicitly Syntactic Rules . . . . . . . . . . . . . . . . . .
2.1.7 Multi-Synchronous Context-Free Grammars . . . . . . . .
2.2 Neural Machine Translation . . . . . . . . . . . . . . . . . . . . .
2.2.1 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10
12
13
14
15
17
18
19
20
20
21

vi

7
8

2.2.2
2.2.3

Attention Mechanism . . . . . . . . . . . . . . . . . . . . .
Decoder . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22
23

3 Low-Resource Machine Translation
3.1 Pivot Translation . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1 Sequential Pivot Translation . . . . . . . . . . . . . . . . .
3.1.2 Pseudo-Parallel Corpus Synthesis . . . . . . . . . . . . . .
3.1.3 Triangulation of Translation Models . . . . . . . . . . . . .
3.1.4 Problems of Pivot-Side Ambiguity . . . . . . . . . . . . . .
3.1.5 Related Work . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Active Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Active Learning for Machine Translation . . . . . . . . . .
3.2.2 Sentence Selection using N -Gram Frequency . . . . . . . .
3.2.3 Phrase Selection using N -Gram Frequency . . . . . . . . .

24
25
25
26
27
28
29
32
32
33
34

4 Contextual Disambiguation in Pivot Translation
4.1 Word Sense Ambiguity . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Triangulation Remembering the Pivot . . . . . . . . . . . . . . . .
4.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . .
4.3.2 Results and Analysis . . . . . . . . . . . . . . . . . . . . .
4.3.3 Influence of Pivot Language Model Strength . . . . . . . .
4.3.4 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . .
4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35
35
37
38
38
40
42
42
45

5 Syntactic Disambiguation in Pivot Translation
5.1 Syntactic Ambiguity . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Triangulation with Syntactic Information . . . . . . . . . . . . . .
5.2.1 Exact Matching of Parse Subtrees . . . . . . . . . . . . . .
5.2.2 Partial Matching of Parse Subtrees . . . . . . . . . . . . .
5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . .
5.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.3 Comparison with Neural MT: . . . . . . . . . . . . . . . .

46
46
48
48
49
50
50
52
58

vii

5.4

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

60

6 Syntactic and Non-Redundant Segment Selection for Active Learning61
6.1 Compact and Syntactically Coherent Segment Selection . . . . . . 61
6.1.1 Segment Selection based on Phrase Maximality . . . . . . 62
6.1.2 Phrase Selection based on Parse Trees . . . . . . . . . . . 64
6.2 Simulation Experiment . . . . . . . . . . . . . . . . . . . . . . . . 66
6.2.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . 66
6.2.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . 68
6.3 Manual Translation Experiment . . . . . . . . . . . . . . . . . . . 75
6.3.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . 75
6.3.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . 77
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7 Conclusion
7.1 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . .

81
81
82

Bibliography

85

Publication List

93

viii

List of Figures
1.1

1.3

An example of (a) triangulation and the resulting phrases in the(b) traditional method of forgetting pivots and (c) our proposed
method of remembering pivots. . . . . . . . . . . . . . . . . . . .
Example of disambiguation by parse subtree matching (Fr-En-Zh),[X1] and [X2] are non-terminals for sub-phrases. . . . . . . . . . .
Conventional and proposed data selection methods . . . . . . . .

2.1
2.2
2.3

En-Ja word alignment . . . . . . . . . . . . . . . . . . . . . . . .
En-Ja phrase extraction . . . . . . . . . . . . . . . . . . . . . . .
Attention-based neural MT . . . . . . . . . . . . . . . . . . . . .

16
16
21

3.1
3.2
3.3
3.4
3.5
3.6
3.7

Sequential pivot translation . . . . . . . . . . . . . . . . . . . . .
Multi-sentential method . . . . . . . . . . . . . . . . . . . . . . .
Pseudo-parallel corpus synthesis . . . . . . . . . . . . . . . . . . .
Triangulation of translation models . . . . . . . . . . . . . . . . .
An example of ambiguity in De-En-It triangulation. . . . . . . . .
Levels of translation (“Vauquois pyramid”) . . . . . . . . . . . . .
An example of n-gram selection method (n = 4) . . . . . . . . . .

25
26
26
27
29
31
34

4.1
4.2
4.3
4.4

Estimation of source-target word correspondence . . . . . . . . . .
Standard triangulated phrases . . . . . . . . . . . . . . . . . . . .
Proposed triangulated phrases . . . . . . . . . . . . . . . . . . . .
Influence of pivot LM size on pivot translation accuracy . . . . . .

35
36
36
42

5.1

Example of disambiguation by parse subtree matching (Fr-En-Zh)

46

6.1
6.2

Conventional and proposed data selection methods . . . . . . . .
Phrase selection based on parse trees . . . . . . . . . . . . . . . .

62
65

1.2

ix

5
6
7

6.3

6.4
6.5

BLEU score vs. number of additional source words in each method(upper: En-Fr translation task, bottom: En-Ja translation task,
left: up to 100k additional words, right: up to 1M additional words) 70
Example of the human translation interface . . . . . . . . . . . . 75
Transition of BLEU score vs. additional source words (left) andvs. cumulative working duration (right) . . . . . . . . . . . . . . . 76x

List of Tables
4.1

5.1
5.2
5.3
5.4
5.5
5.6
6.1
6.2

6.3
6.4

6.5
6.6
6.7

Results of each method, comparing the proposed triangulation remembering the pivot with other methods. . . . . . . . . . . . . .

40

Results of each method, comparing the proposed syntactic matching methods in triangulation with other methods. . . . . . . . . . 52
Comparison of rule table coverage in proposed triangulation methods. 53
Comparison of noise ratio in triangulated rule table . . . . . . . . 54
Comparison of distribution error rate in triangulated rule table . . 55
Main parameters of NMT training . . . . . . . . . . . . . . . . . . 58
Comparison of SMT and NMT in multilingual translation tasks. . 59
Details of parallel data . . . . . . . . . . . . . . . . . . . . . . . .
Transition of BLEU score according to the number of additional
words. Underlines indicate that the score is the maximum among
the random selection method and the baseline method at eachpoint in time immediately after the addition of 1M words. Boldface indicates the scores the proposed method exceeding the underlined scores. Daggers † indicate the best score in all the methods
in each stage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Number of segments and average words/segment in each method .
Eﬀect on coverage in each selection method (rounded oﬀ to thesecond decimal place). Bold face indicates the highest coverage foreach number of additional words. . . . . . . . . . . . . . . . . . .
Reduction amount of duplicated segments selected in 4gram-freq .
Total working time and statistics of confidence level evaluation . .
Detail of selected segments by length . . . . . . . . . . . . . . . .

xi

67

69
72

74
75
77
77

6.8

Average working time of manual translation corresponding to segment length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.9 Average confidence level of manual translation corresponding tophrase length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.10 BLEU score when training on phrases with a certain confidence level 79

xii

1 Introduction
1.1 Background
Language is a key communication tool for human beings, and also signifying
group identity deeply rooted in social and cultural background. Translation isan essential tool to communicate with foreign language speakers. However, it
requires specialized knowledge, and thus expectations are heightening towardmachine translation (MT), which has potential to perform translation tasks in an
automated fashion. Nowadays many practical applications of MT translate fromEnglish into other widely spoken languages and vice versa. On the other hand,
MT quality has not yet reached a practical level in many language pairs that do
not include English. Therefore, it is hard to say that users who are not familiar
with English can use MT between various languages without diﬃculty.

1.2 Known Issues in Multilingual MachineTranslation
The most traditional framework of MT is Rule-Based Machine Translation (RBMT(Nirenburg, 1989)), which is implemented by manual description of translation
rules. This requires knowledge of experts familiar with both of the source andtarget languages, and thus it is diﬃcult to cover a wide variety of expressions inmany language pairs. Therefore, this thesis discusses the framework of Statistical Machine Translation (SMT (Brown et al., 1993)), which can automaticallyobtain translation rules from given bilingual corpora (also referred to as “parallel corpora”), consisting of a set of sentences in two languages, through machinelearning. It has been observed that translation with models trained on larger parallel corpora can achieve higher accuracy, and usually millions of sentence pairs

1are required in order to produce a high quality translation system (Dyer et al.,2008; Koehn et al., 2007). Such large bilingual corpora can be obtained for a
few language pairs through the on-going translation eﬀorts of organizations suchas the Canadian parliament (English-French), the United Nations (6 oﬃcial languages, including Arabic and Chinese), and the European Parliament (spanning
21 European languages).
Unfortunately, readily available parallel corpora are limited for most otherlanguage pairs, particularly those that do not include English, having fewer than
100k sentence pairs, or none at all. Moreover, the cost of manually producing
a high quality parallel corpus is estimated to be in the millions of dollars. For
example, Germann (2001) estimated the cost of hiring professional translators tocreate a Tamil-English corpus at $0.36 per word, or in other words, $1.44M for200k sentences with a 20 words per sentence average.
Indeed, this scarcity of parallel corpora makes it diﬃcult to construct reasonably performing MT systems for most language pairs, a problem which has
therefore received attention by both researchers and industries and is referred toas “low-resource machine translation” (Irvine and Callison-Burch, 2013; Lopez
and Post, 2013). Advances in this field could facilitate communication acrosscultures, enable faster commercial expansion to new growing markets, and may
even assist during disaster relief operations (Munro, 2013).

1.3 Approaches toward Low-Resource MachineTranslation
As mentioned above, it is not practical to create a bilingual corpus in a straightforward manner from the viewpoint of budget and time cost. There are two
possible approaches to MT between low-resource language pairs.
Eﬀective use of indirectly available data: One approach is an indirect
method reusing existing data to realize translation in the intended language
pairs even if no direct source-target parallel corpus is available. An eﬀective
and representative method of this approach is to introduce a pivot language forwhich parallel data with the source and target languages exists (pivot translation)2

(de Gispert and Mariño, 2006).
For example, since there is almost no large Japanese-French bilingual corpus(e.g. of more than 1M sentence pairs), it is diﬃcult to directly train MT models.
However, there are many large-scale and well-maintained corpora available forJapanese-English and English-French respectively. In many cases, each of thewidely spoken languages other than English, like Japanese and French in thisexample, has a suﬃciently large bilingual corpus with English. Therefore MT
between these languages via English as a pivot is possible and becomes a realistic
solution.
Although various methods using pivot languages have been proposed (Cohnand Lapata, 2007; Utiyama and Isahara, 2007), ambiguity due to expressions in
the pivot language often causes incorrect selection of translation rules and harms
translation quality. Therefore, pivot-side disambiguation is a key issue in pivot
translation.
Eﬃcient construction of bilingual corpus: While in specific cases large corpora can be collected, for example by crawling the web (Resnik and Smith, 2003),
in many domains or language pairs it is still necessarily to create data by hand, either by hiring professionals or crowdsourcing (Zaidan and Callison-Burch, 2011).
In theses cases, active learning, which selects which data to annotate based on
their potential benefit to the translation system, has been shown to be eﬀective
to improving SMT systems while keeping the required amount of annotation toa minimum (Ananthakrishnan et al., 2010a; Eck et al., 2005; González-Rubio
et al., 2012; Green et al., 2014; Haﬀari and Sarkar, 2009; Haﬀari et al., 2009;
Turchi et al., 2008).
Most work on active learning for SMT assigns priority to sentences or subsentences that contain data that is potentially useful to the MT system according to a number of criteria. Sub-sentential annotation methods can remove many
redundant segments duplicated in sentences to be selected in full-sentential annotation and reduce annotation workload theoretically. However, selected segments
are not always easy to translate for humans, causing actual results to be far fromthe optimal. For example, fragments of complex phrases that have only incomplete syntactic information may harm the annotation quality. Therefore, it will3

be necessary to consider methods that are truly eﬃcient for human annotators.

1.4 Thesis Scope
The research purpose of this thesis is to improve multilingual SMT, specificallybetween low-resource language pairs, resolving the known problems of previously
proposed methods in pivot translation and active learning for SMT. This thesis
addresses the problems to disambiguate in pivot translation through pivot-side
contextual and syntactic information, and to provide more eﬃcient and humanfriendly active learning method emphasizing new criteria of non-redundancy andsyntactic coherence.

1.4.1 Better Pivot Translation by Remembering the Pivot
Among various methods using pivot languages, the triangulation method (Cohnand Lapata, 2007; Utiyama and Isahara, 2007), which translates by combining
source-pivot and pivot-target translation models into a source-target model, hasbeen shown to be one of the most eﬀective approaches. However, word sense
ambiguity and interlingual diﬀerences of word usage cause diﬃculty in accurately
learning correspondences between source and target phrases.
Figure 1.1 (a) shows an example of three words in German and Italian thateach correspond to the English polysemic word “approach.” In such a case, finding associated source-target phrase pairs and estimating translation probabilities
properly becomes a complicated problem. Furthermore, in the conventional triangulation method, information about pivot phrases that behave as bridges between
source and target phrases is lost after learning phrase pairs, as shown in Figure
1.1 (b).
To overcome these problems, we propose a novel triangulation method that remembers the pivot phrase connecting source and target in the records of phrase/rule
table, and estimates a joint translation probability from the source to target andpivot simultaneously. We show an example in Figure 1.1 (c). The advantageof this approach is that generally we can obtain rich monolingual resources inpivot languages such as English, and SMT can utilize this pivot-side contextual
information to improve the translation quality.

4Annäherung	

approach	approccio	

Ansatz	approximation	accesso	Zufahrt	entrance	ravvicinamento	

(a) Triangulation (De-En-It)
Annäherung(via: approach)

Annäherung(via: approach)approccio
accessoravvicinamento(via: approach, approximation)
Ansatzapproccio(via: approach)

Annäherung・・・(b) Traditional Triangulated Phrases
Annäherung〈approccio, approach〉

Annäherung〈ravvicinamento, approach〉

Annäherung〈ravvicinamento, approximation〉

Ansatz・・・〈approccio, approach〉(c) Proposed Triangulated Phrases
Figure 1.1: An example of (a) triangulation and the resulting phrases in the (b)traditional method of forgetting pivots and (c) our proposed methodof remembering pivots.
To utilize information about the pivot language at translation time, we train
a Multi-Synchronous Context-free Grammar (MSCFG) (Neubig et al., 2015), a
generalized extension of Synchronous CFGs (SCFGs) (Chiang, 2007), that cangenerate strings in multiple languages at the same time. To create the MSCFG,
we triangulate source-pivot and pivot-target SCFG rule tables not into a single
source-target SCFG, but into a source-target-pivot MSCFG rule table that remembers the pivot. During decoding, we use language models (LMs) over boththe target and the pivot to assess the naturalness of the derivation.

5

[X1] 记录 [X2]

[X1] enregistrer [X2]
[X1] record [X2]

[X2] [X1] 记录[X1] dossier [X2](a) Standard triangulation method matching phrases
VP
VP[X1] enregistrer [X2]TO

[X1] 记录 [X2]VB

NP

record [X2]

[X1]

NP
NP

[X1] dossier [X2]DT
[X1]

[X2] [X1] 记录NNNPrecord [X2](b) Proposed triangulation method matching subtrees

Figure 1.2: Example of disambiguation by parse subtree matching (Fr-En-Zh),[X1] and [X2] are non-terminals for sub-phrases.

1.4.2 Syntactic Matching Methods in Pivot Translation
In the triangulation method, source-pivot and pivot-target phrase pairs are connected as a source-target phrase pair when a common pivot-side phrase exists.
In Figure 1.2 (a), we show an example of standard triangulation on Hiero (Chiang, 2007) translation models that combines hierarchical rules of phrase pairs bymatching pivot phrases with equivalent surface forms. This example also demonstrates problems of ambiguity: the English word “record” can correspond toseveral diﬀerent parts-of-speech according to the context. More broadly, phrases
including this word also have diﬀerent possible grammatical structures, but it
is impossible to uniquely identify this structure unless information about thesurrounding context is given.
This varying syntactic structure will aﬀect translation. For example, the Frenchverb “enregistrer” corresponds to the English verb “record”, but the French noun
“dossier” also corresponds to “record” — as a noun. As a more extreme example,
Chinese is a languages that does not have inflections according to the part-ofspeech of the word. As a result, even in the contexts where “record” is used6

with diﬀerent parts-of-speech, the Chinese word “记录” will be used, although
the word order will change. These facts might result in an incorrect connection of“[X1] enregistrer [X2]” and “[X2] [X1] 记录” even though proper correspondenceof “[X1] enregistrer [X2]” and “[X1] dossier [X2]” would be “[X1] 记录 [X2]” and“[X2] [X1] 记录”. Hence a superficial phrase matching method based solely on
the surface form of the pivot will often combine incorrect phrase pairs, causingtranslation errors if their translation scores are estimated to be higher than theproper correspondences.
Given this background, we hypothesize that disambiguation of these cases
would be easier if the necessary syntactic information such as phrase structures
are considered during pivoting. To incorporate this intuition into our models,
we propose a method that considers syntactic information of the pivot phrase,as shown in Figure 1.2 (b). In this way, the model will distinguish translation
rules extracted in contexts in which the English symbol string “[X1] record [X2]”behaves as a verbal phrase, from contexts in which the same string acts as nounphrase.

1.4.3 Syntactic and Non-Redundant Segment Selection
for Active Learning
Most work on active learning for SMT, and natural language tasks in general,
has focused on choosing which sentences to give to annotators. These methods
generally assign priority to sentences that contain data that is potentially useful to
the MT system according to a number of criteria. For example, there are methods
to select sentences that contain phrases that are frequent in monolingual data but
not in bilingual data (Eck et al., 2005), have low confidence according to the MT
system (Haﬀari et al., 2009), or are predicted to be poor translations by an MT
quality estimation system (Ananthakrishnan et al., 2010a). However, while theselected sentences may contain useful phrases, they will also generally containmany already covered phrases that nonetheless cost time and money to translate.
To solve the problem of wastefulness in full-sentence annotation for active learning, there have been a number of methods proposed to perform sub-sentential
annotation of short phrases for natural language tasks (Bloodgood and Callison-7

any one of the preceding claims(a) Conventional n-gram selection method (n = 4)any  one  of  the  preceding  claims(b) Proposed maximal phrase selection method
NP
PP
NP
DT CD IN

NP
DT

JJ

NNS

any one of the preceding claims(c) Proposed parse subtree selection method

Figure 1.3: Conventional and proposed data selection methods
Burch, 2010; Settles and Craven, 2008; Sperber et al., 2014; Tomanek and Hahn,
2009). For MT in particular, Bloodgood and Callison-Burch (2010) have proposeda method that selects poorly covered n-grams to show to translators, allowing
them to focus directly on poorly covered parts without including unnecessary
words. Nevertheless, our experiments identified two major practical problems
with this method. First, as shown in Figure 1.3 (a), many of the selected phrases
overlap with each other, causing translation of redundant phrases, damaging eﬃciency. Second, it is common to see fragments of complex phrases such as “one ofthe preceding,” which may be diﬃcult for workers to translate into a contiguous
phrase in the target language.
In this work, we propose two methods that aim to solve these two problems andimprove the eﬃciency and reliability of segment-based active learning for SMT.
For the problem of overlapping phrases, we note that by merging overlappingphrases, as shown in Figure 1.3 (b), we can reduce the number of redundant
words annotated and improve training eﬃciency. We adopt the idea of maximal
substrings (Okanohara and Tsujii, 2009; Yamamoto and Church, 2001) which

8both encode this idea of redundancy, and can be calculated to arbitrary length
in linear time using enhanced suﬃx arrays. For the problem of phrase structure
fragmentation, we propose a simple heuristic to count only well-formed syntactic
constituents in a parse tree, as shown in Figure 1.3 (c).

1.5 Document StructureThis thesis contains the following chapters:• In Chapter 2, the four representative frameworks of MT in historical order
are introduced. Then, the detailed mechanism of SMT is explained, and alsoa basic summary of MT framework based on Neural Networks is provided.
• In Chapter 3, conventional methods in pivot translation and active learning for SMT are described. Specifically their own problems and potential
methods to solve them are discussed.
• In Chapter 4, a proposed method in pivot translation to resolve pivot-side
contextual ambiguity is presented. This proposed method lets MT models
remember the information of the pivot phrase. This information can help
to select appropriate translation rules considering pivot-side context withpivot language models. This chapter is based on an ACL’2015 paper (Miuraet al., 2015).
• In Chapter 5, a proposed method in pivot translation to resolve the pivotside syntactic ambiguity is presented. This proposed method introduces an
explicitly syntax-aware matching condition to find correct correspondenceof source-pivot and pivot-target translation rules, and can produce morereliable models. This chapter is based on a WMT’2017 paper (Miura et al.,2017).
• In Chapter 6, a proposed method in active learning for SMT to introduce
new criteria of segment selection, based on non-redundancy and syntactic coherence. This proposed method provides more compact and humanfriendly annotation task than conventional methods, resulting in a higher

9quality parallel corpus with lower annotation cost. This chapter is basedon an NAACL’2016 paper (Miura et al., 2016).
• In Chapter 7, summary and contributions of this thesis are described, anddirections for future work are discussed.

10

2 Machine Translation
Frameworks
Machine Translation (MT) is a computer-aided technology converting sentencesof a certain language into sentences of diﬀerent languages. To realize MT, various frameworks have been proposed in history, and as representative there are
Rule-Based Machine Translation (RBMT (Nirenburg, 1989)1 ), Example-Based
Machine Translation (EBMT (Nagao, 1984))2 Statistical Machine Translation(SMT (Brown et al., 1993)) and Neural Machine Translation (NMT (Bahdanau
et al., 2015; Sutskever et al., 2014)).
Since translation rules are manually described in RBMT, it has an advantagethat translation results can be controlled based on linguistically motivated grammar rules. On the other hand, knowledge of experts familiar with both of source
and target languages is required for each language pair, it is necessary to describe a huge and complicated variety of rules, and it is diﬃcult to cover a varietyof language expressions. In the other three frameworks, there are many common points, such as automatically acquiring translation rules based on bilingual
corpora.
In EBMT, a target language sentence is generated by combining examples
having a high degree of similarity with respect to an input sentence based on
the example database of parallel sentences. This framework is known to exhibit
extremely high performance when the bilingual corpus used for examples andthe field of the sentence to be translated conform to each other. However, it isnecessary to consider complicated combination problems, statistical models are
not assumed for scoring at the time of selecting examples, and thus lack versatility.
1
2

RBMT is referred also as “Knowledge-Based Machine Translation (KBMT).”
EBMT is referred also as “Machine Translation by Principal Analogy.”

11

SMT, which is discussed centrally in this thesis, automatically acquires correspondence relationships of words and phrases from bilingual corpora as translation rules, gives probabilistic scores to each rule. Then, for each input sentence, it
searches for an output sentence that maximizes the translation probability score
for given input sentence. Depending on the translation rules to be used, there is amore detailed classification of the SMT frameworks, and these are introduced in
Sections 2.1.3-2.1.7. Generally the versatility of frameworks based on statisticalmodels is high, and generalized log-linear models can be used for eﬃcient searchand optimization (Och, 2003). Since it has become possible to operate large-scalecomputing resources and rich language resources in recent years, SMT researchand development is spurred and many of MT systems that adopt SMT.
Research on NMT, which implements MT using neural networks (NNs), also
tends to increase in recent years, and various methods have been devised. In
some cases, NNs are used to strengthen SMT models, whereas there are manymethods to train translation models directly from parallel corpora. It has many
attractive advantages that are diﬃcult with ordinary SMT, such as training oflong distance dependencies, joint optimization of multiple models, etc. However,problems remain in practical application, including the fact that advanced parallel computing environments are indispensable, and interpretation and control of
trained models is diﬃcult.
Although each framework has advantages and disadvantages, this thesis focuses on SMT, and only some experiments have comparison and verification with
NMT. As the reason for using SMT in this study, specifically since automaticallyacquired translation rules are scored based on statistical model, it is possible toexplicitly give meanings when synthesizing models, and therefore, SMT is moreeasily integrated with pivot translation and active learning. It has also reported
that NMT can not exert much performance and is inferior in accuracy of SMT
when bilingual corpus used for training is not suﬃcient. Therefore, SMT should
be more suitable for low-resource scenarios.
The following sections describe about detailed mechanism of SMT (Section 2.1)
and explain briefly about the representative mechanisms of NMT (Section 2.2).

12

2.1 Statistical Machine Translation
The basic idea of SMT is based on a noisy channel model (Shannon, 1948). For
given sentence f in source language, let E(f ) be a set of all the possible translated
sentences in target language. Assume that P r(e|f ), translation probability fromf to e, can be computed for any sentence e in target language. SMT searches forthe ê ∈ E(f ) that maximizes P r(e|f ), having maximal translation probability
in target language.1ê = arg max P r(e|f )(2.1)

e∈E(f )

P r(f |e)P r(e)P r(f )e∈E(f )= arg max P r(f |e)P (e)= arg max

(2.2)
(2.3)

e∈E(f )

However, since it is diﬃcult to implement this actual model accurately, it iscommon to reformulate as a weight optimization problem based on the following
log-linear model (Och, 2003).
(2.4)

ê = arg max P r(e|f )e∈E(f )(

exp wT h(f , e≈ arg max ∑e∈E(f )e′)exp (wT h(f , e′ ))= arg max wT h(f , e)

(2.5)
(2.6)

e∈E(f )

where h is a feature vector, with feature values such as logarithmic probability
scores of translation models (Section 2.1.1) and language models (Section 2.1.2),
word reordering score accompanying derivation, and various penalties. The exact
number and type of features varies from framework to framework. w is a vectorof parameters, having the same dimension number with h, that weights eachelement of the feature vector. To adjust each element of w to be optimum, it
is necessary to use development data (referred also as tuning data) obtained by
1

Such a process that searches for the optimal translation candidate maximizing the translation
probability is referred as decoding.

13

separating a bilingual corpus from training data and testing data, and automatic
evaluation measure that computers the similarity score of translation result withthe reference sentence in target language. Then the optimizer finds parameterssuch that the evaluation score such as BLEU (Papineni et al., 2002) is maximized
for the input sentences (Och, 2003). The various SMT frameworks describedin Sections 2.1.3-2.1.7 are also based on this log-linear model, but use diﬀerent
features.

2.1.1 Translation Models
Translation model (TM) P r(f |e) is a statistical model for prescribing the likelihood of a translation, and trained from bilingual corpus. Translation models do
not directly associate e and f , and assume that f is generated from f throughsome steps, referred as derivation d. In statistical models, the derivation is treatedas latent variables, and Equation 2.3 is rewritten according to the following equations:ê = arg max P r(f |e)P r(e)e∈E(f )∑= arg maxe∈E(f )

P r(f , d|e)P r(e)(2.7)

d∈D(f ,e)where D(f , e) is the set of derivations given f and e.
The IBM model is known as a specific example of derivation, that associates
f and e by word correspondence between the language pair, and maximizes thetranslation probability (Brown et al., 1993). In IBM models, P r(f |e) is defined
as a statistical model that generates f from e on word by word through wordalignment a.
P r(f |e) =∑P r(f , a|e)

(2.8)

a

Training word alignment is regarded as a problem of finding the â that maximizes
the conditional probability of translation model.
â = arg max P r(f , a|e)
a

14

(2.9)

Word alignment is expressed as a set of position pairs (j, i) representing wordcorrespondence ⟨fj , ei ⟩ in sentence pair ⟨f , e⟩.
a = {· · · , (j, i), · · · }(2.10)Such word-oriented translation is valid for pairs of languages that are similar
enough to make sense only by replacing words. However, it is insuﬃcient since
there are many cases where words do not correspond one-to-one in reality and
it is diﬃcult to consider word order. Phrase-Based Machine Translation (PBMT(Koehn et al., 2003)) dramatically improved translation accuracy than wordbased SMT, using correspondence of consecutive word strings extracted fromthe trained word alignment. However, translation is diﬃcult for language pairs
with significantly diﬀerent word orders due to complicated reordering problems in
PBMT, then SMT frameworks based on tree structure has been also proposed todeal with such advanced reordering problems. Sections 2.1.3-2.1.7 describe theseframeworks in detail.

2.1.2 Language Models
Language model (LM) P r(e) is used to evaluate how natural and fluent theword sequence of a given sentence is in the target language. A good language
model accurately gives high probability to natural sentences and low probability
to unnatural sentences as well. By referring to this information in the translation process, SMT can select more natural sentences from translation candidates,
leading to more fluent translation results. This subsection describes the n-gram
language models widely used in SMT.
First, assume that the naturalness of target language e sentence can be computed by probability chain as follows:
P (eI1 ) =I+1∏

PM L (ei |e0i−1 )(2.11)

i=1

where I is the length of e and assume that e = eI1 = e1 · · · eI . e0 = ⟨s⟩ represents the start-of-sentence symbol and eI+1 = ⟨/s⟩ represents the end-of-sentence

15

symbol. Each conditional probability can be obtained using maximum likelihood
estimation as follows:
PM L (ei |ei−10 ) =ctrain (ei0 )ctrain (e0i−1 )(2.12)

where ctrain represents the occurrence count of the word string in the training
data.
However, this definition gives probability 0 to a sentence which does not appear in the training data, and can not determine the superiority or inferiority ofnaturalness evaluation for many translation candidates that are evaluated withscore 0. Therefore, to calculate the conditional probability of ei , it is better to
i−1
use the conditional probability that considers only the ei−n+1, immediately prei−1
ceding n − 1 words, not e0 , all the word string before ei . According to this idea,
Equation 2.11 is rewritten as following:
P (eIi ) ≈I+1∏

PM L (ei |ei−1
i−n+1 )

(2.13)

i=1

The n-gram language models trained in this manner, can give probability scores
to sentences that do not exist in training data as well. However, even in thisequation, there still remains the problem of estimating probability 0 for sentences containing n-gram that do not appears in training data. To solve thisproblem, there are smoothing methods that estimate the probability score byi−1
combining the conditional probabilities P (ei |ei−1
i−n+1 ) of n-gram and P (ei |ei−n+2 )of (n−1)-gram. Various methods have been proposed for smoothing, and the representative methods are linear interpolation and the Kneser-Ney method (Chenand Goodman, 1996).

2.1.3 Phrase-Based SMT
PBMT is the most representative method in SMT. The training process of PBMTtranslation models first trains the word alignment from bilingual corpus, extracts
phrase pairs based on the trained word alignment, and scores each phrase pair.

16

John

hit

a

ball

.

John
ジョン は

ジョン は ボール を 打 っ た 。

Figure 2.1: En-Ja word alignment

hit

a

ball

.

ボール を 打 っ た。

Figure 2.2: En-Ja phrase extraction

Figure 2.1 shows an example of English-Japanese word alignment obtained
from bilingual corpus by the method described in Section 2.1.1.2 Figure 2.2 shows
an example of phrase extraction by finding correspondence of phrases from theobtained word-alignment. As shown in the figure, the lengths of extracted phrases
are not uniquely determined, and multiple phrase pairs with diﬀerent lengths are
extracted according to the word alignment. However, the extracted phrase pairs
impose a constraint that there is no word alignment that traverses inside andoutside the two corresponding phrases, and the maximum phrase length also
should be limited. PBMT translation models are trained by counting occurrence
count of phrases and co-occurrence count of phrase pairs based on all the listed
phrase pairs extracted in this manner.
Unlike the word-based translation models, the phrase-based translation models use the extracted phrases as the basic units of translation, thereby eﬃciently
training translation rules of consecutive word strings such as idioms, achieving
higher quality of translation. Depending on how phrases are delimited, there
are multiple candidates for derivations that derivations that result in a given
sentence to be translated into the target language. Then the translation probability is finally estimated considering the probability scores and reordering scoresof the phrase pairs used in each step of derivation. PBMT follows the log-linear
model of the Equation 2.6 to search for the translation candidate with the highest
translation probability score, and the feature functions include both directions ofphrase translation probability, both directions of lexical translation probability,
word penalty, phrase penalty, and language model score.
PBMT is used in many research and practical systems because it can easily
train and perform high-speed translation just by preparing a bilingual corpus
2

In languages such as Japanese, Chinese, and Thai where words are not separated by spaces inordinary texts, first it is necessary to perform tokenization using word segmentation tools.

17between language pairs to be translated. However, since it is a method notconsidering the structure of sentences, it is diﬃcult to eﬀectively address the
problem of word reordering. Although it is possible to introduce advanced word
reordering models (Galley et al., 2004; Goto et al., 2013), reordering over long
distances is still diﬃcult and it is not easy to use in pivot translation.

2.1.4 Synchronous Context-Free Grammars
This section first covers Synchronous Context-Free Grammars (SCFGs), which
are widely used in machine translation, particularly hierarchical phrase-based
SMT (Hiero) (Chiang, 2007). In SCFGs, the elementary structures used in translation are synchronous rewrite rules with aligned pairs of source and target symbols on the right-hand side:X → ⟨s, t⟩(2.14)

where X is the head symbol of the rewrite rule, and s and t are both stringsof terminals and non-terminals on the source and target side respectively. Eachstring in the right side pair has the same number of indexed non-terminals, andidentically indexed non-terminals correspond to each-other. For example, a synchronous rule could take the form of:⟨⟩

X → X0 of X1 , X1 的 X0 .

(2.15)

Synchronous rules can be extracted based on parallel sentences and automatically obtained word alignments, similarly to PBMT. Each extracted rule is scoredwith phrase translation probabilities in both directions ϕ(s|t) and ϕ(t|s), lexicaltranslation probabilities in both directions ϕlex (s|t) and ϕlex (t|s), a word penalty
counting the terminals in t, and a constant phrase penalty of 1.
At translation time, the decoder searches for the target sentence that maximizes the derivation probability, which is defined as the sum of the scores of therules used in the derivation, and the log of the language model probability over
the target strings. When not considering a language model, it is possible to eﬃciently find the best translation for an input sentence using the CKY+ algorithm

18

(Chappelier et al., 1998). When using a language model, the expanded search
space is further reduced based on a limit on expanded edges, or total states per
span, through a procedure such as cube pruning (Chiang, 2007).

2.1.5 Hierarchical Rules
In this section, we specifically cover the rules used in Hiero. Hierarchical rules
are composed of initial head symbol S, and synchronous rules containing terminals and single kind of non-terminal X.3 Hierarchical rules are extracted using
the same phrase extraction procedure used in phrase-based translation (Koehn
et al., 2003) based on word alignments, followed by a step that performs recursive
extraction of hierarchical phrases (Chiang, 2007).
For example, hierarchical rules could take the form of:⟨

X → Oﬃcers, 主席团 成員⟩⟨(2.16)⟩

X → the Committee, 委员会(2.17)

X → X0 of X1 , X1 的 X0 .

(2.18)⟨⟩

From these rules, we can translate the input sentence by derivation:
S → ⟨X0 , X0 ⟩
⇒
⇒

⟨
⟨

X1 of X2 , X2 的 X1⟩

Oﬃcers of X2 , X2 主席团 成員⟩⇒ ⟨Oﬃcers of the Committee,⟩

委员会 的 主席团 成員 .
Hiero has an advantage that it is able to achieve relatively high word re-ordering
accuracy (compared to other symbolic SMT alternatives such as standard PBMT)
without language-dependent processing. On the other hand, since it does not use
syntactic information and tries to extract all possible combinations of rules, it
has the tendency to extract very large translation rule tables and also tends tobe less syntactically faithful in its derivations.
3

It is also standard to include glue rules S → ⟨X0 , X0 ⟩, S → ⟨S0 X1 , S0 X1 ⟩, S →⟨S0 X1 , X1 S0 ⟩ to fall back on when standard rules can not result in a proper derivation.

19

2.1.6 Explicitly Syntactic Rules
An alternative to Hiero rules is the use of synchronous context-free grammaror synchronous tree-substitution grammar (Graehl and Knight, 2004) rules that
explicitly take into account the syntax of the source side (tree-to-string rules),target side (string-to-tree rules), or both (tree-to-tree rules). Taking the exampleof tree-to-string (T2S) rules, these use parse trees on the source language side,and the head symbols of the synchronous rules are not limited to S or X, butinstead use non-terminal symbols corresponding to the phrase structure tags ofa given parse tree. For example, T2S rules could take the form of:⟨⟩

XNP → (NP (NNS Oﬃcers)), 主席团 成員 ,⟨(2.19)⟩

XNP → (NP (DT the) (NNP Committee)), 委员会 ,(2.20)

XPP → (PP (IN of) XNP,0 ), X0 的 ,(2.21)

XNP → (NP XNP,0 XPP,1 ), X1 X0 .

(2.22)⟨⟩⟨⟩

Here, parse subtrees of the source language rules are given in the form of Sexpressions. From these rules, we can translate from the parse tree of the input
sentence by derivation:
XROOT →
⇒
⇒
∗

⇒

⟨
⟨
⟨

XNP,0 , X0⟩

(NP XNP,1 XPP,2 ), X2 X1⟩⟩(NP (NP (NNS Oﬃcers) XPP,2 )), X2 主席团 成員⟨(NP(NP (NNS Oﬃcers))(PP (IN of), 委员会 的 主席团 成員(NP (DT the)
(NNP Committee))))⟩.

In this way, it is possible in T2S translation to obtain a result conforming to thesource language’s grammar. This method also has the advantage the number ofless-useful synchronous rules extracted by syntax-agnostic methods such as Hieroare reduced, making it possible to train more compact rule tables and allowing
for faster translation.

20

2.1.7 Multi-Synchronous Context-Free Grammars
Multi-Synchronous Context-Free Grammars (MSCFGs (Neubig et al., 2015)) area generalization of SCFGs that are be able to generate sentences in multiple target
languages simultaneously. The single target side string t in the SCFG production
rule is extended to have strings for N target languages:
X → ⟨s, t1 , ..., tN ⟩ .

(2.23)

Performing multi-target translation with MSCFGs is quite similar to translating using standard SCFGs, with the exception of the expanded state space causedby having one language model for each target. Neubig et al. (2015) has proposeda sequential search method, that ensures diversity in the primary target search
space by first expanding with only primary target language model, then additionally expands the states for other language models, a strategy we also adopt
in this work.
In the standard training method for MSCFGs, the multi-target rewrite rules
are extracted from multilingual sentence-aligned corpora by applying an extended
version of the standard SCFG rule extraction method, and scored with featuresthat consider the multiple targets. It should be noted that this training method
requires a large amount of sentence-aligned training data including the source
and all target languages. This assumption breaks down when we have little
parallel data, and thereby we propose a method to generate MSCFG rules by
triangulating 2 SCFG rule tables in Chapter 4.

2.2 Neural Machine Translation
As shown previously, the training of SMT is carried out through various steps, and
it is necessary to prepare independent translation knowledge such as phrase/rule
table and language model, and design features to be considered at translation.
On the other hand, in NMT, training and translation can all be done in the same
framework by preparing only a single NN. Just by providing bilingual sentencepairs, NNs automatically learn some information necessary for translation. Such a21

input

ジョン は

ボール を

打

っ

た

。 


embeddingencoder

encoder RNN

attention

decoder RNN
decodersoftmax
output

John

hit

a

ball

.




Figure 2.3: Attention-based neural MT
framework in which input to output is completed in a single model is called end-toend. In this section, we introduce the commonly used elements in NMT: encoderdecoder model (Sutskever et al., 2014) and attention mechanism (Bahdanau et al.,2015; Luong et al., 2015).
NMT is largely divided into three parts. The first is an encoder that encodes
an input sentence into a vector representation of continuous values. The second is
an attention mechanism that controls where in the encoded input sentence should
be noticed when determining the word to be output. The third is a decoder that
decodes (generates) the output sentence based on the encoded input sentence andthe attention information. A schematic diagram of NMT composed of these threeparts is shown in Figure 2.3. In general, a model consisting of an encoder that
encodes input information and a decoder that decodes the encoded informationand obtains a desired output is called an encoder-decoder model. This model
is applied in various tasks such as document summarization and image caption
generation. Each element will be explained in the following subsections.

2.2.1 Encoder
The encoder first coverts each language into a vector of continuous values consisting hundreds of dimensions called distributed representation. This operation
is called embedding (Mikolov et al., 2013). Although such distributed represen-22tation can be trained from a large-scale monolingual corpus independent from
NMT, training of distributed representation is also commonly trained simultaneously as part of the network in NMT.
In embedding, the information of each word is represented independently ofeach other. Therefore, words whose meanings are determined by association withother words, such as function words and compound nouns, can not be represented
well. To deal with this problem, the recurrent NN (RNN) layer reads each word
one by one, and creates a vector representation that considers words before orafter.
A basic NN only transmit information in one direction and do not have thefunction of using past information or storing information, though a RNN has amechanism to feed back its own output as its input again. The hidden states ofthe forward RNN are computed as:
(

⇀

)

⇀

hi = f hi−1 , s(2.24)⇀

where f is the RNN computation, hi is the hidden states of the forward RNN intime step t, and s is the source sentence representation. This mechanism makes it
possible to use past information and current information at the same time. Since
only the previous words can be considered in a forward RNN alone, we often use
it as as a bidirectional RNN in combination with a backward RNN that allows
for consideration of the following words:
↼

(

↼

)

[

]

hi = f hi+1 , s ,⇀↼hi = concat hi ; hi .

(2.25)(2.26)

2.2.2 Attention Mechanism
The attention mechanism plays a role of determining the point to be noticed
when translating the next word with the encode input sentence and the internal
state of the decoder as the decision information. Attention weights for each hi ineach step t are computed as:

23

(score hi , ht−1

at (i) = ∑j score()

hj , ht−1)(2.27)

where the attentional weight at (i) is determined by a score function, which receives one encoder state hi and the previous decoder state ht as input. Everytime one word is translated, these attention weights are recalculated, and thepoints to be noticed are changed every time.
For the decoder part, a context vector ct , which is a weighted summarizationof encoder states is calculated as:
ct =∑at (i)hi .

(2.28)i2.2.3 Decoder
The decoder is generally composed of a single RNN, receives the context vector
and the information of the word outputted one before, and outputs the next
word. For this reason, translated sentences are generated one word at a time inorder from the beginning. The output from RNN is a vector having the same
number of dimensions as the number of unique words in the target language. The
computation of each hidden state ht of the decoder RNN is shown as follows:
ht = f (ht−1 , ct , yt−1 ) ,(2.29)p(yj |y<j , s) = sof tmax (Wo ht + bo )(2.30)

where yt−1 is the embedding of previous generated word, and directly drawn fromthe target translation. The softmax function normalizes the output of RNN layer
like a probability distribution, and outputs the word with the highest probability.
This is the outline of the mechanism of NMT.

24

3 Low-Resource MachineTranslation
In Chapter 2, we mentioned that SMT can obtain translation rules automaticallyfrom a bilingual corpus, and searches for the translation candidate that maximizes
the translation probability score. Since it is based on statistical models, thereliability of the probability estimation improves as the target-side monolingual
corpus used for training the language model and the bilingual corpus used fortraining the translation model become large, and higher translation accuracy canbe expected. Although there are influences such as the number of speakers orinternet users in the target language, the language model is unlikely to be a
problem because it is relatively easy to acquire training data. On the other hand,the bilingual corpus is the key issue to SMT, and it is impossible to translatewords or expressions not covered by the training data. Therefore it is desirable toacquire as larger a bilingual corpus as possible. It has been reported that building
a practical SMT system requires more than million sentence pairs (Dyer et al.,2008; Koehn et al., 2007). However, considering language pairs not including
English, such as Japanese and French, while it is possible to acquire abundantmonolingual corpora in each language, it is diﬃcult to acquire a bilingual corpus
of more than 1M sentence pairs. Thus, the bilingual corpus, which is a major
premise of SMT, can not be immediately acquired in a suﬃcient size in many
language pairs, and there is a problem in performing translation between an
arbitrary language pair.
This chapter describes the two representative approaches for coping with thescarceness of bilingual corpus: pivot translation (Section 3.1) and active learning
for SMT (Section 3.2).

25

3.1 Pivot Translation
Several methods have been proposed for SMT using pivot languages. These arecategorized into 3 categories: sequential pivot translation (Section 3.1.1), pseudoparallel corpus synthesis (Section 3.1.2), and triangulation of translation models(Section 3.1.3), these are covered in following sections.

3.1.1 Sequential Pivot Translation
Parallelsource-pivot

sourceSMT
source → pivot

Parallelpivot-target

pivot

SMT
pivot → target

targetFigure 3.1: Sequential pivot translation
Figure 3.1 shows a diagram of translation from source sentence to target sentence by sequential pivot translation method (referred as cascade in experiments)(de Gispert and Mariño, 2006). This method first trains source-pivot and pivottarget MT models using respective bilingual corpora. Then, it is possible totranslate from a source sentence into the target language by translating the source
sentence into the pivot language and translating again the pivot sentence into thetarget language. Since this method pipelines only input and output of MT systems, it is not necessary in use only SMT and any MT systems can be combined.
The advantage of this method is its ease of implementation just reusing available
systems, and accurate pivot translation can be expected if two accurate MT systems are given. However, there are disadvantages that additive error propagatedfrom two systems tends to damage translation accuracy, and optimization of theentire connected system is complicated.
In naïve implementation of this method, a source-pivot system may translate
into only a single pivot sentence with the highest translation probability, thoughmulti-sentential implementation has been proposed (Utiyama and Isahara, 2007)
to expand the search space by leaving top n candidates with the highest translation probability as shown in Figure 3.2. However, n times more decoding time26

parallelsource-pivot

pivot 1

SMT
source → pivot

sourcepivot 2・・・

parallelpivot-target

SMT
pivot → target

targetpivot nFigure 3.2: Multi-sentential method
is required than usual, and no significant improvement in accuracy has been reported (Utiyama and Isahara, 2007).

3.1.2 Pseudo-Parallel Corpus Synthesis
parallelpivot-target

parallelsource-pivot

SMT
pivot → target

pseudo-parallel
source-target

SMT
source → target

sourcetargetFigure 3.3: Pseudo-parallel corpus synthesis
Figure 3.3 shows a diagram of translation with SMT trained with source-target
pseudo-parallel corpus by pseudo-parallel corpus synthesis method (referred assynthetic in experiments) (de Gispert and Mariño, 2006). This method first uses
only one of source-pivot and pivot-target parallel corpora to train SMT system, forexample, pivot-target is used in this figure. Then, source-target pseudo-parallel
corpus is obtained by translating all the pivot-side sentences in source-pivot parallel corpus into target language. Thus, it is possible to train translation models
using the obtained source-target pseudo-parallel corpus. Accurate translation
can be expected if translation errors in the pseudo-parallel corpus do not significantly aﬀect the training of statistical models. Since this method rebuilds new

27

training data from existing systems, it has the advantage that once it creates a
pseudo-parallel corpus, it can use the same training procedure as regular SMT.
de Gispert and Mariño (2006) have performed experiments on pivot translationof Catalan and English using Spanish as a pivot language, comparing sequential
pivot translation method and pseudo-parallel corpus synthesis. As a result, nosignificant diﬀerence between these methods has been reported.

3.1.3 Triangulation of Translation Models
parallelsource-pivot

parallelpivot-target

SMT
source → pivot

SMT
pivot → target

sourceSMT
source → target

targetFigure 3.4: Triangulation of translation models
The training procedures of PBMT and SCFGs store the phrase pairs extracted
and scored from a bilingual corpus into a structured file called the phrase/rule
table. Figure 3.4 shows a diagram of triangulation of source-pivot and pivot-target
phrase/rule tables into a source-target table (Cohn and Lapata, 2007; Utiyama
and Isahara, 2007).
The triangulation method for PBMT first trains source-pivot and pivot-target
phrase tables as TSP and TP T respectively. Then this method triangulates TSP
and TP T by matching source-pivot and pivot-target phrase pairs having a common pivot phrase, and synthesizes them into source-target phrase pairs to create source-target phrase table TST . For all the combined source-target phrases,
phrase translation probability ϕ(·) and lexical translation probability ϕlex (·) are
estimated according to the following equations:28

ϕ (t|s) =∑≈ϕ (t|p, s) ϕ (p|s)

p∈TSP ∩TP T

ϕ (s|t) =∑∑≈ϕ (s|p, t) ϕ (p|t)

∑

(3.1)

∑

ϕ (s|p) ϕ (p|t) ,(3.2)

p∈TSP ∩TP T

ϕlex (t|p, s) ϕlex (p|s) ≈p∈TSP ∩TP T

ϕlex (s|t) =ϕ (t|p) ϕ (p|s) ,p∈TSP ∩TP T

p∈TSP ∩TP T

ϕlex (t|s) =∑∑ϕlex (t|p) ϕlex (p|s) , (3.3)

p∈TSP ∩TP T

ϕlex (s|p, t) ϕlex (p|t) ≈p∈TSP ∩TP T∑

ϕlex (s|p) ϕlex (p|t) . (3.4)

p∈TSP ∩TP T

where s, p and t are phrases in source, pivot and target respectively, and p ∈TSP ∩ TP T indicates that p is commonly contained in phrase tables TSP and TP T .
The triangulation method for SCFGs (Miura et al., 2015) can be done following the same idea for PBMT, using pre-trained source-pivot and pivot-target rule
tables TSP and TP T respectively. Then this method searches TSP and TP T forsource-pivot and pivot-target rules having a common pivot symbols, and synthesize them into source-target rules to create rule table TST :X → ⟨s, t⟩ ∈ TST
s.t. X → ⟨s, p⟩ ∈ TSP ∧ X → ⟨p, t⟩ ∈ TP T .

(3.5)

Although SCFG triangulation is diﬀer from PBMT triangulation in that s, p and t
are symbol strings which may contain terminals and non-terminals, its procedureof estimating translation probability scores is same with Equations 3.1-3.4. Word
penalty and phrase penalty X → ⟨s, t⟩ are set as the same values of X → ⟨p, t⟩.
Utiyama and Isahara (2007) have performed experiments on pivot translation ofseveral language pairs with English as a pivot language, comparing the sequential
pivot translation method and the triangulation method. As a result, it has beenreported that the triangulation method achieved higher BLEU score than simple
sequential pivot translation with n = 1 and multi-sentential method with n = 15.

3.1.4 Problems of Pivot-Side Ambiguity
Although triangulation is known for achieving higher translation accuracy thanother simple methods and has become a popular and standard work of pivot trans-29lation nowadays, there still remains the problem of ambiguity. This subsection
describes reason causing the problem and examples.
In triangulation, Equations (3.1)-(3.4) are based on the memoryless channel
model, which assumes:ϕ (t|p, s) = ϕ (t|p) ,(3.6)

ϕ (s|p, t) = ϕ (s|p) .

(3.7)

For example, in Equation (3.6), it is assumed that the translation probabilityof target phrase given pivot and source phrases is never aﬀected by the source
phrase. However, it is easy to come up with examples where this assumption does
not hold.
Annäherung	

approach	approccio	

Ansatz	approximation	accesso	Zufahrt	entrance	ravvicinamento	

Figure 3.5: An example of ambiguity in De-En-It triangulation.
Figure 3.5 shows an example of three words in German and Italian each ofwhich corresponds to the English polysemic word “approach.” In such a case,finding associated source-target phrase pairs and estimating translation probabilities properly become complicated problems. As a result, pivot translation issignificantly more ambiguous than standard translation.
This thesis distinguishes the ambiguity problem in pivot translation into two
types: semantic ambiguity and syntactic ambiguity, and proposes approaches toresolve each of them.

3.1.5 Related Work
Up to this point, we have explained the representative pivot translation methods
in SMT. Other related research in pivot translation is mainly based on the triangulation for PBMT, and focuses on discussion to further improve accuracy (Dabreet al., 2015; Levinboim and Chiang, 2015; Zhu et al., 2014). In the triangulation,
it is a problem how to correctly estimate the translation probability.

30

Zhu et al. (2014) have proposed an estimation method of source-target translation probability by first estimating source-target co-occurrence counts instead ofthe direct estimation from source-pivot and pivot-target translation probabilities(Equations 3.1-3.4). They have reported that stable translation accuracy can beobtained even in triangulation of two phrase tables with unbalanced table size.
Levinboim and Chiang (2015) have reported that it is especially diﬃcult toestimate word-level translation probability among phrase correspondence in thetriangulation stage. Then they have proposed a method of improving the qualityof the triangulation by estimating translation probability even for the correspondence of words which can not be directly observed, using distributed expressionof words (Mikolov et al., 2013).
In this thesis, we discuss pivot translation with English as a pivot language,
though it is also known that the translation accuracy varies depending on how
to select a pivot language. The influence of pivot language selection on pivot
translation has been discussed in detail by Paul et al. (2009). In reality, there are
not many situations in which the pivot language can be selected from multiple
candidates, though in the case where bilingual corpora of the same scale can beobtained via several languages, we should choose a pivot language having similar
language structure with source or target language.
In addition, there is no need to limit the number of pivot languages necessarily to one, and methods to consider multiple pivot languages at the same time
have been also proposed. As representative for that purpose, there are methods
such as aggregating multiple source-target phrase/rule tables obtained by triangulation with respective pivot languages into one table with linear interpolation,and searching by considering multiple translation models simultaneously (Dabreet al., 2015).
Here we mention the relationship between pivot language in pivot translationand interlingual language (interlingua (Vauquois, 1968)) in RBMT. An attempt
to MT using interlingua has a long history, and before the SMT was invented, thelevels of translation has been discussed from the beginning when RBMT has beendevised (Nirenburg, 1989). Figure 3.6 is called Vauquois pyramid and famous asa diagram showing the levels of analysis and generation in MT. It is expectedthat source language text is translated with less information loss by analyzing to31

Interlingua
Analysis

Generation

Semantic
Semantic
Semantic
Representation Transfer Representation
Syntactic
Structure

Words

Syntactic Transfer
Direct Translation

Source Language

Syntactic
Structure
Words
Target Language Text

Figure 3.6: Levels of translation (“Vauquois pyramid”)
higher level and transferring into target language. Interlingua, which is the top of
the pyramid, is an ideal language that can cover expressions of all the languages,
though such natural languages do not exist. In the past, there has been an
attempt to convert arbitrary text into an artificial interlingua with RBMT which
describes translation rules manually. However, with such rule-based approaches,
it is impossible to cover various domains and expressions with hands of experts,
and indeed, there is no human being who is familiar with every language.
With the invention of SMT, using bilingual corpora it became possible to translate highly accurately far more eﬃciently than rule description by experts. However, it is said that today’s MT systems are wandering around the bottom of
Vauquois pyramid. When an arbitrary natural language is used as a pivot language such as interlingua, some information might be lost depending on the expressiveness of the selected language, and then it may be impossible to reproduce
the information of the source language text in the process of generation. For thisreason, a form with higher expressiveness than an simple word sequence should
be used for pivot language, and as an example, MT frameworks using ontology
as intermediate representation have been proposed (Hovy, 1998). In addition,
training methods of multilingual NMT, which improve translation accuracy by
causing translation tasks of plural language pairs to be trained as a common encoder, have been also proposed (Dong et al., 2015; Johnson et al., 2017; Zoph andKnight, 2016). This can be interpreted as handling distributed representation oftranslation units well as an pivot language.

32

3.2 Active Learning
Active learning is a framework that makes it possible to eﬃciently train statistical models by selecting informative examples from a pool of unlabeled data.
Most work on active learning for SMT, and natural language tasks in general,
has focused on choosing which sentences to give to annotators. These methods
generally assign priority to sentences that contain data that is potentially usefulto the MT system according to a number of criteria.
For example, there are methods to select sentences that contain phrases thatare frequent in monolingual data but not in bilingual data (Eck et al., 2005), have
low confidence according to the MT system (Haﬀari et al., 2009), or are predicted
to be poor translations by an MT quality estimation system (Ananthakrishnan
et al., 2010a). However, while the selected sentences may contain useful phrases,
they will also generally contain many already covered phrases that nonethelesscost time and money to translate.
To solve the problem of wastefulness in full-sentence annotation for active learning, there have been a number of methods proposed to perform sub-sentential
annotation of short phrases for natural language tasks (Bloodgood and CallisonBurch, 2010; Settles and Craven, 2008; Sperber et al., 2014; Tomanek and Hahn,
2009). For MT in particular, Bloodgood and Callison-Burch (2010) have proposeda method that selects poorly covered n-grams to show to translators, allowing
them to focus directly on poorly covered parts without including unnecessary
words (Section 3.2.3).

3.2.1 Active Learning for Machine Translation
In this section, we first provide an outline of the active learning procedure toselect phrases for SMT data. We regard a segment as a word sequence with
arbitrary length, which indicates that full sentences and single words both qualify
as segments. In Algorithm 1, we show the general procedure of incrementally
selecting the next candidate for translation from the source language corpus,
requesting and collecting the translation in the target language, and retraining
the models.
In lines 1-4, we define the datasets and initialize them. SrcP ool is a set with33

Algorithm 1 Active learning for SMT
1: Init:2:
SrcP ool ← source language data including candidates for translation
3:
T ranslated ← translated parallel data
4:
Oracle ← oracle giving the correct translation for an input phrase
5: Loop Until StopCondition:
6:
T M ← T rainT ranslationM odel(T ranslated)
7:
N ewSrc ← SelectN extSegment(SrcP ool, T ranslated, T M )8:
N ewT rg ← GetT ranslation(Oracle, N ewSrc)∪
9:
T ranslated ← T ranslated{⟨N ewSrc, N ewT rg⟩}
each sentence in source language corpus as an element. T ranslated indicates a
set with source and target language phrase pairs. T ranslated may be empty, but
in most cases it consists of a seed corpus upon which we would like to improve.
Oracle is an oracle (e.g. a human translator), that we can query for a correcttranslation for an arbitrary input phrase.
In lines 5-9, we train models incrementally. StopCondition in line 5 is anarbitrary timing when to stop the loop, such as when we reach an accuracy goalor when we expend our translation budget. In line 6, we train the translation
model using T ranslated, the available parallel data at this point. We evaluate
the accuracy after training the translation model for each step in the experiments.
In line 7, we select the next candidate for translation using features of SrcP ool,
T ranslated and T M to make the decision.
In the following sections, we discuss existing methods to implement the selection criterion in line 7.

3.2.2 Sentence Selection using N -Gram Frequency
The first traditional method that we cover is a sentence selection method. Specifically, it selects the sentence including the most frequent uncovered phrase witha length of up to n words in the source language data. This method enables us
to eﬀectively cover the most frequent n-gram phrases and improve accuracy withfewer sentences than random selection. Bloodgood and Callison-Burch (2010)
demonstrate results of a simulation showing that this method required less than34

80% of the data required by randomly selected sentences to obtain the sameaccuracy.
However, the selected full sentences include many phrases already covered in
the parallel data. This may cause an additional cost for words in redundantsegments, a problem resolved by the phrase selection approach detailed in thefollowing section.

3.2.3 Phrase Selection using N -Gram Frequency
In the second approach, we directly select and translate n-gram phrases that are
the most frequent in the source language data but not yet covered in the translated
data (Bloodgood and Callison-Burch, 2010). This method allows for improvementof coverage with fewer additional words than sentence selection, achieving higher
eﬃciency by reducing the amount of data unnecessarily annotated. Bloodgood
and Callison-Burch (2010) showed that by translating the phrases selected bythis method using a crowdsourcing website, it was possible to achieve a large
improvement of BLEU score, outperforming similar sentence-based methods.
any one of the preceding claims

Figure 3.7: An example of n-gram selection method (n = 4)
Nevertheless, our experiments identified two major practical problems with thismethod. First, as shown in Figure 3.7, many of the selected phrases overlap witheach other, causing translation of redundant phrases, damaging eﬃciency. Second,
it is common to see fragments of complex phrases such as “one of the preceding,”
which may be diﬃcult for workers to translate into a contiguous phrase in thetarget language.
In addition to the two major issues, this method limits the maximum phrase
length to n = 4, precluding the use of longer phrases. However, using a larger
limit such as n = 5 is not likely to be a fundamental solution, as it increases thenumber of potentially overlapping phrases, and also computational burden.

35

4 Contextual Disambiguation in
Pivot Translation
As mentioned in Section 3.1.3, triangulation method should estimate corresponding source-target phrase pairs given source-pivot and pivot-target phrase pairs
and their probability scores. Section 3.1.4 mentioned that pivot-side ambiguity
causes diﬃculty of triangulation. This chapter discusses the problem of semantic
ambiguity due to word sense ambiguity and interlingual diﬀerences, and a method
to solve this problem by incorporating pivot-side contextual information.

4.1 Word Sense Ambiguity
近似
アプローチ
接近

approachapproachapproccio

approximation

approximation

accessoaccessaccess(a) Ja-En phrase correspondence

ravvicinamento(b) En-It phrase correspondence

近似

approccio

近似

approccio

アプローチ

accesso

アプローチ

accesso接近接近ravvicinamento(c) All the Ja-It candidates

ravvicinamento(d) Correct Ja-It phrase correspondence

Figure 4.1: Estimation of source-target word correspondenceThis section illustrates the diﬃculty of triangulation due to polysemy. In Figure 4.1, it is shown that (a) Japanese-English and (b) English-Italic phrase cor-36

respondence are given in individually trained translation models, then (c) many
candidates to be triangulated resulting diﬃculty to accurately connect as (d) the
correct Japanese-Italic phrase correspondence.
近似
近似
近似

(via: approach)(via: approach)approccio
accessoravvicinamento(via: approach, approximation)
アプローチ
approccio(via: approach)・・・Figure 4.2: Standard triangulated phrases
Furthermore, in the conventional triangulation method, information aboutpivot phrases that behave as bridges between source and target phrases is lost
after learning phrase pairs, as shown in Figure 4.2.
近似

〈approccio, approach〉

近似〈ravvicinamento, approach〉

近似〈ravvicinamento, approximation〉

アプローチ〈approccio, approach〉
・・・

Figure 4.3: Proposed triangulated phrases
To overcome the problem, this thesis proposes a novel triangulation methodthat remembers the pivot phrase connecting source and target in the records ofphrase/rule table, and estimates a joint translation probability from the source
to target and pivot simultaneously. Figure 4.3 shows an example of triangulated
phrases in this proposed method. The advantage of this approach is that generally
we can obtain rich monolingual resources in pivot languages such as English, andSMT can utilize this additional information to improve the translation quality.

37

4.2 Triangulation Remembering the Pivot
To help reduce this ambiguity in standard triangulation method, our proposed
triangulation method remembers the corresponding pivot phrase as additional information to be utilized for disambiguation. Specifically, instead of marginalizing
over the pivot phrase p (Section 3.1.3), we create an MSCFG rule (Section 2.1.7)
for the tuple of the connected source-target-pivot phrases such as:
X → ⟨s, t, p⟩ ∈ TST P
s.t. X → ⟨s, p⟩ ∈ TSP ∧ X → ⟨p, t⟩ ∈ TP T

(4.1)

where TST P indicates the MSCFG rule table triangulated from SCFG rule tables
TSP and TP T . The advantage of translation with these rules is that they allow
for incorporation of additional features over the pivot sentence such as a strong
pivot language models.
In addition to the equations 3.1-3.4, we also estimate translation probabilities
ϕ(t, p|s), ϕ(s|p, t) that consider both target and pivot phrases at the same time
according to:ϕ (t, p|s) = ϕ (t|p) ϕ (p|s) ,(4.2)

ϕ (s|p, t) = ϕ (s|p) .

(4.3)

Translation probabilities between source and pivot phrases ϕ(p|s), ϕ(s|p), ϕlex (p|s),and ϕlex (s|p) can also be used directly from the source-pivot rule table. This
results in 13 features for each MSCFG rule: 10 translation probabilities ϕ (t|s)，ϕ (s|t)，ϕ (p|s)，ϕ (s|p)，ϕlex (t|s)，ϕlex (s|t)，ϕlex (p|s)，ϕlex (s|p)，ϕ (t, p|s)，ϕ (s|p, t),
2 word penalties counting the terminals in t and p, and a constant phrase penalty
of 1.
It should be noted that remembering the pivot results in significantly larger rule
tables. To save computational resources, several pruning methods are conceivable.
Neubig et al. (2015) show that an eﬀective pruning method in the case of a main
target T1 with the help of target T2 is the T1 -pruning method, namely, using L
candidates of t1 with the highest translation probability ϕ(t1 |s) and selecting t2
with highest ϕ(t1 , t2 |s) for each t1 . We follow this approach, using the L best t,
and the corresponding 1 best p .

38

4.3 Experiments
To investigate the eﬀect of the proposed approach, we perform experiments comparing pivot translation among multiple language pairs with the following procedure.

4.3.1 Experimental Set-Up
We evaluate the proposed triangulation method through pivot translation experiments on the United Nations Parallel Corpus (UN6Way (Ziemski et al., 2016))
corpus. UN6Way is a line-aligned multilingual parallel corpus that includes data
in English (En), Arabic (Ar), Spanish (Es), French (Fr), Russian (Ru) and Chinese (Zh), covering diﬀerent families of languages. It contains more than 11M
sentences for each language pair, and is therefore suitable for multilingual translation tasks such as pivot translation. In these experiments, we fixed English as thepivot language considering that it is the language most frequently used as a pivot
language. This has the positive side-eﬀect that accurate phrase structure parsers
are available in the pivot language, which is good for our proposed method. We
perform pivot translation on all the combinations of the other 5 languages, andcompared the accuracy of each method.
In pivot translation, source-pivot and pivot-target parallel corpus is respectively
used to train translation models. However, it is assumed that direct sentence tuples of source, pivot, and target are rarely found in scenes where pivot translationis required. Therefore, we ensure that there will be no common pivot sentence insource-pivot and pivot-target corpus used for training in this experiment.
In our basic training setup, we use 100k sentences for training both the TMs
and the target LMs. We assume that in many situations, a large amount ofEnglish monolingual data is readily available and therefore, we train pivot LMs
with diﬀerent data sizes up to 5M sentences. The archive of this corpus contains
already split data sets: about 11M sentences for training, 4,000 sentences respectively for evaluation and for parameter tuning. As preprocessing, the parallel
sentences including duplicated English sentence are removed to ensure uniqueness of pivot sentences, and the sentences of length over 60 words in training
data are filtered out to ensure accuracy of word alignment, and the sentences

39

over 80 words in evaluation and tuning data are filtered out as well. Then, there
remain about 8M sentences for training, and about 3,800 sentences respectively
for evaluation and tuning. Since the plain texts of Chinese are not divided forwords, they are tokenized with Chinese segmentation model of KyTea (Neubig
et al., 2011). However, since the number of combinations to be evaluated isenormous, we hold out much small data sets compared with preprocessed data
size: 100k sentences respectively for training source-pivot (referred as train1 ) andpivot-target (referred as train2 ) translation models, and 1,500 sentences respectively for evaluation and tuning, such that train1 and train2 do not have any
duplicated English sentences.
As a decoder, we use Travatar (Neubig, 2013), and train SCFG TMs with
its Hiero extraction code. All the translation tasks use 5-gram language models
trained from 200k target-side sentences of train1+train2 using KenLM (Heafield,2011) to evaluate naturalness during decoding. Translation results are evaluatedby BLEU (Papineni et al., 2002) and we tuned to maximize BLEU scores using
MERT (Och, 2003). For trained and triangulated TMs, we use T1 rule pruningwith a limit of 20 rules per source rule. For decoding using MSCFG, we adopt
the sequential search method.
We evaluate 3 pivot translation methods and 1 direct translation method:
Cascade:Sequential pivot translation with source-pivot and pivot-target SCFG models (Section 3.1.1). “w/ PvtLM 200k/5M” indicates translation with pivot
LM trained using 200k/5M sentences respectively.
Tri. SCFG:
Triangulating source-pivot and pivot-target SCFG TMs into a source-target
SCFG TM using the traditional method (Section 3.1.3).
Tri. MSCFG:
Triangulating source-pivot and pivot-target SCFG TMs into a source-targetpivot MSCFG TM in our approach. “w/o PvtLM” indicates translating
without a pivot LM and “w/ PvtLM 200k/5M” indicates a pivot LM trainedusing 200k/5M sentences respectively (proposed, Section 4.2).

40

BLEU Score [%]
Src

Ar

Es

Fr

Ru

Zh

Trg

CascadeDirect 1/2

w/ PvtLM

Tri. MSCFG

Tri. SCFG

200k

5M

(baseline)

w/o PvtLM

w/ PvtLM
200k

5M

Es

29.12 / 29.41

26.78‡ 28.52

27.84‡ 28.44‡ 28.48‡ 29.13Fr

23.68 / 22.19

20.76‡ 22.0221.37‡ 21.94‡ 22.01‡ 22.52Ru

17.28 / 16.82

14.71

15.91

16.22

16.38

† 16.61‡ 16.76

Zh

14.51 / 14.54

13.85‡ 15.01

14.38

14.43

‡ 14.93‡ 15.50Ar

14.46 / 14.17

13.87† 14.35

13.97

14.19

† 14.34‡ 14.42Fr

35.87 / 34.81

29.72

30.20

32.62

32.70

32.81

† 32.94Ru

21.58 / 22.18

19.55

20.52

20.91

20.92

20.95

‡ 21.52

Zh

17.56 / 18.10

17.54‡ 18.37

17.79

17.56

† 18.13‡ 18.70Ar

12.37 / 12.49

11.82

11.96

12.35

12.00

12.35

‡ 12.71Es

38.34 / 38.82

33.16

33.90

36.10

36.00

36.33

‡ 36.93Ru

20.10 / 20.86

18.33

18.95

19.51

† 19.91‡ 20.23‡ 20.44Zh

15.99 / 15.90

15.73

16.35

16.17

16.20

16.40

† 16.58Ar

12.29 / 12.13

11.47

11.72

11.75

11.84

11.91

‡ 12.18Es

31.01 / 31.39

28.67‡ 30.2329.60† 29.90‡ 30.59‡ 31.40Fr

25.74 / 25.71

23.64† 25.09

24.60

24.63

‡ 25.20‡ 25.32Zh

15.85 / 15.75

15.99† 16.68

16.12

15.75

‡ 16.66‡ 16.81Ar

10.25 / 10.13

9.85

9.79

9.90

9.98

10.04

10.07

Es

23.55 / 23.76

23.10† 23.98

23.41

23.62

† 23.83‡ 24.45

Fr

18.83 / 18.72

18.15

18.45

18.39

18.30

18.51

‡ 19.16Ru

14.96 / 15.44

13.92

13.65

14.00

14.24

‡ 14.41‡ 14.67

Table 4.1: Results of each method, comparing the proposed triangulation remembering the pivot with other methods.
Direct:Translating with a direct SCFG trained on the source-target parallel corpus(not using a pivot language) for comparison. “Direct 1/2” indicates theevaluation scores of SCFG models trained respectively with train1/train2.

4.3.2 Results and Analysis
The result of experiments using all combinations of pivot translation tasks via
English is shown in Table 4.1. Significant diﬀerence test is tested for the results
according to bootstrap resampling method (Koehn, 2004). Bold face indicates41

higher BLEU score than the baseline triangulation Tri. SCFG, underline indicatesthe highest BLEU score in pivot translation, and daggers indicate statisticallysignificant gains over Tri. SCFG († : p < 0.05, ‡ : p < 0.01). From the results,
we can see that the proposed triangulation method considering pivot language
models outperforms the traditional triangulation method for all language pairs,
and translation with larger pivot language models improves the BLEU scores.
For all languages, the pivot-remembering triangulation method with the pivot
language model trained with 5M sentences achieves the highest score of the pivot
translation methods, with gains of up to 1.8 BLEU points from the baseline
method. The average gain of BLEU score in Tri. MSCFG w/ PvtLM 5M is about
0.75 points. This shows that remembering the pivot and using it to disambiguate
results is consistently eﬀective in improving translation accuracy.
Also to investigate the eﬀects of diﬀerent factors separately, we compare triangulation method that creates MSCFG rule table but not uses pivot language
model for decoding (Tri. MSCFG w/o PvtLM). We can also see that the MSCFG
triangulated model without using the pivot language model slightly outperforms
the standard SCFG triangulation method for the majority of language pairs. It
is conceivable that the additional scores of translation probabilities with pivot
phrases are eﬀective features that allow for more accurate rule selection.
Since a method using a large-scale pivot language model is possible not only by
proposed triangulation into MSCFG rule table, but also by conventional method,
we evaluate the translation score of sequential pivot translation with pivot language model trained with 5M sentences (Cascade w/ PvtLM 5M). Cascade w/
PvtLM 5M achieves higher score in all language pairs except Zh-Ar and Zh-Ru,
compared with the sequential pivot translation method which uses pivot language
model of only 200k sentences, then it has been confirmed that using simply large
scale of language model lead to improvement. However, this accuracy improvement depends on language pairs, and Cascade w/ PvtLM 5M did not result instable improvement when compared with the conventional triangulation method
Tri. SCFG. In addition, when comparing the proposed method Tri. MSCFG w/
PvtLM 5M with Cascade w/ PvtLM 5M, the proposed method achieves higherscore in the tested language pairs in spite of using the same pivot language model,
and thus it can be said that the combination of triangulation and large-scale pivot

4225. 0 Translation Accuracy vs. Pivot-LM Size (Zh-Es via En)

17. 6 Translation Accuracy vs. Pivot-LM Size (Ar-Ru via En)

24. 8

17. 4

Direct 1
Direct 2

Tri. SCFG

Tri. MSCFG

Tri. SCFG

Tri. MSCFG

17. 2

BLEU Score [%]BLEU Score [%]24. 6

Direct 1
Direct 2

24. 4
24. 2
24. 0
23. 8

17. 0
16. 8
16. 6
16. 4

23. 6

16. 2

23. 4
0

1. 0 × 10 6

2. 0 × 10 6

3. 0 × 10 6

Pivot-LM Size [Sent.]

4. 0 × 10 6

5. 0 × 10 60

1. 0 × 10 6

2. 0 × 10 6

3. 0 × 10 6

Pivot-LM Size [Sent.]

4. 0 × 10 6

5. 0 × 10 6

Figure 4.4: Influence of pivot LM size on pivot translation accuracy
language model is eﬀective.

4.3.3 Influence of Pivot Language Model Strength
Although the magnitude of the influence of the pivot language model size on pivot
translation diﬀers on language pairs, it can also be confirmed that the accuracy
improves as the training data size of the pivot language model increases. Figure
4.4 shows the influence the intermediate language model trained with diﬀerent
data sizes on the accuracy of Zh-Es (left) and Ar-Ru (right) translation. It canalso be observed from the figure that the pivot language model helps to disambiguate and contributes to the improvement of translation accuracy. Although
it can also be seen that the influence on the translation accuracy by increasing
the training data size of pivot language model is logarithmic, this is the sametendency as the influence of the target language model size on the translation
accuracy (Brants et al., 2007). In the Zh-Es translation, there is a prospect of
accuracy improvement by using a larger pivot language model from the tendencyof the graph, whereas in the case of Ar-Ru, there is a limit to further improvement
since pivot language model of 5M sentences hardly improve over 2M sentences.

43

4.3.4 Qualitative Analysis
We show an example of a translated sentence for which pivot-side ambiguity
seems to be resolved in the proposed triangulation method.
Input (French):
Le nom du candidat proposé est indiqué dans l’annexa à la présente
note .
Reference (Spanish):
El nombre del candidato propuesto se presenta en el anexo de la presente
nota .
Corresponding English Sentence:
The name of the candidate thus nominated is set out in the annex to thepresent note .
Tri. SCFG:El nombre del proyecto de un candidato se indica en el anexo a la presente
nota . (BLEU+1: 34.99)
Tri. MSCFG w/ PvtLM 5M:
El nombre del candidato propuesto se indica en el anexo a la presente
nota . (BLEU+1: 61.13)
The name of the candidate proposed indicated in the annex to the present
note . (Generated English Sentence)In the example above, the French participle “prposé” (meaning “nominated”)
in the input sentence corresponds to the Spanish participle “propuesto” in translation. However, in the conventional triangulation method, the Spanish noun
“proyecto” (meaning “project” or “plan”), inappropriate correspondence, is selected in derivation, causing incorrect translation. On the other hand, it seems
that the translation result improved since “propuesto” in Spanish and “proposed”
in English are simultaneously associated with “proposé” in the input sentencewith the proposed method, prompting appropriate vocabulary selection from thecontext of the words in the generated English sentence.

44

Conversely, we show an example of translation result with no improvement,
since proposed method does not work well.
Input (French):
J . Risques d’aspiration : citère de viscosité pour la classification desmélanges ;Reference (Spanish):J . Peligros por aspiración : criterio de viscosidad para la clasificación demezclas ;Corresponding English Sentence:
J . Aspiration hazards : viscosity criterion for classification of
mixtures ;Direct 1:
J . Riesgos d’aspiration : criterio de viscosité para la clasificación de losmélanges ; (BLEU+1: 34.20)
Direct 2:
J . Riesgos d’aspiration : criterio de viscosité para la clasificación demezclas ; (BLEU+1: 49.16)
Tri. MSCFG w/ PvtLM 5M:
J . Riesgos d’aspiration : viscosité criterios para la clasificación demélanges ; (BLEU+1: 27.61)
J . Risk d’aspiration : viscosité criteria for the categorization ofmélanges ; (Generated English Sentence)In this example, French technical terms such as “d’aspiration” (meaning “aspiration”) and “mélanges” (meaning “mixture”) are less frequent in the training
corpus and, “d’aspiration” does not appear in both of train1 and train2, thereof
it is not translatable in both Direct 1/2 and treated as an unknown word, and“mélanges” is treated as unknown in Direct 1 since it appears only in train2.
Since this problem can not be solved also by sequential pivot translation or
pseudo-parallel corpus synthesis, it is necessary to supplement such uncovered
expressions using external dictionaries or active learning methods.

45

Moreover in this example, besides the problem of unknown words, there is also
a problem that the singular form of Spanish noun “criterio” (meaning “criterion”)
is in plural form “criterios” in the proposed triangulation method, and word orderis also wrong.

4.4 Summary
In this chapter, we have proposed a method for pivot translation using triangulation of SCFG rule tables into an MSCFG rule table that remembers the pivot,
and performing translation with pivot LMs. In experiments, we found that thesemodels are eﬀective in the case when a strong pivot language model exists.
This proposed method is eﬀective in the case that available source-pivot andpivot-target parallel corpora are not large (e.g. of less than hundred thousands
sentence pairs) and amount of available pivot monolingual corpus is large on thecontrary. Since MSCFG decoder demands huge amount of memory and computational time and is diﬃcult to distributed processing, it is not realistic to scale
up with the same framework. Moreover, although MSCFG decoder helps to select appropriate translation rule with pivot language model, it can not essentiallysolve the ambiguity problem and inappropriately connected rules still remain in
the triangulated rule table as a noise. Therefore, in the following chapter, we
discuss how to reduced the noisy rules in triangulated TMs and make it closer todirectly trained TMs.

46

5 Syntactic Disambiguation in
Pivot Translation
In Section 4, we showed example of semantic ambiguity and proposed triangulation method to consider pivot-side contextual information. However, there occurs
not only semantic ambiguity but also syntactic ambiguity in pivot translation.
This chapter discusses the problem of syntactic ambiguity due to the fact thatparts-of-speech or phrase structures are not uniquely determined in local context.

5.1 Syntactic Ambiguity[X1] 记录 [X2]

[X1] enregistrer [X2]
[X1] record [X2]

[X2] [X1] 记录[X1] dossier [X2](a) Standard triangulation method matching phrases
VP
VP[X1] enregistrer [X2]TO

[X1] 记录 [X2]VB

NP

record [X2]

[X1]

NP
NP

[X1] dossier [X2]DT
[X1]

[X2] [X1] 记录NNNPrecord [X2](b) Proposed triangulation method matching subtrees

Figure 5.1: Example of disambiguation by parse subtree matching (Fr-En-Zh)

47

In Figure 5.1 (a), we show an example of standard triangulation on Hiero TMs
that combines hierarchical rules of phrase pairs by matching pivot phrases withequivalent surface forms. This example also demonstrates problems of ambiguity:
the English word “record” can correspond to several diﬀerent parts-of-speechaccording to the context. More broadly, phrases including this word also havediﬀerent possible grammatical structures, but it is impossible to uniquely identify
this structure unless information about the surrounding context is given.
This varying syntactic structure will aﬀect translation. For example, the Frenchverb “enregistrer” corresponds to the English verb “record”, but the French noun
“dossier” also corresponds to “record” — as a noun. As a more extreme example,
Chinese is a languages that does not have inflections according to the part-ofspeech of the word. As a result, even in the contexts where “record” is used with
diﬀerent parts-of-speech, the Chinese word ‘‘记录” will be used, although theword order will change. These facts might result in an incorrect connection of“[X1] enregistrer [X2]” and “[X2] [X1] 记录” even though proper correspondenceof “[X1] enregistrer [X2]” and “[X1] dossier [X2]” would be “[X1] 记录 [X2]” and“[X2] [X1] 记录”. Hence a superficial phrase matching method based solely on
the surface form of the pivot will often combine incorrect phrase pairs, causingtranslation errors if their translation scores are estimated to be higher than theproper correspondences.
Given this background, we hypothesize that disambiguation of these cases
would be easier if the necessary syntactic information such as phrase structures
are considered during pivoting. To incorporate this intuition into our models,
we propose a method that considers syntactic information of the pivot phrase,as shown in Figure 5.1 (b). In this way, the model will distinguish translation
rules extracted in contexts in which the English symbol string “[X1] record [X2]”behaves as a verbal phrase, from contexts in which the same string acts as a nounphrase.
While standard triangulation uses only the surface forms for performing triangulation, we propose two methods for triangulation based on syntactic matching(Section 5.2.1). The first places a hard restriction on exact matching of parsetrees (Section 5.2.1) included in translation rules, while the second places a softer
restriction allowing partial matches (Section 5.2.2).

48

5.2 Triangulation with Syntactic Information
In Section 5.1, we explained the standard triangulation method and mentionedthat the pivot-side ambiguity causes incorrect estimation of translation probability and the translation accuracy might decrease. To address this problem, it isdesirable to be able to distinguish pivot-side phrases that have diﬀerent syntacticroles or meanings, even if the symbol strings are identical. In the following two
sections, we describe two methods to distinguish pivot phrases that have syntactically diﬀerent roles, one based on exact matching of parse trees, and one basedon soft matching.

5.2.1 Exact Matching of Parse Subtrees
In the exact matching method, we first train pivot-source and pivot-target T2S
TMs by parsing the pivot side of parallel corpora, and store them into rule tables
as TP S and TP T respectively. Synchronous rules of TP S and TP T take the formof X → ⟨p̂, s⟩ and X → ⟨p̂, t⟩ respectively, where p̂ is a symbol string that
expresses pivot-side parse subtree (S-expression), s and t express source and target
symbol strings. The procedure of synthesizing source-target synchronous rules
essentially follows equations (3.1)-(3.4), except using TP S instead of TSP (directionof probability features is reversed) and pivot subtree p̂ instead of pivot phrase
p. Here s and t do not have syntactic information, and thus the synthesizedsynchronous rules should be hierarchical rules explained in Section 2.1.5.
The matching condition of this method has harder constraints than matchingof superficial symbols in standard triangulation, and has the potential to reduceincorrect connections of phrase pairs, resulting in a more reliable triangulated
TM. On the other hand, the number of connected rules decreases as well in thisrestricted triangulation, and the coverage of the triangulated model might be
reduced. Therefore it is important to create TMs that are both reliable and havehigh coverage.

49

5.2.2 Partial Matching of Parse Subtrees
To prevent the problem of the reduction of coverage in the exact matching
method, we propose a partial matching method that keeps coverage just likestandard triangulation by allowing connection of incompletely equivalent pivot
subtrees. To estimate translation probabilities in partial matching, we first define
weighted triangulation generalizing the equations (3.1)-(3.4) of standard triangulation with the weight function ψ(·):ϕ (t|s) =∑∑pˆTϕ (s|t) =ϕ (t|pˆT ) ψ (pˆT |pˆS ) ϕ (pˆS |s) ,(5.1)

ϕ (s|pˆS ) ψ (pˆS |pˆT ) ϕ (pˆT |t) ,(5.2)

ϕlex (t|pˆT ) ψ (pˆT |pˆS ) ϕlex (pˆS |s) ,(5.3)

ϕlex (s|pˆS ) ψ (pˆS |pˆT ) ϕlex (pˆT |t)

(5.4)

pˆS

∑∑
pˆS pˆT

ϕlex (t|s) =∑∑pˆTϕlex (s|t) =pˆS∑∑pˆS pˆT

where pˆS ∈ TSP and pˆT ∈ PP T are pivot parse subtrees of source-pivot andpivot-target synchronous rules respectively. By adjusting ψ(·), we can control
the magnitude of the penalty for the case of incompletely matched connections.
If we define ψ(pˆT |pˆS ) = 1 when pˆT is equal to pˆS and ψ(pˆT |pˆS ) = 0 otherwise,
equations (5.1)-(5.4) are equivalent with equations (3.1)-(3.4).
Better estimating ψ(·) is not trivial, and co-occurrence counts of pˆS and pˆT are
not available. Therefore we introduce a heuristic estimation method as follows:w(pˆS , pˆT )· max w(pˆS , p̂)p̂∈TP T w(pˆS , p̂) p̂∈TP T
w(pˆS , pˆT )ψ(pˆS |pˆT ) = ∑· max w(p̂, pˆT )p̂∈TSP w(p̂, pˆT ) p̂∈TSP
ψ(pˆT |pˆS ) = ∑



0

(5.5)
(5.6)

(f lat(pˆS ) ̸= f lat(pˆT ))w(pˆS , pˆT ) = exp (−d (pˆS , pˆT ))(otherwise)(5.7)

d(pˆS , pˆT ) = T reeEditDistance(pˆS , pˆT )(5.8)

where f lat(p̂) returns the symbol string of p̂ keeping non-terminals, andT reeEditDistance(pˆS , pˆT ) is minimum cost of a sequence of operations (contract50

an edge, uncontract an edge, modify the label of an edge) needed to transformpˆS into pˆT (Klein, 1998).
According to equations (5.5)-(5.8), we can assure that incomplete match ofpivot subtrees leads d(·) ≥ 1 and penalizes such that ψ(·) ≤ 1/ed ≤ 1/e, while
exact match of subtrees leads to a value of ψ(·) at least e ≈ 2.718 times larger
than when using partially matched subtrees.

5.3 Experiments
5.3.1 Experimental Set-Up
To investigate the eﬀect of our proposed approach, we evaluate the translation
accuracy through pivot translation experiments on the United Nations Parallel
Corpus (UN6Way) (Ziemski et al., 2016). UN6Way is a line-aligned multilingual parallel corpus that includes data in English (En), Arabic (Ar), Spanish(Es), French (Fr), Russian (Ru) and Chinese (Zh), covering diﬀerent families oflanguages. It contains more than 11M sentences for each language pair, and istherefore suitable for multilingual translation tasks such as pivot translation. In
these experiments, we fixed English as the pivot language considering that it is
the language most frequently used as a pivot language. This has the positiveside-eﬀect that accurate phrase structure parsers are available in the pivot language, which is good for our proposed method. We perform pivot translationon all the combinations of the other 5 languages, and compared the accuracy ofeach method. For tokenization, we adopt SentencePiece,1 an unsupervised text
tokenizer and detokenizer, that is although designed mainly for neural MT, we
confirmed that it also helps to reduce training time and even improves translation
accuracy in our Hiero model as well. We first trained a single shared tokenization
model by feeding a total of 10M sentences from the data of all the 6 languages, set
the maximum shared vocabulary size to be 16k, and tokenized all available text
with the trained model. We used English raw text without SentencePiece tokenization for phrase structure analysis and for training Hiero and T2S TMs on thepivot side. To generate parse trees, we used the Ckylark PCFG-LA parser (Oda1https://github.com/google/sentencepiece51

et al., 2015), and filtered out lines of length over 60 tokens from all the parallel
data to ensure accuracy of parsing and alignment. About 7.6M lines remained.
Since Hiero requires a large amount of computational resources for training anddecoding, so we decided not to use all available training data but first 1M lines
for training each TM. As a decoder, we use Travatar (Neubig, 2013), and train
Hiero and T2S TMs with its rule extraction code. We train 5-gram LMs over
the target side of the same parallel data used for training TMs using KenLM(Heafield, 2011). For testing and parameter tuning, we used the first 1,000 linesof the 4,000 lines test and dev sets respectively. For the evaluation of translation
results, we first detokenize with the SentencePiece model and re-tokenized withthe tokenizer of the Moses toolkit (Koehn et al., 2007) for Arabic, Spanish, French
and Russian and re-tokenized Chinese text with KyTea tokenizer (Neubig et al.,2011), then evaluated using case-sensitive BLEU-4 (Papineni et al., 2002).
We evaluate 6 translation methods:
Cascade:Sequential pivot translation with source-pivot and pivot-target Hiero TMs(baseline, Section 3.1.1).
Tri. Hiero:
Triangulating source-pivot and pivot-target Hiero TMs into a source-target
Hiero TM using the traditional method (baseline, Section 3.1.3).
Tri. TreeExact
Triangulating pivot-source and pivot-target T2S TMs into a source-target
Hiero TM using the proposed exact matching of pivot subtrees (proposed
1, Section 5.2.1).
Tri. TreePartial
Triangulating pivot-source and pivot-target T2S TMs into a source-target
Hiero TM using the proposed partial matching of pivot subtrees (proposed
2, Section 5.2.2).
Direct:Translating with a Hiero TM directly trained on the source-target parallel
corpus without using pivot language (as an oracle).

52

BLEU Score [%]CascadeTri. Hiero Tri. TreeExact(baseline)(baseline)(proposed 1)

SourceTarget

Ar

Es
Fr
Ru
Zh

38.49
33.34
24.63
27.27

30.95
25.08
18.70
21.77

34.20
29.93
22.94
22.78

‡ 34.97‡ 30.68‡ 23.94‡ 25.17‡ 35.94‡ 30.83‡ 24.15‡ 25.07

Es

Ar
Fr
Ru
Zh

27.18
43.24
28.83
27.08

22.72
35.40
22.43
23.36

22.97
38.74
26.35
24.54

‡ 24.09‡ 39.62‡ 27.25
25.00‡ 24.45‡ 40.12‡ 27.41† 25.16

Fr

Ar
Es
Ru
Zh

25.10
45.20
27.42
25.84

19.88
37.75
20.64
21.79

21.65
40.16
24.71
23.16

21.40
‡ 41.03† 25.2423.56† 22.13‡ 41.99‡ 25.64
23.53

Ru

Ar
Es
Fr
Zh

22.53
37.60
34.05
28.03

18.71
31.33
27.11
21.81

19.82
34.56
30.75
24.88

19.86
34.96
† 31.43
25.07

20.35
‡ 35.62‡ 31.67
25.12

Zh

Ar
Es
Fr
Ru

20.09
30.66
25.97
21.16

14.82
23.15
19.55
14.79

16.66
27.84
23.82
18.63

17.01
27.99
24.34
‡ 19.58‡ 17.7328.05† 24.35‡ 19.59

DirectTri. TreePartial(proposed 2)

Table 5.1: Results of each method, comparing the proposed syntactic matching
methods in triangulation with other methods.

5.3.2 Results
Translation accuracy: The result of experiments using all combinations ofpivot translation tasks for 5 languages via English is shown in Table 5.1. From theresults, we can see that the proposed partial matching method of pivot subtrees
in triangulation outperforms the standard triangulation method for all language
pairs and achieves higher or almost equal scores than proposed exact matching
method. The exact matching method also outperforms the standard triangulation
method in the majority of the language pairs, but has a lesser improvement thanpartial matching method. Sequential pivot translation is uniformly weak than all
the triangulation methods as shown in the previous research and Chapter 4.
Eﬀect on coverage: In Table 5.2 we show the comparison of coverage of each53

Number of source-side unique phrases/words
Tri. TreeExact
Tri. TreePartial

SourceTarget

Ar

Es
Fr
Ru
Zh

2.580M / 5,072
2.589M / 5,067
2.347M / 5,085
2.324M / 5,034

2.646M / 5,077
2.658M / 5,071
2.406M / 5,088
2.386M / 5,040

Es

Ar
Fr
Ru
Zh

1.942M / 5,182
2.062M / 5,205
1,978M / 5,191
1,920M / 5,175

2.013M / 5,188
2.129M / 5,210
2.037M / 5,197
1.986M / 5,180

Fr

Ar
Es
Ru
Zh

2.176M / 5,310
2.302M / 5,337
2.203M / 5.311
2.162M / 5.313

2.233M / 5,316
2.366M / 5,342
2.266M / 5,318
2.215M / 5,321

Ru

Ar
Es
Fr
Zh

2.437M / 5,637
2.478M / 5.677
2.479M / 5,661
2.466M / 5,682

2.505M / 5,644
2.536M / 5,682
2.531M / 5,665
2.515M / 5,688

Zh

Ar
Es
Fr
Ru

1.480M / 9,428
1.504M / 9,523
1.499M / 9,490
1.518M / 9,457

1.556M / 9,474
1.570M / 9,555
1,568M / 9,520
1.593M / 9,487

Table 5.2: Comparison of rule table coverage in proposed triangulation methods.

proposed triangulated method. From this table, we can see that the exact matching method reduces several percent in number of unique phrases while the partial
matching method keeps the same coverage with surface-form matching. We canconsider that it is one of the reasons of the diﬀerence in improvement stability
between the partial and exact matching methods.
Noise reduction: The main motivation of using parse trees in the proposed
methods is to prevent inappropriate connection of phrase correspondences andreduce the noises in rule table. To investigate how the syntactic matching methods success to remove noisy rules, we perform analysis of noise ratio. Noisy rules
must contain source and target phrases having no correspondence in meaning,
though we can not make a decision for all phrase pair candidates in rule tables.
Therefore we assume that directly trained TMs with source-target parallel corpus

54

Noise Ratio in Triangulated Table [%]Tri. Hiero Tri. TreeExact Tri. TreePartial

SourceTarget

Ar

Es
Fr
Ru
Zh

78.40
81.39
81.87
75.70

63.61 (-14.79)
67.31 (-14.08)
69.23 (-12.64)
63.06 (-12.64)

68.51 (-9.88)
72.22 (-9.17)
73.84 (-8.03)
67.72 (-7.98)

Es

Ar
Fr
Ru
Zh

80.03
81.55
81.45
74.05

64.97 (-15.06)
65.30 (-16.26)
68.02 (-13.43)
61.60 (-12.45)

69.80 (-10.23)
70.61 (-10.94)
72.67 (-8.78)
65.90 (-8.15)

Fr

Ar
Es
Ru
Zh

81.77
80.94
82.77
76.14

67.69 (-14.08)
64.94 (-16.00)
69.84 (-12.93)
64.29 (-11.85)

72.39 (-9.38)
70.20 (-10.74)
74.52 (-8.25)
68.53 (-7.61)

Ru

Ar
Es
Fr
Zh

82.15
79.80
82.41
76.07

70.15 (-12.00)
67.16 (-12.64)
70.10 (-12.31)
64.31 (-11.76)

74.60 (-7.55)
71.68 (-8.12)
74.76 (-7.65)
68.67 (-7.40)

Zh

Ar
Es
Fr
Ru

80.05
77.94
78.24
79.80

66.90 (-13.15)
65.53 (-12.41)
68.54 (-9.70)
67.07 (-12.73)

71.23 (-8.82)
69.70 (-8.24)
72.51 (-5.73)
71.27 (-8.53)

Table 5.3: Comparison of noise ratio in triangulated rule table

have fine approximation close to the ideal distribution of translation probability.
To compute the noise ratio noise(Ttri |Tdir ) of triangulated rule table Ttri withdirectly trained table Tdir , we define:∑noise (Ttri |Tdir ) =(s,t)∈Ttri \Tdir ϕ (t|s)(s,t)∈Ttri ϕ (t|s)

∑

(5.9)

where ϕ(t|s) is forward translation probability which can be considered as a mostimportant feature of rule table. In Table 5.3, we show the calculated noise ratioof rule table for each triangulation method and language pair. This result showsthat although triangulated rule tables contain many noisy rules, the syntactic
matching methods indeed success to reduce them. Tri.TreeExact reduces up to-16.26% of noisy rules, and Tri.TreePartial reduces up to -10.94%. The reason

55

Distribution Error Rate (MAE / RMSE) [%]Tri. Hiero
Tri. TreeExact Tri. TreePartial

SourceTarget

Ar

Es
Fr
Ru
Zh

14.16 / 10.62
13.01 / 9.72
12.64 / 9.51
15.88 / 11.96

14.49 / 11.06
13.52 / 10.19
12.33 / 9.24
13.69 / 10.42

13.96 / 10.56
12.90 / 9.65
12.03 / 8.97
13.81 / 10.42

Es

Ar
Fr
Ru
Zh

13.90 / 10.29
13.39 / 10.61
12.81 / 9.71
16.02 / 12.09

13.84 / 10.30
14.51 / 11.30
12.92 / 9.70
13.94 / 10.69

13.44 / 9.92
13.95 / 10.89
12.52 / 9.38
14.01 / 10.64

Fr

Ar
Es
Ru
Zh

13.40 / 9.98
14.25 / 11.38
12.58 / 9.58
15.45 / 11.74

13.10 / 9.76
14.39 / 11.29
12.46 / 9.37
13.34 / 10.28

12.70 / 9.38
14.05 / 11.03
11.98 / 8.99
13.40 / 10.23

Ru

Ar
Es
Fr
Zh

12.68 / 9.35
13.27 / 10.05
12.29 / 9.28
15.34 / 11.72

12.36 / 9.16
13.68 / 10.54
12.84 / 9.78
13.13 / 10.10

11.98 / 8.79
13.12 / 10.00
12.13 / 9.17
13.25 / 10.11

Zh

Ar
Es
Fr
Ru

12.57 / 9.11
13.25 / 9.78
12.86 / 9.49
12.22 / 9.14

12.86 / 9.39
13.58 / 10.16
12.67 / 9.44
12.40 / 9.31

12.57 / 9.09
13.22 / 9.79
12.25 / 9.07
12.12 / 9.03

Table 5.4: Comparison of distribution error rate in triangulated rule table

why the noise reduction rate at Tri.TreePartial is smaller than Tri.TreePartial that
Tri.TreePartial weakens the influence of noisy rules instead of removing them tokeep the coverage.
Improvement of probability estimation: Although we see that syntactic
matching methods help to reduce noisy rules, there is no guarantee that they can
improve the estimation of translation probabilities. In Table 5.4, we show mean
absolute error (MAE) and root mean square error (RMSE) for the distributionof forward translation probability scores of triangulated rule tables with directlytrained rule tables. To calculate MAE and RMSE, we ignore the noisy rules which
are not contained in directly trained rule tables, to separate diﬀerent factors.
From the result, we can see that Tri.TreePartial reduces MAE and RMSE, namely
makes the distribution closer to the ideal, in almost language pairs. On the56other hand, Tri.TreeExact does not stably reduce them. It may be induced by
the fact that the restricted matching condition of Tri.TreeExact excludes many
unmatched phrase pair candidates and remove even translation rules which are
not noisy. Therefore, we can consider that soften restriction of matching condition
helps also to improve the estimation of translation probabilities.
Qualitative analysis: We show an example of a translated sentences for which
pivot-side ambiguity is resolved in the syntactic matching methods:
Source Sentence in French:
La Suisse encourage tous les États parties à::::::::::
soutenir ::le:::::::
travail::::::::::::
conceptuelquefait actuellement le Secrétariat .
::::::::::::::::::::::::::::::::::::
Corresponding Sentence in English:
Switzerland encourages all parties to support the current conceptual workof the secretariat.
Reference in Spanish:Suiza alienta a todos los Estados partes :a:::::
que::::::::
apoyen:::la:::::::
actual:::::::
labor
conceptualde la Secretaría .
:::::::::::::::::::::::::::
Direct:Suiza alienta a todos los Estados partes a que apoyen el trabajo conceptualque se examinan en la Secretaría . (BLEU+1: 55.99)
Tri. Hiero:Suiza :::::::::::::
conceptuales :::::
para :::::::
apoyar:::la ::::::
labor ::::
que :::
en :::::
estos:::::::::::
momentos:::la :::::::::::
Secretaría
alienta a todos los Estados Partes . (BLEU+1: 29.74)
Tri. TreeExact:
Suiza alienta a todos los Estados Partes a apoyar la labor conceptualque actualmente la Secretaría . (BLEU+1: 43.08)
Tri. TreePartial:
Suiza alienta a todos los Estados Partes a apoyar la labor conceptualque actualmente la Secretaría . (BLEU+1: 43.08)

57

The results of Tri.TreeExact and Tri.TreePartial are same in this example.
Digest of the derivation process in Tri. Hiero is:S ⇒ ⟨X0 , X0 ⟩⇒ ⟨La Suisse X1 , Suiza X1 ⟩⇒ ⟨La Suisse X2 ., Suiza X2 .⟩⟨⇒ La Suisse X3 partie X., Suiza X
X3 P artes .
::4
::4

⟩

⇒ ...
On the other hand, digest of the derivation process in Tri.TreeExact/Tri.TreePartialis:S ⇒ ⟨X0 , X0 ⟩⇒ ⟨La Suisse X1 , Suiza X1 ⟩⇒ ⟨La Suisse encourage X2 X3 , Suiza alienta a X2 X3 ⟩⇒ ⟨La Suisse encourage tous les X4 partie X3 , Suiza alienta a todos X4 P artes X3 ⟩⇒ ⟨La Suisse encourage tous les Etats partie X3 , Suiza alienta a todos los Estados P artes X3 ⟩⇒ ...

Here we can see that the derivation in Tri.Hiero uses rule X → ⟨X0 parties X1 ,X1 X0 Partes⟩2 causing incorrect re-ordering of phrases followed by steps of incorrect word selection.3 On the other hand, derivation in Tri.TreeExact and
Tri.TreePartial uses rule X → ⟨tous les X0 parties, todos X0 Partes⟩4 synthesized from T2S rules with common pivot subtree (NP (DT all) (NP’ XNNP(NNS parties)). We can confirm that the derivation improves word-selection
and word-reordering by using this rule.

2

The words emphasized with underline and wavy-underline in the example correspond to X0
and X1 respectively.
3
For example, the word “conceptuales” with italic face in Tri.Hiero takes the wrong form andposition.
4
The words emphasized in bold face in the example correspond to the rule.

58

vocabulary size:
source embedding size:
target embedding size:
output embedding size:
encoder hidden size:
decoder hidden size:
LSTM layers:attention type:attention hidden size:
optimizer type:loss integration type:batch size:
max iteration:
dropout rate:
decoder type:

16k (shared)
512
512
512
512
512
1
MLP
512
Adam
mean
2048
200k
0.3
Luong+ 2015

Table 5.5: Main parameters of NMT training

5.3.3 Comparison with Neural MT:
Recent results (Firat et al., 2016; Johnson et al., 2017) have found that neuralmachine translation systems can gain the ability to perform translation with zeroparallel resources by training on multiple sets of bilingual data. However, previous
work has not examined the competitiveness of these methods with pivot-based
symbolic SMT frameworks such as PBMT or Hiero. In this section, we comparea zero-shot NMT model and other pivot NMT methods with our pivot-based
Hiero models. To train and evaluate NMT models, we adopt NMTKit.5 Detailed
parameters to train NMT models are shown in Table 5.5.
We evaluate 4 additional translation methods:
Cascade NMT:
Sequential pivot translation with source-pivot and pivot-target NMTs (Section 3.1.1).
5

https://github.com/odashi/nmtkit

59

Source

BLEU Score [%]Tri.
Cascade Cascade
TreePartial
Heiro
NMT

TargetDirectHiero

DirectNMT

Synthetic
NMT

Zero-Shot
NMT

Ar

Es
Fr
Ru
Zh

38.49
33.34
24.63
27.27

38.25
33.16
27.00
30.04

35.94
30.83
24.15
25.07

30.95
25.08
18.70
21.77

31.62
26.91
21.67
23.70

32.35
29.51
21.81
25.63

8.18
8.57
5.79
5.04

Es

Ar
Fr
Ru
Zh

27.18
43.24
28.83
27.08

26.02
41.83
30.65
32.36

24.45
40.12
27.41
25.16

22.72
35.40
22.43
23.36

21.21
31.84
23.60
26.03

23.01
36.57
25.97
27.31

5.22
15.04
7.57
8.62

Fr

Ar
Es
Ru
Zh

25.10
45.20
27.42
25.84

23.28
44.49
28.29
29.10

22.13
41.99
25.64
23.53

19.88
37.75
20.64
21.79

18.66
32.93
20.87
23.14

18.83
36.78
23.60
24.96

8.08
14.37
8.77
11.95

Ru

Ar
Es
Fr
Zh

22.53
37.60
34.05
28.03

23.19
38.67
33.26
31.39

20.35
35.62
31.67
25.12

18.71
31.33
27.11
21.81

19.71
31.25
27.34
24.25

19.21
31.22
29.10
25.46

3.18
10.42
9.76
9.46

Zh

Ar
Es
Fr
Ru

20.09
30.66
25.97
21.16

20.17
32.69
27.68
23.17

17.73
28.05
24.35
19.59

14.82
23.15
19.55
14.79

16.89
26.01
23.35
18.40

18.01
27.80
25.46
20.53

10.38
6.13
7.12
3.21

Table 5.6: Comparison of SMT and NMT in multilingual translation tasks.

Synthetic NMT:
Generating pseudo-parallel corpus synthesized by translating pivot-side ofsource-pivot parallel corpus with pivot-target NMT (Section 3.1.2).
Zero-Shot NMT:
Training single shared model with pvt ↔ {src,target} parallel data according to Johnson et al. (2017).
Direct NMT:
Translating with NMT directly trained on the source-target parallel corpus
without using pivot language (for comparison).
Training data for Cascade NMT, Synthetic NMT, and Zero-Shot NMT are same

60with Pivot Hiero methods (source-pivot and pivot-target corpora), and training
data for Direct NMT is same with Direct Hiero.
In Table 5.6, we show BLEU score of each translation task and language pair.
From the results we see the tendency of NMT that directly trained model achieves
high translation accuracy even for translation between languages of diﬀerent families, on the other hand, the accuracy is drastically reduced in the situation when
there is no source-target parallel corpora for training. Among pivot and zero-shot
methods for NMT, Synthetic NMT achieves the highest score for almost language
pairs. The reason why Synthetic NMT outperforms Cascade NMT for the majority of language pairs, may be that multi-layer NNs have robustness for noisy
training data, and can optimize the trained model with fine-tuning technique. On
the other hand, for Cascade NMT, fine-tuning is available only for source-pivot
and pivot-target TMs separately and not for the whole pipelined system.
In our setting, while bilingually trained NMT systems were competitive oroutperformed Hiero-based models, zero-shot translation is uniformly weaker. Thismay be because we used only single LSTM layer for each of encoder and decoder,or because the amount of parallel corpora or language pairs were not suﬃcient.
Thus, we can posit that while zero-shot translation has demonstrated reasonable
results in some settings, successful zero-shot translation systems are far fromtrivial to build, and pivot-based symbolic MT systems such as PBMT or Hieromay still be a competitive alternative.

5.4 Summary
In this chapter, we have proposed a method of pivot translation using triangulation with exact or partial matching method of pivot-side parse subtrees. In
experiments, we found that these triangulated models are eﬀective in particularwhen allowing partial matching. From the analysis, we confirmed that the syntactic matching methods indeed help to reduce inappropriately connected rules
and specifically partial matching method can stably improve the estimation oftranslation probabilities.

61

6 Syntactic and Non-Redundant
Segment Selection for Active
Learning
In Section 3.2, we showed the representative segment selection methods and theiradvantages and disadvantages, and mentioned problems of redundancy and fragmentation in segment selection phrase selection method using n-gram as shownin Figure 6.1 (a). In this chapter, we propose two methods that aim to solve thesetwo problems and improve the eﬃciency and reliability of segment-based active
learning for SMT. For the problem of overlapping phrases, we note that by merging overlapping phrases, as shown in Figure 6.1 (b), we can reduce the number of
redundant words annotated and improve training eﬃciency. We adopt the idea of
maximal substrings (Okanohara and Tsujii, 2009; Yamamoto and Church, 2001)
which both encode this idea of redundancy. For the problem of phrase structure
fragmentation, we propose a simple heuristic to count only well-formed syntactic
constituents in a parse tree, as shown in Figure 6.1 (c).

6.1 Compact and Syntactically Coherent
Segment Selection
In this section, we explain the two proposed methods to solve the problems of
redundancy and fragmentation and the combination method of them.

62any one of the preceding claims(a) Conventional n-gram selection method (n = 4)any  one  of  the  preceding  claims(b) Proposed maximal phrase selection method
NP
PP
NP
DT CD IN

NP
DT

JJ

NNS

any one of the preceding claims(c) Proposed parse subtree selection method

Figure 6.1: Conventional and proposed data selection methods

6.1.1 Segment Selection based on Phrase Maximality
To solve both the problem of overlapping phrases and the problem of requiring
limits on phrase length for computational reasons, we propose a method using theidea of maximal substrings (Okanohara and Tsujii, 2009). Maximal substrings areformally defined as “a substring that is not always included in a particular longer
substring.” For example, if we define pi as a phrase and occ(pi ) as its occurrence
count in a corpus, and have the following data:
p1 = “one of the preceding”,

occ(p1 ) = 200, 000

p2 = “one of the preceding claims”,

occ(p2 ) = 200, 000

p3 = “any one of the preceding claims”,

occ(p3 ) = 190, 000

p1 = “one of the preceding” always co-occurs with the longer p2 = “one of thepreceding claims” and thus is not a maximal substring. On the other hand, p2
does not always co-occur with p3 , and thus p2 will be maximal. This relationship63can be defined formally with the following semi-order relation:
s1 ⪯ s2 ⇔ ∃α, β : s2 = αs1 β ∧ occ(s1 ) = occ(s2 ).

(6.1)

Demonstrating this by the previous example, p1 = αp2 β, α = “”, β = “claims”
hold, meaning p1 is a sub-sequence of p2 , and p2 is a sub-sequence of p3 in asimilar manner. Since p1 is a sub-sequence of p2 and occ(p1 ) = occ(p2 ) = 200, 000,
p1 ⪯ p2 holds. However, although p2 is a sub sequence of p3 , because occ(p2 ) =200, 000 ̸= 190, 000 = occ(p3 ), the relation p2 ⪯ p3 does not hold. Here, we say p
has maximality 1 if there does not exist any q other than p itself that meets p ⪯ q,
and we call such a phrase a maximal phrase.
All the maximal phrases in the source language corpus with N words can beeﬃciently enumerated with linear time O(N ) by using enhanced suﬃx array (Kasai et al., 2001). Since it is required to compare strings O(logN ) times to obtainoccurrence count of each maximal phrase for binary search, it is required to compare strings O(N logN ) times for all the maximal phrases (Okanohara and Tsujii,2009). Since the number of maximal phrases to be extracted is at most N − 1, we
can adopt sorting algorithm of computational complexity O(N logN ) to enumerate them in descending order by frequency. In actual implementation of maximal
phrase extraction, we exclude word strings containing newline characters, andextract only those with occurrence counts of 2 or more. This is to prevent alarge number of word substrings including most sentences in the source language
corpus from being selected as the maximal phrase of occurrence count 1.
To apply this concept to active learning, our proposed method limits translation
data selection to only maximal phrases. This has two advantages. First, it reduces
overlapping phrases to only the maximal phrases, allowing translators to cover
multiple high-frequency phrases in the translation of a single segment. Second,
it removes the need to set arbitrary limits on the length of strings such as n = 4
used in previous work.
However, it can be easily noticed that while in the previous example p2 is
included in p3 , their occurrence counts are close but not equivalent, and thus bothare maximal phrases. In such a case, the naïve implementation of this method
1

Maximality is a term of algebra, and x ∈ S is called maximal element of a subset S ⊂ P ofsome partially ordered set (P, ⪯) if all y ∈ S, x ⪯ y implies x = y.

64can not remove these redundant phrases, despite the fact that it is intuitively
preferable that the selection method combines phrases if they have almost thesame occurrence count. Thus, we also propose to use the following semi-orderrelation generalized with parameter λ:∗s1 ⪯ s2 ⇔ ∃α, β : s2 = αs1 β ∧ λ · occ(s1 ) < occ(s2 )(6.2)

where λ takes a real numbered value from 0 to 1. We redefine maximality using∗∗
this semi-order ⪯ as λ-maximality, and call maximal phrases defined with ⪯ λmaximal phrases in contrast to standard maximal phrases. We propose a segment
selection method in which we sequentially add highly frequent λ-maximal phrases
uncovered in existing parallel corpus.
By setting the parameter λ of λ-maximal phrase selection to be smaller than1, we can remove the restriction that the two segments under comparison beof exactly equal counts, allowing them to have only approximately the sameoccurrence count. As special cases, it is equivalent with standard maximal phrase
selection when λ = 1 − ϵ (ϵ is a small positive number close to zero), and it
becomes a random selection of full-sentences when λ = 0. In consideration of
the possibility that both advantages can be compatible, we set to λ = 0.5, which
is an intermediate value between two special values, for comparison with othermethods in this research. By using λ-maximal phrases with λ = 0.5 instead ofstandard maximal phrases, we can remove a large number of phrases that areincluded in a particular longer phrase more than half the time, indicating that it
might be preferable to translate the longer phrase.
Since λ-maximal phrases with λ < 1 always satisfies the condition of standard maximal phrase, all candidates for λ-maximal phrase in the source language
corpus can be searched among all standard maximal phrases.

6.1.2 Phrase Selection based on Parse Trees
In this section, we propose a second phrase selection method based on the resultsfrom the syntactic analysis of source language data. This method first processes
all the source language data with a phrase structure parser, traverses and counts
up all the subtrees of parse trees as shown in Figure 6.2, and finally selects

65

NP
PP
NP
DT CD IN

NP
DT

JJ

NNS

any one of the preceding claims

OK

OK
NG

Figure 6.2: Phrase selection based on parse trees
phrases corresponding to a subtree in frequency order. We propose this method
because we expect the selected phrases to have syntactically coherent meaning,potentially making human translation easier than other methods that do not use
syntactic information. In this research, we aim to investigate the influence of
selected phrases on active learning eﬃciency, and thus we use parse trees only
for phrase extraction so that it is easier to compare with other phrase selection
methods.
Since the phrase selection methods based on n-gram or phrase maximality countphrases as superficial sequences of words, for example if we have a phrase “areproposed and discussed”, the methods count up also the sub sequence of words“are proposed”, while the proposed method based on parse trees does not selectthat because of fragmentation. Therefore, the occurrence counts by this syntactic
method tend to be lower than superficial phrase counting, and in consequence thepriority that phrases with length 2 or more words are selected tends to be lower
than single words.
It should be noted that because this method counts all subtrees, it is capable of
selecting overlapping phrases like the methods based on n-grams. Therefore wealso experiment with a method using together both subtrees and the λ-maximal
phrases proposed in Section 6.1.1 to select both syntactic and non-redundantsegments.

66

6.2 Simulation Experiment6.2.1 Experimental Set-Up
To investigate the eﬀects of the phrase selection methods proposed in Section
6.1, we first performed a simulation experiment in which we incrementally retraintranslation models and evaluate the accuracy after each step of data selection. In
this experiment, we chose English as a source language and French and Japanese
as target languages. To simulate a realistic active learning scenario, we started
from given parallel data in the general domain and sequentially added additionalsource language data in a specific target domain. For the English-French translation task, we adopted the Europarl corpus2 (Koehn, 2005) from WMT20143 asa base parallel data source and EMEA4 (Tiedemann, 2009), PatTR5 (Wäschle
and Riezler, 2012), and Wikipedia titles, used in the medical translation task, as
the target domain data. For the English-Japanese translation task, we adopted
the broad-coverage example sentence corpus provided with the Eijiro dictionary6as general domain data, and the ASPEC7 (Nakazawa et al., 2016) scientific paper abstract corpus as the target domain data. For pre-processing, we tokenizedJapanese corpora using the KyTea word segmenter (Neubig et al., 2011) and filtered out the lines of length over 60 from all the training parallel data to ensure
accuracy of parsing and alignment. We show the details of the parallel dataset
after pre-processing in Table 6.1.
For the machine translation framework, we used phrase-based SMT (Koehn
et al., 2003) with the Moses toolkit8 (Koehn et al., 2007) as a decoder. To eﬃciently re-train the models with new data, we adopted inc-giza-pp,9 a specializedversion of GIZA++ word aligner (Och and Ney, 2003) supporting incremental
training, and the memory-mapped dynamic suﬃx array phrase tables (MMSAPT)
feature of Moses (Germann, 2014) for on-memory construction of phrase tables.
2

http://www.statmt.org/europarl/
http://statmt.org/wmt14/
4
http://opus.lingfil.uu.se/EMEA.php
5
http://www.cl.uni-heidelberg.de/statnlpgroup/pattr/
6
http://eijiro.jp
7
http://lotus.kuee.kyoto-u.ac.jp/ASPEC/8
http://www.statmt.org/moses
9
https://github.com/akivajp/inc-giza-pp
3

67

Lang Pair

DomainDatasetAmountGeneral (Base)Train

En: 47.6M Words

1.89M Sent.
Fr: 49.4M Words
En-Fr

15.5M Sent.
Medical

Train(Target)Test

En: 393M Words
Fr: 418M Words

Dev1000 Sent.
500 Sent.
414k Sent.

General (Base)Train

En: 6.72M Words
Ja: 9.69M Words

En-Ja

1.87M Sent.
ScientificTrain

En: 46.4M Words
Ja: 57.6M Words(Target)Test

1790 Sent.

Dev

1790 Sent.

Table 6.1: Details of parallel data

We train 5-gram models over the target side of all the general domain and target
domain data using KenLM (Heafield, 2011). For the tuning of decoding parameters, since it is not realistic to run MERT (Och, 2003) at each retraining step,
we tuned the parameters to maximize the BLEU-4 score (Papineni et al., 2002)
for the baseline system, and re-used the parameters thereafter. We compare thefollowing 8 segment selection methods, including 2 random selection methods, 2
conventional methods and 4 proposed methods:
sent-rand:
Select sentences randomly.
4gram-rand:
Select n-gram strings with length of up to 4 in random order.
sent-by-4gram-freq:
Select the sentence including the most frequent uncovered phrase withlength of up to 4 words (baseline 1, Section 3.2.2).

68

4gram-freq:
Select the most frequent uncovered phrase with length of up to 4 words(baseline 2, Section 3.2.3).
maxsubst-freq:
Select the most frequent uncovered maximal phrase (proposed, Section
6.1.1)
reduced-maxsubst-freq:
Select the most frequent uncovered λ-phrase with λ = 0.5 (proposed, Section 6.1.1)
struct-freq:
Select the most frequent uncovered phrase extracted from the subtrees (proposed, Section 6.1.2).
reduced-struct-freq:
Select the most frequent uncovered λ-maximal phrase (λ = 0.5) extracted
from the subtrees (proposed, Section 6.1.1 and Section 6.1.2).
To generate oracle translations, we used an SMT system trained on all of thedata in both the general and target-domain corpora. To generate parse trees, weused the Ckylark parser (Oda et al., 2015).

6.2.2 Results and Discussion
Comparison of eﬃciency: From the results obtained by simulation experiments, Table 6.2 shows the transition of BLEU scores for each method at thetime of adding no words, 10k words, 100k words, 1M words, and all segments.10
The score at the time of adding all segments is considered to be the performance
10

BLEU score of the base system in English-Japanese translation started from a low value lower
than 10 because the domain-specific parallel sentences are extremely short. In the previous
research (Ananthakrishnan et al., 2010a,b; Bloodgood and Callison-Burch, 2010; Haﬀari et al.,2009), BLEU score is commonly used in domain adaptation with active learning from thesituation where the parallel sentences are insuﬃcient. Therefore, we used the same evaluation
measure in this research.

69

BLEU-4 Score [%]Lang Pair

En-Fr

En-Ja

Selection Method

10k Words

100k Words

1M Words

All Segments

sent-rand

25.57

25.72

27.35

30.02

4gram-rand
sent-by-4gram-freq

25.53
25.55

25.52
26.12

27.16
27.93

28.32
30.69

25.61
25.55

26.16
25.84

27.89
27.49

28.75
29.60

reduced-maxsubst-freq
struct-freq

25.63
25.85

26.10
26.86

27.91
29.06

29.81
30.03

reduced-struct-freq† 26.08† 27.18† 29.40

30.20

sent-rand

10.44

13.03

15.58

21.22

4gram-rand
sent-by-4gram-freq

10.57
11.14

13.37
14.49

17.61
17.66

19.71
21.06

11.49
11.72

15.07
15.13

18.27
18.58

19.74
19.88

reduced-maxsubst-freq
struct-freq

11.87
12.02

† 15.72
15.44

18.71
18.61

19.59
19.97

reduced-struct-freq† 12.27

15.66† 18.91

19.83

4gram-freq
maxsubst-freq

4gram-freq
maxsubst-freq

No Addition

25.39

9.37

Table 6.2: Transition of BLEU score according to the number of additional words.
Underlines indicate that the score is the maximum among the random
selection method and the baseline method at each point in time immediately after the addition of 1M words. Bold face indicates thescores the proposed method exceeding the underlined scores. Daggers† indicate the best score in all the methods in each stage.

limit of each method, since this is the translation accuracy by using all the translated segments. However, since the number of source words to be added varies
significantly, comparison can not be made simply from the viewpoint of active
learning eﬃciency.
Comparing two random selection methods and the two baseline methods form
the table, the accuracy elongation with 4gram-freq is stably high until additionof 100k words. Accordingly, we can confirm the advantages of selecting highly
frequent segments instead of whole sentences. However, at the time of additionof 1M words in English-French translation, the score of 4gram-freq is lower thansent-by-4gram-freq, and the score of 4gram-freq is lower than sent-by-4gram-freq
and sent-rand at the time of addition of all segments in both language pairs.
From this fact, it can be seen that in the case of adding a number of words70%/(86FRUHYV$GGLWLRQDO:RUGV(Q)U




%/(86FRUH>@

%/(86FRUHYV$GGLWLRQDO:RUGV(Q)U
VHQWE\JUDPIUHT
JUDPIUHT
UHGXFHGPD[VXEVWIUHT
UHGXFHGVWUXFWIUHT

%/(86FRUH>@







N



NNNN
$GGLWLRQDO:RUGV>6RXUFH:RUGV@N

%/(86FRUHYV$GGLWLRQDO:RUGV(Q-D





%/(86FRUHYV$GGLWLRQDO:RUGV(Q-D%/(86FRUH>@%/(86FRUH>@





N

N







N
N
N
N
$GGLWLRQDO:RUGV>6RXUFH:RUGV@








N

VHQWE\JUDPIUHT
JUDPIUHT
UHGXFHGPD[VXEVWIUHT
UHGXFHGVWUXFWIUHT2UDFOH6FRUH 



VHQWE\JUDPIUHT
JUDPIUHT
UHGXFHGPD[VXEVWIUHT
UHGXFHGVWUXFWIUHTNNNN
$GGLWLRQDO:RUGV>6RXUFH:RUGV@N


N

VHQWE\JUDPIUHT
JUDPIUHT
UHGXFHGPD[VXEVWIUHT
UHGXFHGVWUXFWIUHT2UDFOH6FRUH 

N
N
N
N
$GGLWLRQDO:RUGV>6RXUFH:RUGV@N

Figure 6.3: BLEU score vs. number of additional source words in each method(upper: En-Fr translation task, bottom: En-Ja translation task, left:
up to 100k additional words, right: up to 1M additional words)
more than a certain amount, active learning eﬃciency of 4gram-freq is lower thansentence selection methods, and its performance limit is not high. It may because
the length of selected segments is limited to 4 words in 4gram-freq as mentioned
in Section 3.2.3, and the inability to learn correspondence of longer segments is
disadvantage in MT.
Next, we compare the proposed methods with the baseline methods. In the proposed methods, reduced-maxsubst-freq is almost always higher than maxsubstfreq, and reduced-struct-freq is almost always higher than struct-freq. Thereby,
it is considered that coverage is improved with fewer additional words by giving

71priority to longer segments based on λ-maximality. Therefore, we compare thetwo proposed methods based on λ-maximality with two baseline methods in moredetail.
In Figure 6.3, we show the evaluation score results by the number of additionalsource words up to 100k and 1M words. Additionally, we show the score of MT
trained and evaluated using the whole data of the base corpus and the additional
corpus as an oracle score, together with the right-side graph of up to the 1M
additional words.
In reduced-maxsubst-freq, English-Japanese translation score was stably higher
than the baseline methods. However in English-French translation, it got almost
the same score as 4gram-freq up to the point of adding 1M words. However, since
the score at the time of adding all the segments in both language pairs greatly
exceeds 4gram-freq and 4gram-rand, the performance limit of this method ishigh. This seems to be a major reason that the problem of the maximum phrase
length limitation as described above does not occur in the proposed methods.
To investigate the reason why there is no significant diﬀerence between reducedmaxsubst-freq and 4gram-freq in English-French translation, we look over thehighly frequent segments selected by both methods. The most frequent uncovered
segments “according to claim” (1,502,455 times), “claim 1” (1,133,243 times),“characterized in that” (858,404 times), etc. were commonly selected in thetwo methods, and redundant segments as described in Section 3.2.3 were few,
but rather fragmentation of phrase structures was conspicuous. A state in which
multiple frequent 4-gram segments have many common word sub-sequence occurs
particularly when frequent segments including more than 4 words. It seems tobe because such frequent longer segments in the medical-domain corpus usedin this experiment does not contain much domain-specific expressions, and they
are already included in the large-scale general-domain corpus. On the other
hand, in English-Japanese translation, many duplicates are seen in such as highly
frequent segments “results suggest that” (6,352 times), “these results suggest”(5,115 times), “these results suggest that” (4,791 times), etc. selected by 4gramfreq. In this situation, reduced-maxsubst-freq can demonstrate higher learningeﬃciency by combining such overlapping segments into one. In particular, since
the general-domain corpus used in English-Japanese translation is comparatively

72

All Selected Segments
Average#Segments #Words Phrase Length

First 10k Words
Average#Segments Segment Length

Lang Pair

Selection Method

En-Fr

sent-by-4gram-freq
4gram-freq
maxsubst-freq
reduced-maxsubst-freq
struct-freq
reduced-struct-freq

10.6M
40.1M
62.4M
45.9M
14.1M
7.33M

269M
134M
331M
246M
94.2M
41.3M

25.4
3.34
5.30
5.36
6.68
5.63

310
3.62k
2.39k
2.95k
4.01k
4.55k

32.1
2.76
4.17
3.39
2.49
2.20

En-Ja

sent-by-4gram-freq
4gram-freq
maxsubst-freq
reduced-maxsubst-freq
struct-freq
reduced-struct-freq

1.28M
8.48M
7.29M
6.06M
1.45M
1.10M

33.6M
26.0M
25.8M
21.7M
4.85M
3.33M

26.3
3.07
3.54
3.58
3.34
3.03

560
4.70k
4.51k
4.76k
6.64k
6.73k

17.8
2.13
2.22
2.10
1.51
1.49

Table 6.3: Number of segments and average words/segment in each method

small scale of 400k sentences which summarizes daily expressions, frequent longsegments in the scientific-domain corpus are not much covered.
In both language pairs, reduced-struct-freq almost stably achieves the highest
score among all the segment selection methods. Only at the time of adding 100kwords in English-Japanese translation, the score of reduced-maxsubst-freq is the
maximum, though it is a small diﬀerence with reduced-maxsubst-freq, and thefluctuation range of the learning curve is large, so it seems to be within the error
range11 . We can see that in English-French translation, the scores of reducedstruct-freq and struct-freq based on parse trees grows more rapidly than othermethods and is significantly better even at the point of 1M additional words.
Besides in English-Japanese translation, the scores of these methods do not havemuch diﬀerence with reduced-maxsubst-freq and 4gram-freq while the numberof additional words is small, but become higher than the other methods fromthe point of adding about 40k words. From the point of adding 500k words in
English-Japanese translation, the score of each selection method becomes almost
flat.

11

When we tested the statistically significant diﬀerence by bootstrap resampling method (Koehn,
2004), no significant diﬀerence of p < 0.1 was observed.

73

Length of selected phrases: Due to the diﬀerent criteria used by each method,
there are also significant diﬀerences in the features of the selected phrases. In
Table 6.3, we show the details of the number of all selected phrases, words andaverage phrase length until the stop condition, and at the point of 10k additionalsource words. The coverage converges when all the selected segments are translated, and thus the less words the method selects finally, the more rapidly thecoverage converges and the translation accuracy may also improves. Whereas,since longer phrase can cover more n-gram phrases at once, methods selecting
longer segments on the average have an advantage for improving 4-gram coverage. We see there is a big diﬀerence of average phrase length between 5.30-6.68
in English-French and 3.03-3.58 in English-Japanese, but this depends only on
the combination of source-side base and additional datasets and of course does
not depend on language pairs. Moreover, we can confirm that the average phrase
length at the point of 10k additional words in Table 6.3 is shorter than adding all
the candidates especially in methods based on parse trees. That is clear because
shorter phrases tend to be more frequent and selected earlier, still we see thetendency that the frequency of longer syntactic phrases drastically decreases.
Here we see the tendency that the selection methods based on parse trees selectshorter phrases than other methods. This is caused by the fact that longer phrases
are only counted if they cover a syntactically defined phrases, and thus longer
substrings that do not form syntactic phrases are removed from consideration.
Eﬀect on coverage: This diﬀerence in the features of the selected phrases also
aﬀects how well they can cover new incoming test data. To demonstrate this,in Table 6.4 we show the 1-gram and 4-gram coverage of the test dataset after
10k, 100k and 1M words have been selected. From the results, we can see thatthe reduced-struct-freq method attains the highest 1-gram coverage, eﬃciently
covering unknown words. On the other hand, it is clear that methods selecting
longer phrases have an advantage for 4-gram coverage, and we see the highest 4gram coverage in the sent-by-4gram-freq method. In English-French translation,
there is no change in the top four digits of 4-gram coverage at the time of adding
10k words. Although, whether to select a long segment or a short segment causes
a trade-oﬀ relationship considering the eﬀect on coverage, it was confirmed that
1-gram coverage and 4-gram coverage can be jointly improved by eliminating

74

Lang Pair

Selection Method

En-Fr

sent-rand
4gram-rand
sent-by-4gram-freq
4gram-freq
maxsubst-freq
reduced-maxsubst-freq
struct-freq
reduced-struct-freq

En-Ja

sent-rand
4gram-rand
sent-by-4gram-freq
4gram-freq
maxsubst-freq
reduced-maxsubst-freq
struct-freq
reduced-struct-freq

No Addition

1-gram / 4-gram Coverage [%]10k Words
100k Words

1M Words

92.72 / 10.60

92.93 / 10.60
92.95 / 10.60
92.95 / 10.60
92.92 / 10.60
92.79 / 10.60
92.92 / 10.60
93.63 / 10.60
94.02 / 10.60

93.73 / 10.71
93.99 / 10.60
93.96 / 10.72
94.46 / 10.66
93.61 / 10.62
94.38 / 10.66
96.15 / 10.65
96.38 / 10.69

95.94 / 11.30
96.42 / 10.64
96.25 / 11.55
96.60 / 11.16
95.99 / 10.92
96.55 / 11.13
97.84 / 11.28
98.00 / 11.38

94.36 / 5.38

94.81 / 5.63
94.80 / 5.38
95.10 / 5.84
95.64 / 5.97
95.59 / 5.96
95.73 / 6.00
96.60 / 5.44
96.64 / 5.44

95.99 / 6.59
96.10 / 5.46
96.28 / 7.23
96.87 / 7.14
96.83 / 7.07
96.97 / 7.19
97.80 / 5.79
97.84 / 5.80

97.54 / 10.06
97.67 / 5.98
97.64 / 11.39
97.97 / 10.43
97.91 / 10.20
98.00/10.57
98.58 / 7.02
98.61 / 7.14

Table 6.4: Eﬀect on coverage in each selection method (rounded oﬀ to the second decimal place). Bold face indicates the highest coverage for eachnumber of additional words.

duplication based on λ-maximality.
Reduction eﬀect: In Section 3.2.3, as a matter of 4gram-freq, we mentionedthat there are many common word sub-sequences appearing duplicated among
selected segments, and to deal with this problem, we proposed selection method
based on λ-maximality in Section 6.1.1. Table 6.5 summarizes the number andpercentage of words which are selected by 4gram-freq and reduced by combining
into longer segments with maxsubst-freq and reduced-struct-freq at the time ofadding 10k words, 100k words and 1M words. From the table, it can be seenthat in both language pairs, maxsubst-freq can reduce only a small amount of
1%-4%. As stated in Section 6.1.1, it is because standard phrase maximality has
severe restrictions that the occurrence counts of segments in inclusion relation areequivalent, and thus many included word sub-sequences become maximal phrases.
On the other hand, in both language pairs, reduced-maxsubst-freq reduce words
by at least 24.70% up to 50.79%. From this results, it can be said that it ispossible to eﬀectively reduce the number of included phrases by relaxing the75

Number of Reduced Words (Reduction Ratio)
10k Add. Words 100k Add. Words
1M Add. Words

Lang Pair

Selection Method

En-Fr

maxsubst-freq
reduced-maxsubst-freq

92 (0.92%)
5,079 (50.79%)

2,077 (2.11%)
42,622 (42.62%)

34,917 (3.49%)
378,938 (37.89%)

En-Ja

maxsubst-freq
reduced-maxsubst-freq

138 (1.38%)
2,560 (25.6%)

686 (1.61%)
24,697 (24.70%)

41,046 (4.10%)
24,697 (24.70%)

Table 6.5: Reduction amount of duplicated segments selected in 4gram-freq

Phrase to be translated:
The morphologies using scanning electron
microscopy ( SEM ) were studied .
Translation input form:
⾛査型電⼦顕微鏡 (SEM)
Conﬁdence level:
3: sure about the translation
2: not so sure about the translation
1: not sure at all

Figure 6.4: Example of the human translation interface
matching condition of occurrence counts of word sequences.

6.3 Manual Translation Experiment6.3.1 Experimental Set-Up
To confirm that the results from the simulation in the previous section carry overto actual translators, we further performed experiments in which professionaltranslators translated the selected segments. This also allowed us to examine theactual amount of time required to perform translation, and how confident thetranslators were in their translations.
We designed a web user interface as shown in Figure 6.4, and outsourced toan external organization that had three professional translators translate theshown phrases. As is standard when hiring translators, we paid a fixed priceper word translated for all of the methods. Because showing only the candidate phrase out of context could cause diﬃculty in translation, we followed76



%/(86FRUHYV$GGLWLRQDO:RUGV(Q-DVHQWE\JUDPIUHT
JUDPIUHT
UHGXFHGVWUXFWIUHT




%/(86FRUH>@%/(86FRUH>@



















%/(86FRUHYV&XPXODWLYH'XUDWLRQ(Q-DVHQWE\JUDPIUHT
JUDPIUHT
UHGXFHGVWUXFWIUHT





$GGLWLRQDO:RUGV>6RXUFH:RUGV@














&XPXODWLYH'XUDWLRQ>+RXUV@





Figure 6.5: Transition of BLEU score vs. additional source words (left) and vs.
cumulative working duration (right)
Bloodgood and Callison-Burch (2010) in showing a sentence including the selected phrase,12 highlighting the phrase, and requesting to translate the highlighted part. We also requested every worker to select from 3 levels indicating
how confident they were of their translation. In the background, the time required to complete the translation was measured from when the new phrase wasshown until when the translation was submitted.
The methods selected for comparative evaluation are sentence selection basedon 4-gram frequency (sent-by-4gram-freq) and phrase selection based on 4-gramfrequency (4gram-freq) as baseline methods, and the phrase selection based on
both parse trees and λ-maximality (reduced-struct-freq) as the proposed method.
For each method we collected translations of 10k source words, alternating between segments selected by each method to prevent bias.
We used the same dataset as the English-Japanese translation task and thesame tools in the simulation experiment (Section 6.2). However, for training
target language models, we interpolated one trained with the base data and asecond trained with collected data by using SRILM (Stolcke, 2002) because thehand-made data set was too small to train a full language model using only
this data13 . We tuned the interpolation coeﬃcient such that it maximizes the12
13Specifically, we selected the shortest sentence including the phrase in the source corpus.
Let PL (e) be the occurrence probability of sentence e in language model L, then the occurrence77Selection Method

Total Working Time Average Confidence Level

Percentage of Confidence Level[Hours](3 Levels)

Skipped Level 1 Level 2 Level 3

sent-by-4gram-freq

25.22

2.689

1.77%

0.00% 30.74% 69.08%

4gram-freq

32.70

2.601

0.53%

1.69% 35.48% 62.29%

reduced-struct-freq

59.97

2.771

0.52%

1.51% 18.82% 79.15%

Table 6.6: Total working time and statistics of confidence level evaluation

perplexity for the tuning dataset.

6.3.2 Results and Discussion
Eﬃciency results: Figure 6.5 shows the evaluation scores of SMT systems
trained using varying amounts of collected phrases. In the left graph, we see theproposed method based on parse trees and phrase λ-maximality rapidly improves
BLEU score, and requires fewer additional words than the conventional methods.
Because the cost paid for translation often is decided by the number of words,
this indicates that the proposed method has better cost performance in thesesituations. The right graph shows improvement by the amount of translation
time. These results here are diﬀerent, showing the 4-gram-freq baseline slightly
superior. As discussed in Table 6.4, the methods based on parse trees selectmore uncovered 1-grams, namely unknown words, and specifically the proposed
method selected more technical terms that took a longer time to translate.
Working time and confidence: We show the total time to collect the translations of 10k source words and statistics of confidence level for each method in
Table 6.6. The total working time for the proposed method is nearly double thatof other methods, and the tendency to focus on selecting uncovered technicalterms as described above can been confirmed. This tendency is seen also bythat the number of selected single words is nearly four times that of 4gram-freq,
probability in interpolated language model L1+2 from L1 and L2 is PL1+2 (e) = αPL1 (e) + (1 −α)PL2 (e). Interpolation coeﬃcient α takes a range from 0 to
1, and is adjusted with(∑) respect tothe development data Edev to minimize P P L(Edev ) = exp

78e∈Edev−logPL1+2 (e)|Edev |.

Number of Segments
Selection Method 1 Word 2 Words 3 Words 4 Words 5+ Words Total
sent-by-4gram-freq

-

-

-

-

565

566

4gram-freq

1,185

2,061

1,045

390

0

4,681

reduced-struct-freq

4,688

1,038

884

96

38

6,744

Table 6.7: Detail of selected segments by length

Average Working Time [Seconds]
Selection Method

1 Word2 Words

3 Words

4 Words

5+ Words

sent-by-4gram-freq

-

-

-

-

160.644gram-freq

30.14

24.76

21.77

21.12

-

reduced-struct-freq

35.61

25.23

21.72

28.13

22.82

Table 6.8: Average working time of manual translation corresponding to segment
lengthand the total number of selected segments is also large, as shown in Table 6.7.
On the other hand, the segments selected by the proposed method were given
the highest confidence level, receiving the maximum value of 3 for about 79% ofphrase pairs, indicating that the generated parallel data is of high quality. To
some extent, this corroborates our hypothesis that the more syntactic phrases
selected by the proposed method are easier to translate.
We can also examine the tendency of working time for segments of diﬀerent
lengths in Table 6.8. Interestingly, single words consistently have a longer averagetranslation time than phrases of length 2-4, likely because they tend to be technical terms that require looking up in a dictionary. In addition, since these are
the average time required for translation of single segment, substantial averagetime cost for single word translation is more than double the word translation
time of segments with 2 words, we can see how expensive the cost of translating
technical terms.
We show the average confidence levels corresponding to phrase length in Table
6.9. The confidence level of single words in the proposed method is lower thanin the baseline method, likely because the baseline selects a smaller amount of79

Selection Method

1 Wordsent-by-4gram-freq
4gram-freq
reduced-struct-freq

2.885
2.802

Average Confidence Level (3 Levels)
2 Words 3 Words 4 Words 5+ Words
2.585
2.796

2.422
2.778

2.300
2.708

2.689
2.737

Table 6.9: Average confidence level of manual translation corresponding to phrase
lengthBLEU Score [%]Selection Methods

Confidence

Confidence

Confidence

1+ (All)2+

3

sent-by-4gram-freq

9.88

9.92

9.85

4gram-freq

10.48

10.54

10.36

reduced-struct-freq

10.70

10.72

10.67

Table 6.10: BLEU score when training on phrases with a certain confidence level

single words, and those selected are less likely to be technical terms. On the other
hand, we can confirm that the confidence level for longer phrases in the baseline
method decreases drastically, while it is stably high in our method, confirming
the eﬀectiveness of selecting syntactically coherent phrases.
Translation accuracy by confidence level: Finally, we show the accuracyof the SMT system trained by all the collected data in each method in Table
6.10. To utilize the confidence level annotation, we tested SMT systems trainedby phrase pairs with confidence levels higher than 2 or 3. From the results, theaccuracy of every method is improved when phrases pairs with confidence level 1
were filtered out. In contrast, the accuracy is conversely degraded if we use only
phrase pairs with confidence level 3. The translation accuracy of 9.37% BLEU
with the base SMT system without additional data became 10.72% after adding
phrase pairs having confidence level 2 or higher, allowing for a relatively large
gain of 1.35 BLEU points.

80

6.4 Summary
In this chapter, we proposed a new method for active learning in machine translation that selects syntactic, non-redundant phrases using parse trees and λmaximal phrases. We first performed simulation experiments and obtained improvements in translation accuracy with fewer additional words. Further manualtranslation experiments also demonstrated that our method allows for greater
improvements in accuracy and translator confidence.

81

7 Conclusion
7.1 Contribution
In SMT and NMT, it has been observed that translation with models trainedon larger parallel corpora can achieve higher accuracy, and usually millions ofsentence pairs are required in order to produce a high quality translation system.
Unfortunately, readily available parallel corpora are limited for most language
pairs, particularly those that do not include English, having few sentence pairs,
or none at all.
In this thesis, we focused on the low-resource data scenario for MT, in which
the size of the bilingual corpus is known to be limited. Specifically, our methods
addressed to improve MT quality with two types of common approaches for coping with the scarceness of bilingual corpus: (1) pivot translation and (2) active
learning for MT. Indeed, as we have demonstrated, MT quality could significantly
benefit from syntactic and contextual information when faced with limited training data.
Contextual Disambiguation with Pivot-Side Language Models:
In Chapter 4, we proposed a new method in pivot translation to resolve pivotside contextual ambiguity. This proposed method lets MT models remember the
information of the pivot phrase. This information can help to select appropriate translation rules considering pivot-side context with pivot language models.
Experimental results on multilingual translation showed significant improvementof MT evaluation scores for all the tested language pairs with relatively small
indirectly parallel corpora, of 100k sentence pairs, and large English monolingual
corpus as an additional resource. This method is eﬀective in the case that available source-pivot and pivot-target parallel corpora are not large and while theamount of available pivot monolingual corpus is large.

82

Syntactic Disambiguation with Pivot-Side Parse Trees:
In Chapter 5, we proposed a new method in pivot translation to resolve thepivot-side syntactic ambiguity. This proposed method introduces an explicitly
syntax-aware matching condition to find correct correspondences between sourcepivot and pivot-target translation rules, and can produce more reliable models.
Experimental results on multilingual translation showed significant improvementsof MT evaluation scores for all the tested language pairs with larger indirectly
parallel corpora than Chapter 4, of 1M sentence pairs, and results of English syntactic parsing as an additional resource. A syntactic matching method allowing
partial matching successfully reduced the number of noisy translation rules andimproved the estimation of translation probabilities causing better translation
accuracy. This method is eﬀective in the case that accurate syntactic parsers forthe pivot language are available, and practical to use for pivot translation withlarger amounts of parallel corpora.
Cost Reduction and Quality Improvement in Human Translation:
In Chapter 6, we proposed a new method in active learning for SMT to introduce new criteria for segment selection, based on non-redundancy and syntactic
coherence. This proposed method provides a more compact and human-friendly
annotation task than conventional methods, resulting in a higher quality parallel corpus with lower annotation cost. Experimental results on multilingualtranslation showed a significant improvement in all the tested language pairs.
Experiments using both simulation and extensive manual translation by professional translators find the proposed method eﬀective, achieving both greater gainof translation score for the same number of translated words, and allowing translators to be more confident in their translations.

7.2 Future Directions
Here, we list directions for future work in low-resource MT.
Pivot Translation Preserving Linguistic Information:
The proposed method in pivot translation in Chapter 4 uses MSCFG models
that have potential to various information of source, target, and pivot languages.

83For example, we should be able to combine the proposed methods in Chapters
4-5 and let to MSCFG model to remember the pivot tree structures.
As a more advanced method, it should be possible to devise compounded
MSCFG models that can store not only pivot-side syntactic information but also
source-side syntactic information, thereby realizing translation with higher reproducibility of source information. We mentioned that pivot translation has the
problem of losing source language information, aﬀected by the expressiveness ofpivot language. In fact, this problem often occurs not only for MT, but also forhuman translators. For example, since modern English is known for its simple
morphology which has no complicated grammatical conjugation such as personal
suﬃxes, linguistic modality such as number, case, gender, etc. This information
is lost when translating into English, resulting that translation from English intoanother language is diﬀerent from the original meaning. In this method, we aim toachieve translation that preserves originally linguistic information by combining
with pivot-side syntactic structures.
Active Learning for Pivot MT:
In many cases, pivot translation approach may solve the scarceness problemof bilingual corpora for many language pairs that have suﬃcient amount of indirectly parallel corpus with English. However, pivot translation alone has poor
performance or no eﬀect at all for truly low-resourced language pairs that have nosuitable candidate to be a pivot with a significantly large parallel corpus. In sucha case, an active learning approach should have a synergy with pivot translation,and be able to be used to eﬃciently supplement a shortage in data that can beused in pivot translation. Therefore, it is important to combine the approachesof pivot translation and active learning to provide a realistic solution in lowresource scenarios. It is natural that source-pivot, pivot-target and source-target
language pairs have diﬀerent annotation costs thereby optimization will become
more complicated. Tackling this is an interesting problem.
Toward Multilinguality and Multimodality:
In recent years, research on NMT to utilize multiple available training datasuch as multi-source NMT (Zoph and Knight, 2016) and multimodal NMT (Specia et al., 2016) has become active and diversified. The previous work has demon-84strated the capability of multi-layer NNs to jointly train shared models. One final
challenge for the future is to design an ecosystem of NNs by combining pivot MT
and active learning with multi-source NMT. As mentioned previously, even today,
many language pairs do not have a suﬃcient amount of parallel data, though thisthesis demonstrates that we can eﬃciently grow the parallel corpus by applying
active learning methods. Of course, the obtained parallel corpus can be used fortraining a single MT model or applying pivot translation in a straightforwardmanner. However, NMT models should be capable of reusing human annotation
results as an additional source, and reuse even their own output. In this idea,
the model will be able to train itself by feeding not only a regular parallel corpus,
also from human translation results, output of other MT systems or of itself,
and other possible resources such as diﬀerent kind of human annotation or otherlanguage data. By applying this idea, the model can assist a human translator insuch a manner of post-editing, and immediately adapt itself with the result, andeven will be capable of automatic post-editing, resulting a substantial reductionof translation cost.

85

Bibliography
Sankaranarayanan Ananthakrishnan, Rohit Prasad, David Stallard, and Prem
Natarajan. A Semi-Supervised Batch-Mode Active Learning Strategy for Improved Statistical Machine Translation. In Proc. CoNLL, pages 126–134, 2010a.
Sankaranarayanan Ananthakrishnan, Rohit Prasad, David Stallard, and Prem
Natarajan. Discriminative Sample Selection for Statistical Machine Translation. In Proc. EMNLP, pages 626–635, 2010b.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Proc. ICLR, pages 1–15,
2015.
Michael Bloodgood and Chris Callison-Burch. Bucking the Trend: Large-Scale
Cost-Focused Active Learning for Statistical Machine Translation. In Proc.
ACL, pages 854–864, 2010.
Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeﬀrey Dean.
Large language models in machine translation. In Proc. EMNLP, pages 858–
867, 2007.
Peter F. Brown, Vincent J.Della Pietra, Stephen A. Della Pietra, and Robert L.
Mercer. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19:263–312, 1993.
Jean-Cédric Chappelier, Martin Rajman, et al. A Generalized CYK Algorithm for
Parsing Stochastic CFG. In Proc. TAPD, volume 98, pages 133–137. Citeseer,
1998.
Stanley F. Chen and Joshua Goodman. An Empirical Study of Smoothing Techniques for Language Modeling. In Proc. ACL, pages 310–318, 1996.

86

David Chiang. Hierarchical Phrase-Based Translation. Computational Linguistics, 33(2):201–228, 2007.
Trevor Cohn and Mirella Lapata. Machine Translation by Triangulation: Making
Eﬀective Use of Multi-Parallel Corpora. In Proc. ACL, pages 728–735, 2007.
Raj Dabre, Fabien Cromieres, Sadao Kurohashi, and Pushpak Bhattacharyya.
Leveraging Small Multilingual Corpora for SMT Using Many Pivot Languages.
In Proc. NAACL, pages 1192–1202, 2015.
Adrià de Gispert and José B. Mariño. Catalan-English Statistical Machine Translation without Parallel Corpus: Bridging through Spanish. In Proc. of LREC
5th Workshop on Strategies for developing machine translation for minority
languages, pages 65–68, 2006.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-Task
Learning for Multiple Language Translation. In Proc. ACL, pages 1723–1732,
2015.
Chris Dyer, Aaron Cordova, Alex Mont, and Jimmy Lin. Fast, Easy, and Cheap:
Construction of Statistical Machine Translation Models with MapReduce. In
Proc. WMT, pages 199–207, 2008.
Matthias Eck, Stephan Vogel, and Alex Waibel. Low Cost Portability for Statistical Machine Translation based in N-gram Frequency and TF-IDF. In Proc.
IWSLT, pages 61–67, 2005.
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural,and Kyunghyun Cho. Zero-Resource Translation with Multi-Lingual Neural
Machine Translation. In Proc. EMNLP, pages 268–277, 2016.
Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. What’s in a
Translation Rule? In Proc. NAACL, pages 273–280, 2004.
Ulrich Germann. Building a statistical machine translation system from scratch:
how much bang for the buck can we expect? In Proc. of the workshop on
Data-driven methods in machine translation-Volume 14, pages 1–8, 2001.

87

Ulrich Germann. Dynamic phrase tables for machine translation in an interactive post-editing scenario. In Proc. AMTA 2014 Workshop on Interactive andAdaptive Machine Translation, pages 20–31, 2014.
Jesús González-Rubio, Daniel Ortiz-Martínez, and Francisco Casacuberta. Active
learning for interactive machine translation. In Proc. EACL, pages 245–254,
2012.
Isao Goto, Masao Utiyama, Eiichiro Sumita, Akihiro Tamura, and Sadao Kurohashi. Distortion Model Considering Rich Context for Statistical MachineTranslation. In Proc. ACL, pages 155–165, 2013.
Jonathan Graehl and Kevin Knight. Training Tree Transducers. In Proc. NAACL,
pages 105–112, 2004.
Spence Green, Sida I. Wang, Jason Chuang, Jeﬀrey Heer, Sebastian Schuster, andChristopher D. Manning. Human Eﬀort and Machine Learnability in Computer
Aided Translation. In Proc. EMNLP, pages 1225–1236, 2014.
Gholamreza Haﬀari and Anoop Sarkar. Active Learning for Multilingual Statistical Machine Translation. In Proc. ACL, pages 181–189, 2009.
Gholamreza Haﬀari, Maxim Roy, and Anoop Sarkar. Active Learning for Statistical Phrase-based Machine Translation. In Proc. NAACL, pages 415–423,
2009.
Kenneth Heafield. KenLM: Faster and Smaller Language Model Queries. In Proc,
WMT, pages 187–197, 2011.
Eduard Hovy. Combining and Standardizing Large-Scale, Practical Ontologies
for Machine Translation and Other Uses. In Proc. LREC, pages 535–542, 1998.
Ann Irvine and Chris Callison-Burch. Combining Bilingual and Comparable Corpora for Low Resource Machine Translation. In Proc. of the Eighth Workshop
on Statistical Machine Translation, pages 262–270, August 2013.
Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng
Chen, Nikhil Thorat, Fernand a Vi ﾃｩ gas, Martin Wattenberg, Greg Corrado,88Macduﬀ Hughes, and Jeﬀrey Dean. Google’s Multilingual Neural MachineTranslation System: Enabling Zero-Shot Translation. TACL, 5:339–351, 2017.
Toru Kasai, Gunho Lee, Hiroki Arimura, Setsuo Arikawa, and Kunsoo Park.
Linear-Time Longest-Common-Prefix Computation in Suﬃx Arrays and Its
Applications. In Proc. CPM, pages 181–192, 2001.
Philip N. Klein. Computing the Edit-Distance Between Unrooted Ordered Trees.
In Proc. of European Symposium on Algorithms, pages 91–102, 1998.
Philipp Koehn. Statistical Significance Tests for Machine Translation Evaluation.
In Dekang Lin and Dekai Wu, editors, Proc. EMNLP, pages 388–395, July 2004.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation.
In MT summit, volume 5, pages 79–86, 2005.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, MarcelloFederico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan
Herbst. Moses: Open Source Toolkit for Statistical Machine Translation. In
Proc. ACL, pages 177–180, 2007.
Phillip Koehn, Franz Josef Och, and Daniel Marcu. Statistical Phrase-BasedTranslation. In Proc. NAACL, pages 48–54, 2003.
Tomer Levinboim and David Chiang. Supervised Phrase Table Triangulation
with Neural Word Embeddings for Low-Resource Languages. In Proc. EMNLP,
pages 1079–1083, 2015.
Adam Lopez and Matt Post. Beyond bitext: Five open problems in machinetranslation. In Proc. of the EMNLP Workshop on Twenty Years of Bitext,
2013.
Thang Luong, Hieu Pham, and Christopher D. Manning. Eﬀective Approaches toAttention-based Neural Machine Translation. In Proc. EMNLP, pages 1412–
1421, 2015.

89

Tomas Mikolov, Wen-tau Yih, and Geoﬀrey Zweig. Linguistic Regularities inContinuous Space Word Representations. In Proc. NAACL, pages 746–751,
2013.
Akiva Miura, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Improving Pivot Translation by Remembering the Pivot. In Proc. ACL,
pages 573–577, 2015.
Akiva Miura, Graham Neubig, Michael Paul, and Satoshi Nakamura. Selecting
Syntactic, Non-redundant Segments in Active Learning for Machine Translation. In Proc. NAACL, pages 20–29, 2016.
Akiva Miura, Graham Neubig, Katsuhito Sudoh, and Satoshi Nakamura. Tree
as a Pivot: Syntactic Matching Methods in Pivot Translation. In Proc. WMT,
pages 90–98, September 2017.
Robert Munro. Crowdsourcing and the crisis-aﬀected community. Information
Retrieval, 16(2):210–266, 2013.
Makoto Nagao. A framework of a Mechanical Translation between Japanese andEnglish by Analogy Principle. In Proc. International NATO Symposium on
Artificial and Human Intelligence, pages 173–180, 1984. ISBN 0-444-86545-4.
Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. ASPEC: Asian ScientificPaper Excerpt Corpus. In Proc. LREC, pages 2204–2208, 2016.
Graham Neubig. Travatar: A Forest-to-String Machine Translation Engine basedon Tree Transducers. In Proc. ACL Demo Track, pages 91–96, 2013.
Graham Neubig, Yosuke Nakata, and Shinsuke Mori. Pointwise Prediction for
Robust, Adaptable Japanese Morphological Analysis. In Proc. ACL, pages
529–533, 2011.
Graham Neubig, Philip Arthur, and Kevin Duh. Multi-Target Machine Translation with Multi-Synchronous Context-free Grammars. In Proc. NAACL, pages
484–491, 2015.

90

Sergei Nirenburg. Knowledge-based machine translation. Machine Translation,
4(1):5–24, 1989.
Franz Josef Och. Minimum Error Rate Training in Statistical Machine Translation. In Proc. ACL, pages 160–167, 2003.
Franz Josef Och and Hermann Ney. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19–51, 2003.
Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Ckylark: A More Robust PCFG-LA Parser. In Proc. NAACL, pages
41–45, 2015.
Daisuke Okanohara and Jun’ichi Tsujii. Text Categorization with All Substring
Features. In Proc. SDM, pages 838–846, 2009.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a
Method for Automatic Evaluation of Machine Translation. In Proc. ACL, pages
311–318, 2002.
Michael Paul, Hirofumi Yamamoto, Eiichiro Sumita, and Satoshi Nakamura. On
the Importance of Pivot Language Selection for Statistical Machine Translation. In Proc. NAACL, pages 221–224, 2009.
Philip Resnik and Noah A Smith. The web as a parallel corpus. Computational
Linguistics, 29(3):349–380, 2003.
Burr Settles and Mark Craven. An Analysis of Active Learning Strategies for
Sequence Labeling Tasks. In Proc. EMNLP, pages 1070–1079, 2008.
Claude E. Shannon. A Mathematical Theory of Communication. Bell System
Technical Journal, 27(3):379–423, 1948.
Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. A Shared Task
on Multimodal Machine Translation and Crosslingual Image Description. In
WMT, pages 543–553, 2016.

91

Matthias Sperber, Mirjam Simantzik, Graham Neubig, Satoshi Nakamura, andAlex Waibel. Segmentation for Eﬃcient Supervised Language Annotation withan Explicit Cost-Utility Tradeoﬀ. TACL, 2:169–180, 2014.
Andreas Stolcke. SRILM - an extensible language modeling toolkit. In Proc.
ICSLP, pages 901–904, 2002.
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to Sequence Learningwith Neural Networks. In Advances in Neural Information Processing Systems,
pages 3104–3112, 2014.
Jörg Tiedemann. News from OPUS-A collection of multilingual parallel corpora
with tools and interfaces. In Proc. RANLP, volume 5, pages 237–248, 2009.
Katrin Tomanek and Udo Hahn. Semi-Supervised Active Learning for Sequence
Labeling. In Proc. ACL, pages 1039–1047, 2009.
Marco Turchi, Tijl De Bie, and Nello Cristianini. Learning performance of amachine translation system: a statistical and computational analysis. In Proc.
WMT, pages 35–43, 2008.
Masao Utiyama and Hitoshi Isahara. A Comparison of Pivot Methods for PhraseBased Statistical Machine Translation. In Proc. NAACL, pages 484–491, 2007.
Bernard Vauquois. A Survey of Formal Grammars and Algorithms for Recognition and Transformation in Mechanical Translation. In Proc. IFIP Congress(2), volume 68, pages 1114–1122, 1968.
Katharina Wäschle and Stefan Riezler. Analyzing Parallelism and Domain Similarities in the MAREC Patent Corpus. Multidisciplinary Information Retrieval,
pages 12–27, 2012.
Mikio Yamamoto and Kenneth Ward Church. Using Suﬃx Arrays to Compute
Term Frequency and Document Frequency for All Substrings in a Corpus.
Computational Linguistics, 27(1):1–30, 2001.
Omar F Zaidan and Chris Callison-Burch. Crowdsourcing translation: Professional quality from non-professionals. In Proc. ACL, pages 1220–1229, 2011.

92

Xiaoning Zhu, Zhongjun He, Hua Wu, Conghui Zhu, Haifeng Wang, and Tiejun
Zhao. Improving Pivot-Based Statistical Machine Translation by Pivoting theCo-occurrence Count of Phrase Pairs. In Proc. EMNLP, pages 1665–1675,
2014.
Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The United
Nations Parallel Corpus v1.0. In Proc. LREC, pages 3530–3534, 2016.
Barret Zoph and Kevin Knight. Multi-Source Neural Translation. In Proc.
NAACL, pages 30–34, June 2016.

93

Publication List
Refereed Domestic Journal Papers
1. 三浦明波, Graham Neubig, Sakriani Sakti, ⼾⽥智基, 中村哲. 中間⾔語情報を記憶するピボット翻訳⼿法. ⾃然⾔語処理, Vol.23, No.5, pp499-528,
December 2016.
2. 三浦明波, Graham Neubig, Michael Paul, 中村哲. 統語的⼀貫性と⾮冗⻑性を重視した機械翻訳のための能動学習⼿法. ⾃然⾔語処理, Vol.24
No.3, pp463-489, June 2017.

Refereed International Conference Papers
1. Akiva Miura, Graham Neubig, Sakriani Sakti, Tomoki Toda, Satoshi Nakamura. Improving Pivot Translation by Remembering the Pivot. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), July 2015.
2. Akiva Miura, Graham Neubig, Michael Paul, Satoshi Nakamura. Selecting
Syntactic, Non-redundant Segments in Active Learning for Machine Translation. Proceedings of the 15th Annual Meeting of the North American
Chapter of the Association for Computational Linguistics (NAACL), June
2016.
3. Akiva Miura, Graham Neubig, Katsuhito Sudoh, Satoshi Nakamura. Tree
as a Pivot: Syntactic Matching Methods in Pivot Translation. Proceedingsof the Second Conference on Machine Translation (WMT), September 2017.

94International Workshop Papers
1. Raphael Shu, Akiva Miura. Residual Stacking of RNNs for Neural MachineTranslation. Proceedings of the 3rd Workshop on Asian Translation (WAT),
December 2016.

Domestic Conference Papers
1. 三浦明波, Graham Neubig, Sakriani Sakti, ⼾⽥智基, 中村哲. 階層的フレーズベース翻訳におけるピボット翻訳⼿法の応⽤. 情報処理学会第 219
回⾃然⾔語処理研究会 (SIG-NL), December 2014.
2. 三浦明波, Graham Neubig, Sakriani Sakti, ⼾⽥智基, 中村哲. 中間⾔語モデルを⽤いたピボット翻訳の精度向上. 情報処理学会第 222 回⾃然⾔語処理研究会 (SIG-NL), July 2015.
3. 三浦明波, Graham Neubig, Michael Paul, 中村哲. 構⽂⽊と句の極⼤性に基づく機械翻訳のための能動学習. 情報処理学会第 224 回⾃然⾔語処理研究会 (SIG-NL), December 2015.
4. 三浦明波, Graham Neubig, Michael Paul, 中村哲. 構⽂情報に基づく機械翻訳のための能動学習⼿法と⼈⼿翻訳による評価. ⾔語処理学会第 22回年次⼤会 (NLP2016), March 2016.
5. 三浦明波, Graham Neubig, 中村哲. ⽊構造を中間表現とするピボット翻訳⼿法. 情報処理学会第 227 回⾃然⾔語処理研究会 (SIG-NL), July 2016.

Awards
1. Asia-Pacific Association for Machine Translation (AAMT), Nagao Award
Student Award, June 2016.
2. NAIST Top Scholarship Program for Excellent Academic Standing, July
2016.

Master’s Thesis
1. 三浦明波. 中間⾔語モデルを⽤いた多⾔語機械翻訳の精度向上. Master’s
thesis, Graduate School of Information Science, Nara Institute of Scienceand Technology, March 2016.

95