Paper

Syntactic Matching Methods in Pivot Translation
Akiva Miura† , Graham Neubig†,†† , Katsuhito Sudoh† and Satoshi Nakamura†
The pivot translation is useful method for translating between languages that containlittle or no parallel data by utilizing equivalents in an intermediate language suchas English. Commonly, phrase-based or tree-based pivot translation methods merge
source–pivot and pivot–target translation models into a source–target model. Thistactic is known as triangulation. However, the combination is based on the surface
forms of constituent words, and it often produces incorrect source–target phrase pairs
because of interlingual diﬀerences and semantic ambiguities in the pivot language.
The translation accuracy is thus degraded. This paper proposes a triangulation approach that utilizes syntactic subtrees in the pivot language to avoid incorrect phrase
combinations by distinguishing pivot language words by their syntactic roles. The
results of the experiments conducted on the United Nations Parallel Corpus demonstrate that the proposed method is superior to other pivot translation approaches inall tested combinations of languages.
Key Words: Pivot Translation, Machine Translation, Parallel Corpus, Low-Resourced
Language Pairs, Syntactic Analysis

1 Introduction
Translation eﬀected using models trained on larger parallel corpora can achieve greater accuracy (Dyer, Cordova, Mont, and Lin 2008) in statistical machine translation (SMT) (Brown,
Pietra, Pietra, and Mercer 1993). Unfortunately, most language pairs are restricted in termsof readily available parallel corpora: some have fewer than 100 k sentence pairs; others do notcontain any. This paucity is especially true of language pairs that do not include English, andthe problem is diﬃcult to overcome because it would cost millions of dollars to manually produce
a high-quality parallel corpus.
One eﬀective solution to surmount the scarceness of bilingual data is to introduce a pivot
language that contains existing parallel data with respect to both the source and target languages(de Gispert and Mariño 2006). The triangulation is both popular and eﬀective among the various
methods that employ pivot languages (Utiyama and Isahara 2007; Cohn and Lapata 2007). This† Graduate School of Information Science, Nara Institute of Science and Technology†† Language Technologies Institute, Carnegie Mellon University

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

process ﬁrst combines source–pivot and pivot–target translation models (TMs) into a source–
target model and then translates data using this merged model. The procedure of triangulating
two TMs into one has been examined for diﬀerent frameworks of SMT and its eﬀectiveness has
been conﬁrmed both for Phrase-Based SMT (PBMT) (Koehn, Och, and Marcu 2003; Utiyama
and Isahara 2007) and for Hierarchical Phrase-Based SMT (Hiero) (Chiang 2007; Miura, Neubig,
Sakti, Toda, and Nakamura 2015). However, word interlingual diﬀerences in word usage andword sense ambiguity cause diﬃculties in accurately learning the correspondences between thesource and target phrases. Thus, the accuracy of triangulated models is lesser than the precisionattained by models trained on direct parallel corpora.
In the triangulation method, source–pivot and pivot–target phrase pairs are connected assource–target pairs if a common pivot-side phrase is available. Fig. 1(a) illustrates a sample
standard triangulation on the Hiero TM, which combines the hierarchical rules of phrase pairs
by matching pivot phrases with equivalent surface forms. This example also demonstrates the
problems of ambiguity: the English word “record” can correspond to several diﬀerent parts-ofspeech according to the context. More broadly, phrases that include this word can also potentiallytake diﬀerent grammatical structures, but it is impossible to uniquely identify these constructions

Fig. 1 Example of disambiguation by parse subtree matching (Fr-En-Zh), [X1] and [X2] are nonterminals for sub-phrases.

600

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

unless information is provided with regard to the surrounding context.
This varying syntactic structure will inﬂuences translation. For example, the French verb
“enregistrer” corresponds to the English verb “record”. At the same time, the French noun
“dossier” matches the noun form of the English word “record”. In a more extreme instance,
Chinese does not incorporate inﬂections depending on the part-of-speech of the word. Thus,
although the word order changes, the Chinese term “记录” is used even in contexts where record
is employed as a diﬀerent grammatical category. These speciﬁcs might result in the incorrectconnection of “[X1] enregistrer [X2]” and “[X2] [X1] 记录” even though the proper correspondenceof “[X1] enregistrer [X2]” and “[X1] dossier [X2]” would be “[X1] 记录 [X2]” and “[X2] [X1] 记录” respectively. Hence, a superﬁcial phrase matching method based solely on the surface formof the pivot often combines incorrect phrase pairs, causing translation errors if the translation
scores of the mached pairs are estimated to be higher than the actual correspondences.
Given this background, it is hypothesized that the disambiguation of these cases would be
simpler if necessary syntactic information such as phrase structures is considered during thepivoting process. To incorporate this intuition into the models introduced in this paper, theauthors propose a method that considers the syntactic information of the pivot phrase as shownin Fig. 1(b). In this manner, the model distinguishes the translation rules extracted in contexts
within which the English symbol string “[X1] record [X2]” behaves as a verbal phrase fromsituations in which the same string acts as a nominal phrase.
Speciﬁcally, the method posited in this paper is based on Synchronous Context-Free Grammars (SCFGs) (Aho and Ullman 1969; Chiang 2007), which are widely used in tree-based machinetranslation frameworks. First, Section 2 of the paper provides a quick review of SCFGs. The
baseline triangulation method that only uses the surface forms for performing the triangulation
is detailed in Section 3, and two methods for triangulation based on syntactic matching are proposed in Section 4. The ﬁrst method places a hard restriction on the exact matching of parsetrees (Section 4.1) included in translation rules, whereas the second places a softer limitation
and allows partial matches (Section 4.2). Experiments of pivot translation on the United Nations Parallel Corpus (Ziemski, Junczys-Dowmunt, and Pouliquen 2016) were performed by theauthors to investigate the proposed method’s impact on pivot translation quality. The resultsof these investigations are presented in Section 5. These ﬁndings demonstrate that the positedprocess indeed provides signiﬁcant gains in accuracy (of up to 2.3 BLEU points), in almost all
tested combinations of ﬁve languages with English used as the pivot language. In addition,as an auxiliary result, the authors compared pivot translations eﬀected through the use of the
proposed method to those made through zero-shot neural machine translation. These outcomes
601

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

conﬁrm that the triangulation of symbolic TMs still signiﬁcantly outperforms neural MT in thezero-resource scenario.1

2 Machine Translation Framework
2.1

Synchronous Context-Free Grammars (SCFGs)

In this section, the authors initially deal with SCFGs, particularly hierarchical phrase-basedtranslation (Hiero) (Chiang 2007), which are widely used in machine translation. The elementary
structures used in translation in SCFGs are synchronous rewrite rules with aligned pairs of source
and target symbols on the right side as in⟨⟩
X → s, t(1)where X is the head symbol of the rewrite rule, and s and t are both strings of terminals andnon-terminals on the source and target sides respectively. Each string in the right-side pair has
the same number of indexed non-terminals, and identically indexed non-terminals correspond toeach-other. A synchronous rule could also, for example, take the form of⟨⟩
X → X0 of X1 , X1 的 X0 .

(2)

Synchronous rules can be extracted based on parallel sentences and automatically obtained
word alignments. Each extracted rule is scored with phrase translation probabilities in both directions ϕ(s|t) and ϕ(t|s), lexical translation probabilities in both directions ϕlex (s|t) and ϕlex (t|s),a word penalty counting the terminals in t, and a constant phrase penalty of 1.
At the time of translation, the decoder searches for the target sentence that maximizes thederivation probability, which is deﬁned as the sum of the scores of the rules used in the derivation,and the log of the language model (LM) probability over the target strings. When not consideringan LM, it is possible to eﬃciently ﬁnd the best translation for an input sentence using the
CKY+ algorithm (Chappelier and Rajman 1998). When using an LM, the expanded search
space is further reduced based on the limit on expanded edges, or total states per span, througha procedure such as cube pruning (Chiang 2007).

1 A preliminary version of this paper has presented in (三浦，Neubig，中村 2016a) and (Miura, Neubig, Sudoh,

and Nakamura 2017).

602

Miura, Neubig, Sudoh and Nakamura

2.2

Syntactic Matching Methods in Pivot Translation

Hierarchical Rules

The rules used in Hiero are speciﬁcally discussed in this section. Hierarchical rules are composed of the initial head symbol S and synchronous rules containing terminals and singular typeof non-terminal X.2 Hierarchical rules are extracted using the same phrase extraction procedure
employed in phrase-based translation (Koehn et al. 2003) based on word alignments, followed bya step that performs a recursive extraction of hierarchical phrases (Chiang 2007).
For example, hierarchical rules could take the form of⟨⟩
X → Oﬃcers, 主席団 成員 ,⟨⟩X → the Committee, 委員会 ,⟨⟩X → X0 of X1 , X1 的 X0 .

(3)
(4)
(5)

From these rules, the input sentence can be translated by the derivation:
S → ⟨X0 , X0 ⟩⟨⟩⇒ X1 of X2 , X2 的 X1⟨⟩
⇒ Oﬃcers of X2 , X2 的 主席団 成員⇒ ⟨Oﬃcers of the Committee,⟩
委員会 的 主席団 成員 .
The advantage of Hiero is that it is able to achieve relatively superior word reordering accuracy(compared to other symbolic SMT alternatives such as standard phrase-based MT) without
language-dependent processing. On the other hand, since it does not use syntactic informationand tries to extract all possible combinations of rules, Hiero tends to extract very large translation
rule tables, and it is also likely to be less syntactically faithful in its derivations.

2.3

Explicitly Syntactic Rules

The use of synchronous context-free grammar or synchronous tree-substitution grammar(Graehl and Knight 2004) rules forms an alternative to the Hiero rules. These options explicitly
take into account the syntax of the source-side (tree-to-string rules), the target-side (string-totree rules), or both (tree-to-tree rules). The tree-to-string (T2S) rules, for example, utilize parsetrees on the source language side, and the head symbols of the synchronous rules are not limited

2 It is also standard to include a glue rule S → ⟨X ,0

X0 ⟩, S → ⟨S0 X1 , S0 X1 ⟩, S → ⟨S0 X1 , X1 S0 ⟩ to fallback on when standard rules cannot result in a proper derivation.

603

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

to S or X, but instead use non-terminal symbols corresponding to the phrase structure tags of agiven parse tree. Thus, T2S rules could take the form of⟨⟩
XNP → (NP (NNS Oﬃcers)), 主席団 成員 ,⟨⟩XNP → (NP (DT the) (NNP Committee)), 委員会 ,⟨⟩XPP → (PP (IN of) XNP,0 ), X0 的 ,⟨⟩
XNP → (NP XNP,0 XPP,1 ), X1 X0 .

(6)
(7)
(8)
(9)

Here, parse subtrees of the source language rules are set in the form of S-expressions. From theserules, the translation can be eﬀected from the parse tree of the input sentence by the derivation:
XROOT

→
⇒
⇒

∗

⇒

⟨
⟨
⟨

⟩
XNP,0 , X0⟩

(NP XNP,1 XPP,2 ), X2 X1(NP (NP (NNS Oﬃcers) XPP,2 )), X2 主席団⟨(NP(NP (NNS Oﬃcers))(PP (IN of), 委員会(NP (DT the)
(NNP Committee))))

的

成員

主席団

⟩

成員

⟩

It is hence possible in T2S translation to obtain a result that conforms to the grammar of thesource language. Also, as an advantage of this method, the number of less-useful synchronous
rules extracted by syntax-agnostic methods such as Hiero is reduced. This decrease makes it
possible to learn more compact rule tables and allows for faster translation.

3 Pivot Translation Methods
Several methods for SMT using pivot languages have been proposed, including cascade methods that consecutively translate from source to pivot then pivot to target (de Gispert and Mariño2006), synthetic methods that machine-translate the training data to generate a pseudo-parallel
corpus (de Gispert and Mariño 2006), and triangulation methods that obtain a source–target
phrase/rule table by merging source–pivot and pivot–target table entries with identical pivot
language phrases (Cohn and Lapata 2007). The triangulation method is particularly notable for
producing higher quality translation results in comparison to other pivot methods (Utiyama andIsahara 2007; Miura et al. 2015), and this approach has thus been employed as the grounding for
the work presented in this paper.

604

Miura, Neubig, Sudoh and Nakamura

3.1

Syntactic Matching Methods in Pivot Translation

Triangulation of TMs

SCFG training procedure stores the extracted and scored phrase pairs from a bilingual corpus
into a structured ﬁle called the rule table. Fig. 2 presents a diagram of the triangulation of thesource–pivot and pivot–target rule tables into the source–target table.
Triangulation for SCFGs searches TSP and TP T for source–pivot and pivot–target rules thathave common pivot symbols and synthesizes these into source–target rules to create rule table
TST :
⟨
⟩
X → s, t ∈ TST⟨ ⟩s.t. X → ⟨s, p⟩ ∈ TSP ∧ X → p, t ∈ TP T .

(10)

Phrase translation probability ϕ(·) and lexical translation probability ϕlex (·) are estimatedfor all combined source–target phrases according to:( )ϕ t|s =
( )ϕ s|t =
( )ϕlex t|s =( )ϕlex s|t =∑p∈TSP ∩TP T∑

p∈TSP ∩TP T∑

p∈TSP ∩TP T∑()
ϕ t|p, s ϕ (p|s)) ( )(ϕ s|p, t ϕ p|t≈p∈TSP ∩TP T≈()
ϕlex t|p, s ϕlex (p|s) ≈()( )ϕlex s|p, t ϕlex p|t ≈p∈TSP ∩TP T∑∑

p∈TSP ∩TP T∑

p∈TSP ∩TP T∑( )ϕ t|p ϕ (p|s) ,

(11)

( )
ϕ (s|p) ϕ p|t ,

(12)

( )
ϕlex t|p ϕlex (p|s) ,

(13)

( )
ϕlex (s|p) ϕlex p|t ,(14)

p∈TSP ∩TP T

where s, p, and t are the phrases in the source, pivot, and target, respectively, and the constructionp ∈ TSP ∩ TP T indicates that p is contained in both phrase tables TSP and TP T . Word penalty⟨⟩⟨⟩
and phrase penalty X → s, t are set as the same values of X → p, t .

Fig. 2 Triangulation of translation models

605

Journal of Natural Language Processing

3.2

Vol. 25 No. 5

December 2018

Problems of Pivot-Side Ambiguity

Although triangulation is known to achieve higher translation accuracy than other simple
methods and although it has become a popular and standard form of pivot translation nowadays,
the problem of ambiguity still remains. This subsection describes the causes of the diﬃculties
and provides pertinent examples.
In triangulation, Equations (11)–(14) are based on the memoryless channel model, whichassumes()( )ϕ t|p, s = ϕ t|p ,()ϕ s|p, t = ϕ (s|p) .

(15)
(16)

In Equation (15), for example, it is presumed that, given the pivot and source phrases, thetranslation probability of the target phrase is not aﬀected by the source phrase. Nonetheless, it
is easy to produce examples where this assumption does not hold.
Fig. 3 exempliﬁes three words in German and Italian. Each of the terms corresponds to thepolysemic English word “approach”. In such a case, it is extremely diﬃcult to ﬁnd associatedsource–target phrase pairs and to estimate the translation probabilities properly. As a result,pivot translation is signiﬁcantly more ambiguous than standard translation.
The authors of this paper have previously proposed a pivot translation method that uses thetriangulation of Synchronous CFG rule tables to a Multi-Synchronous CFG (MSCFG) (Neubig,
Arthur, and Duh 2015) rule table that remembers the pivot as shown in Fig. 4. This method
performs the translation using pivot LMs (Miura et al. 2015), and experiments have establishedFig. 3 An example of ambiguity in De-En-It triangulation.

Fig. 4 An example of the triangulation method remembering the pivot.

606

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translationthat the process is eﬀective in cases when a strong pivot LM exists.
This previously conceived method is eﬀective in instances where the existing source–pivot andpivot–target parallel corpora are not large (i.e., containing less than hundreds of thousands ofsentence pairs) and, conversely, where the available pivot monolingual corpus is sizable. Since
the MSCFG decoder demands an immense quantity of memory and computational time and it
is diﬃcult to accomplish distributed processing, it is not realistic to scale up within the same
framework. Further, although the MSCFG decoder helps in selecting the appropriate translation
rules in a pivot LM, it cannot essentially solve the problem of ambiguity and of inappropriately
connected rules, which remain in the triangulated rule table as noise. This paper attempts toelucidate the manner in which the noisy rules in triangulated TMs can be reduced to bring themcloser to the accuracy attained by directly trained TMs.

4 Triangulation with Syntactic Matching
The previous section outlined the standard triangulation method and marked that the pivotside ambiguity causes an incorrect estimation of translation probability and that the translation
accuracy might decrease for this reason. To address this problem, it is desirable to be able todistinguish pivot-side phrases that have diﬀerent syntactic roles or meanings, even if the symbol
strings are equivalent. The next two subsections describe two methods of discerning pivot phrases
that take on syntactically discrete roles: the ﬁrst technique involves exact matching of parse trees;
the second pertains to soft matching.

4.1

Exact Matching of Parse Subtrees

In the exact matching method, the pivot–source and pivot–target T2S TMs are ﬁrst trainedby parsing the pivot side of parallel corpora. Next, these data are stored into rule tables as TP S
and TP T , respectively. The synchronous rules of TP S and TP T correspondingly take the form
⟨ ⟩of X → ⟨p̂, s⟩ and X → p̂, t , where p̂ is a symbol string that expresses the pivot-side parse
subtree (S-expression), and s and t express the source and target symbol strings in that order.
The procedure of synthesizing source–target synchronous rules essentially follows Equations (11)–(14), except that it utilizes TP S instead of TSP (the direction of probability features is reversed)
and the pivot subtree p̂ instead of pivot phrase p. In this case, s and t do not have syntactic
information, and thus, the synthesized synchronous rules should be hierarchical rules as explainedin Section 2.2.
The matching conditions of this method are more stringent in their constraints than the cor607

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

respondence of superﬁcial symbols in standard triangulation and thus potentially lessen incorrectconnections of phrase pairs, resulting in a more reliable triangulated TM. Conversely, the numberof connected rules decreases as well in this restricted triangulation, and hence, the coverage ofthe triangulated model might be reduced. Therefore, it is important to create TMs that are bothreliable and that comprise superior coverage.

4.2

Partial Matching of Parse Subtrees

To prevent the problem of the reduction of coverage in the exact matching method, theauthors of this paper propose a partial matching method that retains the coverage of standard
triangulation by allowing the connection of incompletely equivalent pivot subtrees. To estimatetranslation probabilities in partial matching, the weighted triangulation generalizing Equations(11)–(14) of standard triangulation with the weight function ψ(·) must ﬁrst be deﬁned as in( ) ∑∑ ()ϕ t|s =
ϕ t|pˆT ψ (pˆT |pˆS ) ϕ (pˆS |s) ,
pˆT

pˆS

pˆS

pˆT

pˆT

pˆS

pˆS

pˆT

(17)

(
)
( ) ∑∑
ϕ (s|pˆS ) ψ (pˆS |pˆT ) ϕ pˆT |t ,ϕ s|t =

(18)( ) ∑∑()
ϕlex t|s =ϕlex t|pˆT ψ (pˆT |pˆS ) ϕlex (pˆS |s) ,( ) ∑∑()
ϕlex (s|pˆS ) ψ (pˆS |pˆT ) ϕlex pˆT |t ,ϕlex s|t =

(19)
(20)

where pˆS ∈ TSP and pˆT ∈ TP T are, respectively, the pivot parse subtrees of source–pivot andpivot–target synchronous rules. By adjusting ψ(·), the magnitude of the penalty for instances ofincompletely matched connections can be controlled. If it is deﬁned that ψ(pˆT |pˆS ) = 1 when pˆTis equal to pˆS and that otherwise ψ(pˆT |pˆS ) = 0, Equations (17)–(20) are equivalent to Equations(11)–(14).
The better estimation of ψ(·) is not trivial, and the co-occurrence counts of pˆS and pˆT are
not available. Therefore, a heuristic estimation method is introduced asψ(pˆT |pˆS ) = ∑w(pˆS , pˆT )· max w(pˆS , p̂),p̂∈TP T w(pˆS , p̂) p̂∈TP T(21)

ψ(pˆS |pˆT ) = ∑w(pˆS , pˆT )· max w(p̂, pˆT ),p̂∈TSP w(p̂, pˆT ) p̂∈TSP(22)

w(pˆS , pˆT ) =
0(f lat(pˆS ) ̸= f lat(pˆT ))exp (−d (pˆS , pˆT ))d(pˆS , pˆT ) = T reeEditDistance(pˆS , pˆT ),
608

,

(23)

(otherwise)(24)

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

where f lat(p̂) returns only the sequence of leaf elements, or the symbol string of p̂ keeping
non-terminals,3 and T reeEditDistance(pˆS , pˆT ) is the minimum cost of a sequence of operations(contract, un-contract, and modify the label of an edge) that are required to transform pˆS intopˆT (Klein 1998).
According to Equations (21)–(24), it may be assured that the incomplete match of pivot
subtrees leads to d(·) ≥ 1 and penalizes in a manner that ψ(·) ≤ 1/ed ≤ 1/e, and an exact matchof subtrees causes a value of ψ(·) that is at least e ≈ 2.718 times larger than those obtained by
the utilization of partially matched subtrees.

5 Experiments
5.1

Experimental Set-Up

5.1.1

Evaluation of Pivot SMT Methods

To investigate the eﬀect of the proposed approach, the authors evaluated translation accuracy through pivot translation experiments conducted on the United Nations Parallel Corpus(UN6Way) (Ziemski et al. 2016). UN6Way is a line-aligned multilingual parallel corpus that
includes data in English (En), Arabic (Ar), Spanish (Es), French (Fr), Russian (Ru) and Chinese(Zh), and accounts for diﬀerent families of languages. The corpus contains over 11 M sentences
for each language pair and is therefore deemed suitable for multilingual translation tasks such as
pivot translation. English was ﬁxed as the pivot language for the experiments reported in thispaper because it is the language most frequently employed for this function. Its utilization as thepivot language thus yields the positive side-eﬀect of the readily available accurate phrase structure parsers, beneﬁts the proposed method. Pivot translation was performed on all combinationsof the other ﬁve languages, and the accuracy of each method was compared. For tokenization,
SentencePiece,4 an unsupervised text tokenizer and detokenizer, was adopted. Although it is designed primarily for use in neural MT, it was conﬁrmed that SentencePiece also helps in reducing
training time and that it also improves translation accuracy in the authors’ previously posited
Hiero model. A single shared tokenization model was ﬁrst trained by feeding a total of 10 M
sentences from the data of all six languages, setting the maximum shared vocabulary size to be
16 k, and all available text was tokenized with the trained model. English raw text was usedwithout SentencePiece tokenization for phrase structure analysis and for training Hiero and T2S
3 For

example, given p̂ = (N P (N P (N N SOﬃcers))(P P (IN of)(N P (DT the)(N N P Committee)))), then,f lat(p̂) = “Oﬃcers of the Committee”.
4 https://github.com/google/sentencepiece609

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

TMs on the pivot side. To generate parse trees, the Ckylark PCFG-LA parser (Oda, Neubig,
Sakti, Toda, and Nakamura 2015) was utilized, and lines over 60 tokens in length were ﬁlteredout from all parallel data to ensure the accuracy of parsing and alignment. Once the sorting was
accomplished, about 7.6 M lines remained. Since Hiero requires a large quantity of computational
resources for training and decoding, the decision was taken to use only the ﬁrst 1 M lines to traineach TM, instead of the entirety of the available training data.5 Travatar (Neubig 2013) was
employed as a decoder. Hiero and T2S TMs were utilized and were trained with Travatar’s rule
extraction code. 5-gram LMs were trained over the target side of the same parallel data utilized
for training TMs using KenLM (Heaﬁeld 2011). The ﬁrst 1,000 lines of the 4,000 lines of test anddev sets were used for testing and parameter tuning, respectively. For the evaluation of translation results, the text was de-tokenized with SentencePiece and re-tokenized with the tokenizer
from the Moses toolkit (Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen,
Moran, Zens, Dyer, Bojar, Constantin, and Herbst 2007) for Arabic, Spanish, French and Russian. The Chinese text was re-tokenized with KyTea tokenizer (Neubig, Nakata, and Mori 2011)
and was then evaluated with the utilization of case-sensitive BLEU-4 (Papineni, Roukos, Ward,and Zhu 2002), RIBES (Isozaki, Hirao, Duh, Sudoh, and Tsukada 2010) and NIST (Doddington2002).
Five translation methods were assessed:
Cascade Hiero:
Sequential pivot translation with source–pivot and pivot–target Hiero TMs (weak baseline).
Tri. Hiero:
Triangulating source–pivot and pivot–target Hiero TMs into a source–target Hiero TM
using the traditional method (baseline, Section 3.1).
Tri. TreeExact
Triangulating pivot–source and pivot–target T2S TMs into a source–target Hiero TM
using the proposed exact matching of pivot subtrees (proposed 1, Section 4.1).
Tri. TreePartial
Triangulating pivot–source and pivot–target T2S TMs into a source–target Hiero TM
using the proposed partial matching of pivot subtrees (proposed 2, Section 4.2).
5 The authors use the same 1 M pivot-side sentences in source-pivot and pivot-target parallel data, even though

it is not a realistic situation in pivot translation. This setting is useful to investigate how pivot translation
tasks degrade accuracy from the ideal condition by comparing with scores of Direct methods. The previous
work of the authors (Miura et al. 2015; 三浦，Neubig，Sakti，戸田，中村 2016b) has not found large diﬀerencesin translation accuracy between situations of using the same pivot sentences and no common pivot sentences.

610

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

Direct Hiero:Translating with a Hiero TM directly trained on the source-target parallel corpus without
using a pivot language (as an oracle).
5.1.2

Comparison with Neural MT

Recent investigations (Firat, Sankaran, Al-Onaizan, Yarman Vural, and Cho 2016; Johnson,
Schuster, Le, Krikun, Wu, Chen, Thorat, Viégas, Wattenberg, Corrado, Hughes, and Dean 2017)
have found that neural machine translation systems can gain the ability to perform translations
with zero parallel resources by training on multiple sets of bilingual data. However, previous work
has not examined the competitiveness of these methods in comparison to pivot-based symbolic
SMT frameworks such as PBMT or Hiero. In this section, a zero-shot NMT model and otherpivot NMT methods are compared to the pivot-based Hiero models. The NMTKit 6 was adopted
to train and evaluate NMT models. The detailed parameters of training the NMT models areshown in Table 1.
For additional translation methods were assessed:
Cascade NMT:
Sequential pivot translation with source-pivot and pivot-target NMTs.

Table 1 Main parameters of NMT training
vocabulary size
source embedding size
target embedding size
output embedding size
encoder hidden size
decoder hidden size
LSTM layers
attention typeattention hidden size
optimizer typeloss integration typebatch size
max iteration
dropout rate
decoder type6 https://github.com/odashi/nmtkit

611

16 k (shared)
512
512
512
512
512
1
MLP
512
Adam
mean
2,048
200 k
0.3
Luong+ 2015

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

Synthetic NMT:
Generating pseudo-parallel corpus synthesized by translating pivot-side of source-pivot
parallel corpus with pivot-target NMT (Sennrich, Haddow, and Birch 2016; Firat et al.
2016).
Zero-Shot NMT:
Training single shared model with pvt ↔ {src,target} parallel data according to (Johnson
et al. 2017).
Direct NMT:
Translating with NMT directly trained on the source-target parallel corpus without using
pivot language (for comparison).
The training data for Cascade NMT, Synthetic NMT, and Zero-Shot NMT were the same
1 M sentences used for the Pivot Hiero methods (source–pivot and pivot–target corpora). The
training data for Direct NMT was identical to that utilized for Direct Hiero.

5.2

Results

Comparison of Direct models among diﬀerent language pairs and frameworks: Before
pivot translation tasks were compared, the performances of SMT and NMT in Direct translation
tasks were ascertained. Table 2 demonstrates the BLEU-4, RIBES, and NIST scores for each
Direct translation task and language pair. The results conﬁrm the tendency that directly trained
NMT models achieve high translation accuracy even in the case of translation between languagesof diﬀerent families On the other hand, these scores are drastically reduced in situations that do
not oﬀer source–target parallel corpora for training.
Since the parameters were optimized for the BLEU score and not for RIBES and NIST, higher
RIBES/NIST scores do not necessarily imply the superior performance of the MT framework.
They merely demonstrate the side eﬀects obtained by optimizing for BLEU. However, the comparison of RIBES and NIST scores shows the obvious propensities of SMT and NMT frameworks.
In almost language pairs, Direct Hiero outperformed Direct NMT in NIST scores, even though
its BLEU and RIBES scores were lower than NMT. It is known that NMT faces diﬃculties withrare words and that it tends to fail in the translation of vocabulary that is not frequently used.
This discovery reﬂects the fact that NIST is recognized as a measurement tool that values thetranslation accuracy of content words that generally occur in lower frequencies in comparison tofunction words. Conversely, NMT is capable of acquiring higher naturalness and ﬂuency in word
sequences that include function words, since BLEU provides weightage to the accuracy of word
sequences (n-grams) and RIBES is known to give importance word order.
612

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

Table 2 Comparison of SMT and NMT in multilingual Direct translation tasks
SourceTarget

Ar
Es
Fr
Ru
Zh

En
En
En
En
En
Ar
Es
Fr
Ru
Zh

BLEU (%) / RIBES (%) / NIST Score
Direct Hiero
Direct NMT
40.00 / 83.98 / 7.838
40.16 / 83.33 / 7.710
49.50 / 87.41 / 9.208
50.51 / 87.91 / 9.130
42.13 / 83.91 / 8.321
43.12 / 84.70 / 8.220
41.15 / 83.75 / 8.026
40.88 / 82.99 / 7.831
34.19 / 78.31 / 7.368
35.50 / 80.64 / 7.249
28.18 / 80.11 / 6.717
30.58 / 80.84 / 6.793
49.70 / 86.91 / 9.201
50.61 / 88.57 / 9.187
40.57 / 82.40 / 8.033
41.56 / 84.24 / 8.112
31.63 / 79.61 / 6.777
34.76 / 80.68 / 6.979
33.07 / 80.89 / 8.170
38.05 / 83.78 / 8.176

Es
Fr
Ru
Zh
Ar
Fr
Ru
Zh
Ar
Es
Ru
Zh
Ar
Es
Fr
Zh
Ar
Es
Fr
Ru

38.49 / 82.85 / 7.442
33.34 / 79.97 / 6.828
24.63 / 75.55 / 5.813
27.27 / 76.31 / 6.827
27.18 / 79.19 / 6.350
43.24 / 85.60 / 8.240
28.83 / 77.84 / 6.434
27.08 / 75.29 / 7.037
25.10 / 77.51 / 5.854
45.20 / 86.48 / 8.317
27.42 / 77.11 / 6.016
25.84 / 74.55 / 6.619
22.53 / 76.03 / 5.722
37.60 / 82.04 / 7.496
34.05 / 79.88 / 6.945
28.03 / 76.31 / 7.083
20.09 / 70.59 / 5.382
30.66 / 74.43 / 6.580
25.97 / 71.87 / 6.012
21.16 / 69.27 / 5.280

En

Ar

Es

Fr

Ru

Zh

38.25 / 83.09 / 7.288
33.16 / 78.80 / 6.641
27.00 / 75.87 / 5.745
30.04 / 79.48 / 6.771
26.02 / 78.73 / 6.184
41.83 / 83.82 / 7.924
30.65 / 78.70 / 6.429
32.36 / 80.85 / 7.320
23.28 / 76.29 / 5.732
44.49 / 85.35 / 8.294
28.29 / 75.80 / 5.963
29.10 / 78.78 / 6.833
23.19 / 76.32 / 5.636
38.67 / 82.09 / 7.409
33.26 / 78.57 / 6.764
31.39 / 79.13 / 6.993
17.73 / 73.59 / 5.210
28.05 / 78.33 / 6.502
24.35 / 74.76 / 5.954
19.59 / 72.44 / 5.218

Bold face indicates higher evaluation score for each language-pair and measurement.

The table also demonstrates that the evaluation scores of language-pairs that do not contain
English are much lower than those that include English. For example, the BLEU scores of
Arabic–English, English–French and Arabic–French in Direct Hiero are 40.00, 40.57, and 33.34,
respectively. It is thought that sentence pairs that do not include English are more noisy than613

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

those that contain English since the multilingual corpus was primarily constructed by sourcing
from English documents as the pivot.

Performance of English-related translation tasks: Pivot translation tasks should depend
strongly on the performance of the source–pivot translation and should rely even more compellingly on the pivot–target translation since the pivot–target translation essentially comprises
the upper bound performance of generating target sentences for the given pivot-side input. It
is natural that the translation for pairs of languages belonging to diﬀerent families exhibits adiﬀerent trend with regard to accuracy. Table 2 illustrates that the TMs of English–Spanish andEnglish–French achieve higher evaluation scores, perhaps because they exhibit relatively closer
language structures than the other evaluated English-relative language pairs. Although English–
Arabic, English–Russian, and English–Chinese translation achieve poorer accuracy, each of theselikely result from diﬀerent language features, such as morphology, word order, and diversity of
expression.
Table 3 illustrates the components of the BLEU score evaluation, including the precision of
1-grams through 4-grams and the brevity penalty (Papineni et al. 2002). This table demonstrates
that English–Chinese translation achieved higher accuracy in translating words, or 1-gram precision than language pairs that comprises Arabic, French, and Russian targets. However, theprecision of 2-grams through 4-grams, or the accuracy of translating word sequences is relatively
lower in English–Chinese and this low BLEU score is primarily caused by the low 4-gram precision. This result reﬂects the fact that word inﬂections do not exist in Chinese, and instead, theTable 3 Components of BLEU score for English-related translation tasks
SourceTarget

Ar
Es
Fr
Ru
Zh

En
En
En
En
En
Ar
Es
Fr
Ru
Zh

En

1-Gram / 2-Gram / 3-Gram / 4-Gram Precision / Brevity Penalty (%)
Direct Hiero
Direct NMT
67.03 / 44.98 / 33.28 / 25.52 / 100.0 65.88 / 45.34 / 33.82 / 25.76 / 100.0
74.83 / 54.48 / 42.78 / 34.42 / 100.0 74.76 / 55.40 / 44.03 / 35.70 / 100.0
68.97 / 47.02 / 35.39 / 27.46 / 100.0 69.33 / 48.30 / 36.78 / 28.61 / 99.53
68.19 / 46.00 / 34.29 / 26.66 / 100.0 66.72 / 45.43 / 34.17 / 26.96 / 100.0
65.17 / 39.89 / 27.16 / 19.34 / 100.0 63.38 / 40.94 / 29.07 / 21.06 / 100.0
57.37 / 34.13 / 22.19 / 14.73 / 99.65 58.75 / 37.66 / 26.15 / 18.43 / 95.16
74.17 / 54.89 / 43.29 / 34.88 / 100.0 74.28 / 56.49 / 45.22 / 36.39 / 98.72
65.89 / 45.46 / 34.20 / 26.44 / 100.0 67.27 / 47.71 / 36.80 / 28.95 / 96.45
58.51 / 37.38 / 26.63 / 19.67 / 96.67 60.10 / 40.67 / 30.36 / 23.22 / 95.96
71.36 / 42.22 / 27.23 / 18.31 / 94.45 70.33 / 45.45 / 31.57 / 22.78 / 97.73

614

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

word order takes on signiﬁcant syntactic roles.
Conversely, the table also clariﬁes that English–Arabic and English–Russian translationachieved lower precision even relating to 1-grams. This lack of accuracy could be caused by
the fact that Arabic and Russian are known for their morphological richness, and it is thus morediﬃcult for MT to translate the source words into correct forms of target words than in the caseof other, more morphologically simple languages.

Translation accuracy of pivot translation methods: The results of the experiments thatused all combinations of pivot translation tasks via English for ﬁve languages are shown in Table
4. These outcomes exhibit that the proposed partial matching method of pivot subtrees intriangulation outperformed the standard triangulation method for all language pairs and that it

Table 4 Comparison of each method

Source

Ar

Es

Fr

Ru

Zh

Target
Es
Fr
Ru
Zh
Ar
Fr
Ru
Zh
Ar
Es
Ru
Zh
Ar
Es
Fr
Zh
Ar
Es
Fr
Ru

Direct
38.49
33.34
24.63
27.27
27.18
43.24
28.83
27.08
25.10
45.20
27.42
25.84
22.53
37.60
34.05
28.03
20.09
30.66
25.97
21.16

Cascade(baseline)
30.95
25.08
18.70
21.77
22.72
35.40
22.43
23.36
19.88
37.75
20.64
21.79
18.71
31.33
27.11
21.81
14.82
23.15
19.55
14.79

BLEU Score (%)
Tri. Hiero Tri. TreeExact(baseline)(proposed 1)
34.20
34.97‡
29.93
30.68‡
22.94
23.94‡
22.78
25.17‡
22.97
24.09‡
38.74
39.62‡
26.35
27.25‡
24.54
25.00
21.65
21.40
40.16
41.03‡
24.71
25.24†
23.16
23.56
19.82
19.86
34.56
34.96
30.75
31.43†
24.88
25.07
16.66
17.01
27.84
27.99
23.82
24.34
18.63
19.58‡

Tri. TreePartial(proposed 2)
35.94‡
30.83‡
24.15‡
25.07‡
24.45‡
40.12‡
27.41‡
25.16‡
22.13‡
41.99‡
25.64‡
23.53
20.35
35.62‡
31.67‡
25.12
17.73‡
28.05
24.35†
19.59‡

Bold face indicates the highest BLEU score in pivot translation, and daggers indicate statistically significant gains over Tri. Hiero († p < 0.05, ‡ p < 0.01).

615

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

achieved higher or almost equal scores than the proposed exact matching method. The exact
matching method also outperformed the standard triangulation method in the majority of thelanguage pairs, but has a lesser improvement than the partial matching method. As demonstrated
by the authors’ previous research undertaking, the sequential pivot translation was uniformly
weaker than all triangulation methods.

Eﬀect on coverage: Table 5 presents the outcomes of the comparison of the coverage achieved byeach proposed triangulation method. This table conﬁrms that Tri. TreeExact reduced the numberof unique phrases by several percentage points and Tri. TreePartial kept the same coverage as
Tri. Hiero. Especially, triangulated TMs from Chinese with exact matching contain substantiallyfewer source phrases and signiﬁcantly source words, and harmed coverage up to 4.884% as Chinesecontains many characters and values the reordering of short tokens instead of inﬂections. Thisanomaly could constitute one of the reasons for the diﬀerence in improvement stability withregard to the partial and exact matching methods.

Table 5 Comparison of rule table coverage in proposed triangulation methods
Source

Ar

Es

Fr

Ru

Zh

Target
Es
Fr
Ru
Zh
Ar
Fr
Ru
Zh
Ar
Es
Ru
Zh
Ar
Es
Fr
Zh
Ar
Es
Fr
Ru

Number of source-side unique phrases/words
Tri. Hiero
Tri. TreePartial Tri. TreeExact
2.646 M / 5,077 2.646 M / 5,077 2.580 M / 5,072
2.658 M / 5,071 2.658 M / 5,071 2.589 M / 5,067
2.406 M / 5,088 2.406 M / 5,088 2.347 M / 5,085
2.386 M / 5,040 2.386 M / 5,040 2.324 M / 5,034
2.013 M / 5,188 2.013 M / 5,188 1.942 M / 5,182
2.129 M / 5,210 2.129 M / 5,210 2.062 M / 5,205
2.037 M / 5,197 2.037 M / 5,197 1.978 M / 5,191
1.986 M / 5,180 1.986 M / 5,180 1.920 M / 5,175
2.233 M / 5,316 2.233 M / 5,316 2.176 M / 5,310
2.366 M / 5,342 2.366 M / 5,342 2.302 M / 5,337
2.266 M / 5,318 2.266 M / 5,318 2.203 M / 5,311
2.215 M / 5,321 2.215 M / 5,321 2.162 M / 5,313
2.505 M / 5,644 2.505 M / 5,644 2.437 M / 5,637
2.536 M / 5,682 2.536 M / 5,682 2.478 M / 5,677
2.531 M / 5,665 2.531 M / 5,665 2.479 M / 5,661
2.515 M / 5,688 2.515 M / 5,688 2.466 M / 5,682
1.556 M / 9,474 1.556 M / 9,474 1.480 M / 9,428
1.570 M / 9,555 1.570 M / 9,555 1.504 M / 9,523
1.568 M / 9,520 1.568 M / 9,520 1.499 M / 9,490
1.593 M / 9,487 1.593 M / 9,487 1.518 M / 9,457

616

loss in coverage (%)
2.494 / 0.985
2.596 / 0.079
2.452 / 0.059
1.844 / 0.119
3.527 / 0.116
3.147 / 0.096
2.896 / 0.115
3.323 / 0.097
2.553 / 0.113
2.705 / 0.094
2.780 / 0.132
2.393 / 0.150
2.715 / 0.124
2.287 / 0.088
2.055 / 0.071
1.948 / 0.105
4.884 / 0.486
4.200 / 0.335
4.401 / 0.315
4.708 / 0.316

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

Noise reduction: The main motivation of using parse trees in the proposed methods is toprevent the inappropriate connection of phrase correspondences and reduce the noise in rule
tables. To investigate the manner in which the syntactic matching methods succeed in removing
noisy rules, an analysis of noise ratio was conducted. Noisy rules must contain source and target
phrases having no correspondence in meaning, though this decision cannot be made for all phrase
pair candidates in rule tables. It was therefore assumed that directly trained TMs that could
avail of a source–target parallel corpus would demonstrate a ﬁne approximation close to the ideal
distribution of translation probability. To compute the noise ratio noise(Ttri |Tdir ) of triangulated
rule table Ttri with directly trained table Tdir

( )∑(s,t)∈Ttri \Tdir ϕ t|s( )noise (Ttri |Tdir ) = ∑(s,t)∈Ttri ϕ t|s(25)

was deﬁned, where ϕ(t|s) represents the forward translation probability that can be considered
the most important feature of the rule table. Table 6 displays the calculated noise ratio of the rule

Table 6 Comparison of noise ratio in triangulated rule table
Source

Ar

Es

Fr

Ru

Zh

Target
Es
Fr
Ru
Zh
Ar
Fr
Ru
Zh
Ar
Es
Ru
Zh
Ar
Es
Fr
Zh
Ar
Es
Fr
Ru

Noise Ratio in Triangulated Table (%)
Tri. Hiero Tri. TreeExact Tri. TreePartial
78.40
63.61 (−14.79)
68.51 (−9.88)
81.39
67.31 (−14.08)
72.22 (−9.17)
81.87
69.23 (−12.64)
73.84 (−8.03)
75.70
63.06 (−12.64)
67.72 (−7.98)
80.03
64.97 (−15.06) 69.80 (−10.23)
81.55
65.30 (−16.26) 70.61 (−10.94)
81.45
68.02 (−13.43)
72.67 (−8.78)
74.05
61.60 (−12.45)
65.90 (−8.15)
81.77
67.69 (−14.08)
72.39 (−9.38)
80.94
64.94 (−16.00) 70.20 (−10.74)
82.77
69.84 (−12.93)
74.52 (−8.25)
76.14
64.29 (−11.85)
68.53 (−7.61)
82.15
70.15 (−12.00)
74.60 (−7.55)
79.80
67.16 (−12.64)
71.68 (−8.12)
82.41
70.10 (−12.31)
74.76 (−7.65)
76.07
64.31 (−11.76)
68.67 (−7.40)
80.05
66.90 (−13.15)
71.23 (−8.82)
77.94
65.53 (−12.41)
69.70 (−8.24)
78.24
68.54 (−9.70)
72.51 (−5.73)
79.80
67.07 (−12.73)
71.27 (−8.53)

617

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018table for each triangulation method and language pair. This result shows that, although triangulated rule tables contain many noisy rules, the syntactic matching methods are indeed successful
in reducing them. Tri. TreeExact decreased noisy rules, up to −16.26%, and Tri. TreePartial
lessened noisy rules up to −10.94%. The reason why the noise reduction rate of Tri. TreePartial
is lower than that of Tri. TreeExact is that the former weakens the inﬂuence of noisy rules insteadof removing them to retain the coverage.

Improvement of probability estimation: Although syntactic matching methods help in
reducing noisy rules, there is no guarantee that they can improve the estimation of translation
probabilities. Table 7 exhibits the mean absolute error (MAE) and the root-mean-square error(RMSE) for the distribution of forward translation probability scores of triangulated rule tables
in comparison to directly trained rule tables. To calculate MAE and RMSE, the noisy rules thatwere not contained in directly trained rule tables were ignored to separate the diﬀerent factors.
The results evince that Tri. TreePartial reduced MAE and RMSE, making the distribution closer

Table 7 Comparison of distribution error rate in triangulated rule table
Source

Ar

Es

Fr

Ru

Zh

Target
Es
Fr
Ru
Zh
Ar
Fr
Ru
Zh
Ar
Es
Ru
Zh
Ar
Es
Fr
Zh
Ar
Es
Fr
Ru

Distribution Error Rate (MAE / RMSE) (%)
Tri. Hiero
Tri. TreeExact Tri. TreePartial
14.16 / 10.62
14.49 / 11.06
13.96 / 10.56
13.01 / 9.72
13.52 / 10.19
12.90 / 9.65
12.64 / 9.51
12.33 / 9.24
12.03 / 8.97
15.88 / 11.96
13.69 / 10.42
13.81 / 10.42
13.90 / 10.29
13.84 / 10.30
13.44 / 9.92
13.39 / 10.61
14.51 / 11.30
13.95 / 10.89
12.81 / 9.71
12.92 / 9.70
12.52 / 9.38
16.02 / 12.09
13.94 / 10.69
14.01 / 10.64
13.40 / 9.98
13.10 / 9.76
12.70 / 9.38
14.25 / 11.38
14.39 / 11.29
14.05 / 11.03
12.58 / 9.58
12.46 / 9.37
11.98 / 8.99
15.45 / 11.74
13.34 / 10.28
13.40 / 10.23
12.68 / 9.35
12.36 / 9.16
11.98 / 8.79
13.27 / 10.05
13.68 / 10.54
13.12 / 10.00
12.29 / 9.28
12.84 / 9.78
12.13 / 9.17
15.34 / 11.72
13.13 / 10.10
13.25 / 10.11
12.57 / 9.11
12.86 / 9.39
12.57 / 9.09
13.25 / 9.78
13.58 / 10.16
13.22 / 9.79
12.86 / 9.49
12.67 / 9.44
12.25 / 9.07
12.22 / 9.14
12.40 / 9.31
12.12 / 9.03

618

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

to the ideal in almost all language pairs. On the other hand, Tri. TreeExact did not diminish
the errors in a stable manner. This consequence may be induced by the fact that the restricted
matching conditions of Tri. TreeExact exclude many unmatched phrase pair candidates andmay remove even those translation rules that are not noisy. It may therefore be posited thatthe softening of restrictions pertaining to matching conditions aids in the improvement of theestimation of translation probabilities.

Comparison with NMT:
Table 8 presents the BLEU score of each translation task and language pair. Perhaps, Synthetic NMT outperformed Cascade NMT for the majority of language pairs because multi-layer
NNs are robust for noisy training data and can optimize the trained model with ﬁne-tuning techniques. On the other hand, the ﬁne-tuning is available for Cascade NMT only separately for thesource–pivot and pivot–target TMs and not for the whole pipelined system.

Table 8 Comparison of SMT and NMT in pivot translation tasks

Source

Ar

Es

Fr

Ru

Zh

Target
Es
Fr
Ru
Zh
Ar
Fr
Ru
Zh
Ar
Es
Ru
Zh
Ar
Es
Fr
Zh
Ar
Es
Fr
Ru

Direct
Hiero
38.49
33.34
24.63
27.27
27.18
43.24
28.83
27.08
25.10
45.20
27.42
25.84
22.53
37.60
34.05
28.03
20.09
30.66
25.97
21.16

Direct
NMT
38.25
33.16
27.00
30.04
26.02
41.83
30.65
32.36
23.28
44.49
28.29
29.10
23.19
38.67
33.26
31.39
20.17
32.69
27.68
23.17

BLEU Score (%)
Tri.
Cascade Cascade
TreePartial
Hiero
NMT
35.94
30.95
31.62
30.83
25.08
26.91
24.15
18.70
21.67
25.07
21.77
23.70
24.45
22.72
21.21
40.12
35.40
31.84
27.41
22.43
23.60
25.16
23.36
26.03
22.13
19.88
18.66
41.99
37.75
32.93
25.64
20.64
20.87
23.53
21.79
23.14
20.35
18.71
19.71
35.62
31.33
31.25
31.67
27.11
27.34
25.12
21.81
24.25
17.73
14.82
16.89
28.05
23.15
26.01
24.35
19.55
23.35
19.59
14.79
18.40

619

Synthetic
NMT
32.35
29.51
21.81
25.63
23.01
36.57
25.97
27.31
18.83
36.78
23.60
24.96
19.21
31.22
29.10
25.46
18.01
27.80
25.46
20.53

Zero-Shot
NMT
8.18
8.57
5.79
5.04
5.22
15.04
7.57
8.62
8.08
14.37
8.77
11.95
3.18
10.42
9.76
9.46
10.38
6.13
7.12
3.21

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

In the setting of the current experiments, although bilingually trained NMT systems werecompetitive to or outperformed Hiero-based models, the zero-shot translation was uniformly
weaker. This outcome could be the result of using only a single LSTM layer for each encoderand decoder or because there was an insuﬃcient quantity of parallel corpora or language pairs.
It may therefore be posited that, although zero-shot translation demonstrated reasonable resultsin some settings, successful zero-shot translation systems are diﬃcult to build, and pivot-based
symbolic MT systems such as PBMT or Hiero might still be competitive alternatives.

Qualitative analysis: A translated sentence for which pivot-side ambiguity is resolved in thesyntactic matching methods is presented as an example:
Source Sentence in French:
La Suisse encourage tous les États parties à soutenir le travail conceptuel que fait˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜actuellement le Secrétariat .

˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜Corresponding Sentence in English:
Switzerland encourages all parties to support the current conceptual work of the secretariat.
Reference in Spanish:Suiza alienta a todos los Estados partes a que apoyen la actual labor conceptual de˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜la Secretarı́a .

˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜

Direct Hiero:Suiza alienta a todos los Estados partes a que apoyen el trabajo conceptual que se examinan
en la Secretarı́a . (BLEU+1: 55.99, RIBES: 91.47, NIST: 9.687)
Tri. Hiero:Suiza conceptuales para apoyar la labor que en estos momentos la Secretarı́a alienta a to˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜˜

dos los Estados Partes . (BLEU+1: 29.74, RIBES: 53.92, NIST: 6.204)
Tri. TreeExact:

Suiza alienta a todos los Estados Partes a apoyar la labor conceptual que actualmente
la Secretarı́a . (BLEU+1: 43.08, RIBES: 88.36, NIST: 8.625)
Tri. TreePartial:
Suiza alienta a todos los Estados Partes a apoyar la labor conceptual que actualmente
la Secretarı́a . (same with Tri. TreeExact)
Direct NMT:
Suiza alienta a todos los Estados partes a que apoyen la labor conceptual que está realizando la Secretarı́a . (BLEU+1: 66.94, RIBES: 95.79, NIST: 11.22)
620

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

Synthetic NMT:
Suiza alentó a todos los Estados Partes a que apoyen el trabajo conceptual que actualmenteestá a la Secretarı́a . (BLEU+1: 28.94, RIBES: 88.01, NIST: 7.267)
The results of Tri. TreeExact and Tri. TreePartial are identical in this example. The digestof the derivation process in Tri. Hiero isS ⇒ ⟨X0 , X0 ⟩⇒ ⟨La Suisse X1 , Suiza X1 ⟩⇒ ⟨La Suisse X2 ., Suiza X2 .⟩⟨⟩⇒ La Suisse X3 partie X4 ., Suiza X4 X3 P artes .
⇒ ...

˜˜˜˜˜

˜˜˜˜˜

On the other hand, the digest of the derivation process in Tri. TreeExact/Tri. TreePartial isS ⇒ ⟨X0 , X0 ⟩⇒ ⟨La Suisse X1 , Suiza X1 ⟩⇒ ⟨La Suisse encourage X2 X3 , Suiza alienta a X2 X3 ⟩⇒ ⟨La Suisse encourage tous les X4 partie X3 , Suiza alienta a todos X4 P artes X3 ⟩⇒ ⟨La Suisse encourage tous les Etats partie X3 , Suiza alienta a todos los Estados P artes X3 ⟩⇒ ...

Here, it is observed that the derivation in Tri. Hiero uses rule X → ⟨X0 parties X1 , X1 X0Partes⟩7 which causes the incorrect reordering of phrases, followed by steps of incorrect word selection.8 On the other hand, the derivation in Tri. TreeExact and Tri. TreePartial uses rule X →⟨tous les X0 parties , todos X0 Partes⟩9 as synthesized from the T2S rules with the common pivot
subtree(NP (DT all) (NP’ XNNP (NNS parties)). It can thus be proved that the derivation improves
the word-selection and word-reordering using this rule.
The following example presents a translated sentence for which the exact matching method
loses the necessary translation rule and instead degrades accuracy.

7 The words emphasized with underline and wavy-underline in the example correspond to X and X respec0
1

tively.
8 For example, the word “conceptuales” with italic face in Tri. Hiero takes the wrong form and position.
9 The words emphasized in bold face in the example correspond to the rule.

621

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

Source Sentence in Chinese
秘书长 关于 “ 人力 资源 管理 改革 概览 : 流动性 ” 的 报告

Corresponding Sentence in English:
Report of the Secretary-General on the overview of human resources management reform: mobility
Reference in RussianДоклад Генерального секретаря об обзоре ходареформы системы управления людскими ресурса
ми : мобильность

Direct Hiero:Доклад Генерального секретаря об реформе управления людскими ресурсами : обзор мобильность » (BLEU+1: 45.47, RIBES: 92.29, NIST: 9.143)

Tri. Hiero:Доклад Генерального секретаря о реформе управления людскими ресурсами обзор : Мобильност
ь » (BLEU+1: 26.25, RIBES: 86.67, NIST: 5.962)

Tri. TreeExact:Доклад Генерального секретаря мобильности обзор реформе управления людскими ресурсами »: ” . (BLEU+1: 26.19, RIBES: 85.66, NIST: 5.652)
Tri. TreePartial:Доклад Генерального секретаря о Обзор реформе системы управления людскими ресурсами : мобильность ” . (BLEU+1: 48.76, RIBES: 89.54, NIST: 8.107)

Direct NMTДоклад Генерального секретаря о “ Обзоре реформы управления людскими ресурсами : мобильно
сть ” (BLEU: 45.37, RIBES: 91.22, NIST: 8.652)

Synthetic NMTДоклад Генерального секретаря о реформе управления людскими ресурсами : доклад Генерального секретаря о реформе управления людскими ресурсами : доклад Генерального секретаря (BLEU+1:

622

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

21.16, RIBES: 61.48, NIST: 3.608)
In this example, the corresponding Russian word form of the Chinese word “流动性” (mobility)
is “мобильность.” However, Tri. TreeExact places this word in the incorrect case form“мобильности” and also positions it far from the correct placing since the translation
rule connecting “流动性” with the correct form “мобильность” is lost in the process
of exact matching, and this rule is maintained in Tri. Hiero and Tri. TreePartial. The selectionof the incorrect case form often causes misplacing because of LMs, and it is aﬃrmed that theresults obtained by the use of Tri. TreeExact contain more incorrect word forms and positions.

5.3

Related Work

Up to this point, representative pivot translation methods in SMT have been explained. Other
related research studies in pivot translation are primarily based on the triangulation for PBMT
and focuse on discussions to further improve accuracy (Zhu, He, Wu, Zhu, Wang, and Zhao
2014; Levinboim and Chiang 2015; Dabre, Cromieres, Kurohashi, and Bhattacharyya 2015). The
process of correctly estimating the translation probability is a problem in triangulation.
Zhu et al. (2014) have proposed an estimation method of source–target translation probability by estimating source-target co-occurrence counts ﬁrst instead of the direct estimation fromsource–pivot and pivot–target translation probabilities (Equations 11–14). They have reported
that stable translation accuracy can be obtained even in the triangulation of two phrase tables
with unbalanced table size.
Levinboim and Chiang (2015) have asserted that it is especially diﬃcult to estimate wordlevel translation probability for phrase correspondence in the triangulation stage. Subsequently,
they have proposed a method for improving the quality of the triangulation by estimating thetranslation probability even for the correspondence of words which cannot be directly observed,
using a distributed expression of words (Mikolov, Yih, and Zweig 2013).
This paper focuses on pivot translation using English as the pivot language, though it is also
known that translation accuracy varies on the manner in which a pivot language is selected. The
inﬂuence of the choice of the pivot language on pivot translation has been discussed in detail byPaul, Yamamoto, Sumita, and Nakamura (2009). In reality, there are few situations in which
the pivot language can be selected from multiple viable candidates, though in the ideal scenario
where bilingual corpora of the same scale can be obtained via several languages, a pivot language
having a similar language structure as the source or target language should be chosen.
Additionally, it is not necessary to limit the number of pivot languages to one, and methods that consider the simultaneous use of multiple pivot languages have also been proposed.
623

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

Representatives of such purposing include methods such as aggregating multiple source–target
phrase/rule tables obtained by triangulation with respective pivot languages into one table withlinear interpolation and those that accomplish searching via the simultaneouse use of multiple
TMs (Dabre et al. 2015).
Alternatively, training methods of multilingual NMT, which improve translation accuracy by
causing translation tasks of multiple language pairs to be trained as a common encoder, havebeen also proposed (Dong, Wu, He, Yu, and Wang 2015; Zoph and Knight 2016; Johnson et al.
2017).

6 Conclusion
In this paper, the authors have proposed a new method of pivot translation to resolve thepivot-side syntactic ambiguity. This proposed method introduces an explicitly syntax-aware
matching condition to ﬁnd the correct correspondences between source-pivot and pivot-target
translation rules. It can thus produce more reliable models. The results on the multilingualtranslation experiments revealed signiﬁcant improvements in MT evaluation scores for the proposed method for all tested language pairs that possessed larger indirectly parallel corpora than
(Miura et al. 2015), of 1 M sentence pairs and that had access to the additional resource ofEnglish syntactic parsing. A syntactic matching method that allowed partial matching successfully reduced the number of noisy translation rules. This noise decrease enhanced the estimationof translation probabilities and better translation accuracy. This method is eﬀective instanceswhere that accurate syntactic parsers for the pivot language are available, and it is practical touse for pivot translations that allow access to larger quantities of parallel corpora.
To estimate translation probabilities, a heuristic that had no guarantee of being optimal was
introduced. The smoothing technique of TMs could present an eﬀective solution as one of thedirections to improve the estimation of probability scores. It is common to apply smoothing
methods such as back-oﬀ and interpolation when the coverage of a single model is poor. In
their experimental setting, the authors have incorporated reliable and high-coverage models,and it should be easy to combine them by linear interpolation with ﬁxed coeﬃcients. In in the
future, therefore, the authors are planning to explore more reﬁned estimation methods that utilize
machine learning.
Incidentally, the previously proposed method of pivot translation (Miura et al. 2015) uses
MSCFG models that possess the potential to access information from the source, target, andpivot languages. For example, it should be possible to combine the proposed methods in Section624

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

4 and (Miura et al. 2015) and to let the MSCFG model remember the pivot tree structures.
As a more advanced method for future research, it should be possible to devise compounded
MSCFG models that can store both pivot-side and source-side syntactic information, thereby
realizing translations with higher reproducibility of source information. The authors have mentioned that pivot translation presents the problem of the loss of source language information,
which is aﬀected by the expressiveness of the pivot language. In fact, this problem is frequent
not just in MT but also for human translators. For example, since modern English is known
for its simple morphology that does not include complicated grammatical conjugations such as
personal suﬃxes, linguistic modalities such as number, case, and gender are lost when translating
into English. This deﬁciency yields a translation that is diﬀerent from the original meaning fromEnglish into another language. With the method posited in this paper, the authors aim to achievemachine translation outcomes that preserve the linguistic information of the original content by
combining it with pivot-side syntactic structures.

Acknowledgement
Part of this work was supported by JSPS KAKENHI Grant Numbers JP16H05873 andJP17H06101. The authors also thank the anonymous reviewers for their insightful comments.

Reference
Aho, A. V. and Ullman, J. D. (1969). “Syntax Directed Translations and the Pushdown Assembler.” Journal of Computer and System Sciences, 3 (1), pp. 37–56.
Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. (1993). “The Mathematics
of Statistical Machine Translation: Parameter Estimation.” Computational Linguistics, 19,
pp. 263–312.
Chappelier, J.-C. and Rajman, M. (1998). “A Generalized CYK Algorithm for Parsing Stochastic
CFG.” In Proceedings of TAPD, Vol. 98, pp. 133–137. Citeseer.
Chiang, D. (2007). “Hierarchical Phrase-Based Translation.” Computational Linguistics, 33 (2),pp. 201–228.
Cohn, T. and Lapata, M. (2007). “Machine Translation by Triangulation: Making Eﬀective Useof Multi-Parallel Corpora.” In Proceedings of ACL, pp. 728–735.
Dabre, R., Cromieres, F., Kurohashi, S., and Bhattacharyya, P. (2015). “Leveraging Small
Multilingual Corpora for SMT Using Many Pivot Languages.” In Proceedings of NAACL,
625

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

pp. 1192–1202.
de Gispert, A. and Mariño, J. B. (2006). “Catalan-English Statistical Machine Translation
without Parallel Corpus: Bridging through Spanish.” In Proceedings of LREC 5th Workshop
on Strategies for Developing Machine Translation for Minority Languages, pp. 65–68.
Doddington, G. (2002). “Automatic Evaluation of Machine Translation Quality Using N-gram
Co-Occurrence Statistics.” In Proceedings of ARPA Workshop on Human Language Technology, pp. 138–145.
Dong, D., Wu, H., He, W., Yu, D., and Wang, H. (2015). “Multi-Task Learning for Multiple
Language Translation.” In Proceedings of ACL, pp. 1723–1732.
Dyer, C., Cordova, A., Mont, A., and Lin, J. (2008). “Fast, Easy, and Cheap: Constructionof Statistical Machine Translation Models with MapReduce.” In Proceedings of WMT,
pp. 199–207.
Firat, O., Sankaran, B., Al-Onaizan, Y., Yarman Vural, F. T., and Cho, K. (2016). “ZeroResource Translation with Multi-Lingual Neural Machine Translation.” In Proceedings of
EMNLP, pp. 268–277.
Graehl, J. and Knight, K. (2004). “Training Tree Transducers.” In Proceedings of NAACL,
pp. 105–112.
Heaﬁeld, K. (2011). “KenLM: Faster and Smaller Language Model Queries.” In Proceedings of
WMT, pp. 187–197.
Isozaki, H., Hirao, T., Duh, K., Sudoh, K., and Tsukada, H. (2010). “Automatic Evaluation ofTranslation Quality for Distant Language Pairs.” In Proceedings of EMNLP, pp. 944–952.
Johnson, M., Schuster, M., Le, Q., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., and Dean, J. (2017). “Google’s Multilingual Neural
Machine Translation System: Enabling Zero-Shot Translation.” TACL, 5, pp. 339–351.
Klein, P. N. (1998). “Computing the Edit-Distance Between Unrooted Ordered Trees.” In
Proceedings of of European Symposium on Algorithms, pp. 91–102.
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B.,
Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007).
“Moses: Open Source Toolkit for Statistical Machine Translation.” In Proceedings of ACL,
pp. 177–180.
Koehn, P., Och, F. J., and Marcu, D. (2003). “Statistical Phrase-Based Translation.” In Proceedings of NAACL, pp. 48–54.
Levinboim, T. and Chiang, D. (2015). “Supervised Phrase Table Triangulation with Neural Word
Embeddings for Low-Resource Languages.” In Proceedings of EMNLP, pp. 1079–1083.
626

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

Mikolov, T., Yih, W.-t., and Zweig, G. (2013). “Linguistic Regularities in Continuous Space
Word Representations.” In Proceedings of NAACL, pp. 746–751.
三浦明波，Graham Neubig，中村哲 (2016a). 木構造を中間表現とするピボット翻訳手法. 情報処
理学会 第 227 回自然言語処理研究会 (SIG-NL), 9 号, pp. 1–6.

Miura, A., Neubig, G., Sakti, S., Toda, T., and Nakamura, S. (2015). “Improving Pivot Translation by Remembering the Pivot.” In Proceedings of ACL, pp. 573–577.
三浦明波，Graham Neubig，Sakriani Sakti，戸田智基，中村哲 (2016b). 中間言語情報を記憶する
ピボット翻訳手法. 自然言語処理, 23 (5), pp. 499–528.

Miura, A., Neubig, G., Sudoh, K., and Nakamura, S. (2017). “Tree as a Pivot: Syntactic Matching
Methods in Pivot Translation.” In Proceedings of WMT, pp. 90–98.
Neubig, G. (2013). “Travatar: A Forest-to-String Machine Translation Engine based on Tree
Transducers.” In Proceedings of ACL Demo Track, pp. 91–96.
Neubig, G., Arthur, P., and Duh, K. (2015). “Multi-Target Machine Translation with MultiSynchronous Context-free Grammars.” In Proceedings of NAACL, pp. 484–491.
Neubig, G., Nakata, Y., and Mori, S. (2011). “Pointwise Prediction for Robust, Adaptable
Japanese Morphological Analysis.” In Proceedings of ACL, pp. 529–533.
Oda, Y., Neubig, G., Sakti, S., Toda, T., and Nakamura, S. (2015). “Ckylark: A More Robust
PCFG-LA Parser.” In Proceedings of NAACL, pp. 41–45.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). “BLEU: A Method for Automatic
Evaluation of Machine Translation.” In Proceedings of ACL, pp. 311–318.
Paul, M., Yamamoto, H., Sumita, E., and Nakamura, S. (2009). “On the Importance of Pivot Language Selection for Statistical Machine Translation.” In Proceedings of NAACL, pp. 221–224.
Sennrich, R., Haddow, B., and Birch, A. (2016). “Improving Neural Machine Translation Models
with Monolingual Data.” In Proceedings of ACL, pp. 86–96.
Utiyama, M. and Isahara, H. (2007). “A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation.” In Proceedings of NAACL, pp. 484–491.
Zhu, X., He, Z., Wu, H., Zhu, C., Wang, H., and Zhao, T. (2014). “Improving Pivot-Based
Statistical Machine Translation by Pivoting the Co-occurrence Count of Phrase Pairs.” In
Proceedings of EMNLP, pp. 1665–1675.
Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2016). “The United Nations Parallel
Corpus v1.0.” In Proceedings of LREC, pp. 3530–3534.
Zoph, B. and Knight, K. (2016). “Multi-Source Neural Translation.” In Proceedings of NAACL,
pp. 30–34.

627

Journal of Natural Language Processing

Vol. 25 No. 5

December 2018

Akiva Miura: received his B.Sc. from Technion - Israel Institute of Technology
in 2013, and his M.Eng. and Ph.D. in information science from Nara Instituteof Science and Technology in 2016 and 2018 respectively. From 2018, he has
been a researcher at Fujitsu Laboratories Ltd., where he is pursuing researchin machine translation and knowledge technology.
Graham Neubig: received his B.E. from University of Illinois, Urbana-Champaign
in 2005, and his M.S. and Ph.D. in informatics from Kyoto University in 2010
and 2012 respectively. He is currently an assistant professor at Carnegie Mellon University and an aﬃliate associate professor at Nara Institute of Scienceand Technology. His research interests include speech and natural language
processing, with a focus on machine learning approaches for applications suchas machine translation, speech recognition, and spoken dialog.
Katsuhito Sudoh: is an associate professor of Nara Institute of Science andTechnology. He received a bachelor’s degree in engineering in 2000, and amaster’s and Ph.D. degree in informatics in 2002 and 2015, respectively, from
Kyoto University. He was in NTT Communication Science Laboratories from2002 to 2017. He currently works on machine translation and natural language
processing. He is a member of the Association for Computational Linguistics(ACL), the Association of Natural Language Processing (ANLP), the Information Processing Society of Japan (IPSJ) and the Acoustical Society of Japan(ASJ)
Satoshi Nakamura: is Professor of Graduate School of Information Science,
Nara Institute of Science and Technology, Honorarprofessor of Karlsruhe Institute of Technology, Germany, and ATR Fellow. He received his B.S. from
Kyoto Institute of Technology in 1981 and Ph.D. from Kyoto University in
1992. He was Director of ATR Spoken Language Communication ResearchLaboratories in 2000–2008 and Vice president of ATR in 2007–2008. He was
Director General of Keihanna Research Laboratories and the Executive Director of Knowledge Creating Communication Research Center, National Instituteof Information and Communications Technology in 2009–2010. He is currently
Director of Augmented Human Communication laboratory and a full professor of Graduate School of Information Science at Nara Institute of Science andTechnology. He has been Elected Board Member of International Speech Com628

Miura, Neubig, Sudoh and Nakamura

Syntactic Matching Methods in Pivot Translation

munication Association, ISCA, since June 2011 and IEEE Signal Processing
Magazine Editorial Board member since April 2012, IEEE SPS Speech andLanguage Technical Committee Member since 2013, and IEEE Fellow since
2016.
(Received May 1, 2018)(Revised August 16, 2018)(Accepted September 18, 2018)629