Improving Pivot Translation by Remembering the Pivot
Akiva Miura, Graham Neubig, Sakriani Sakti, Tomoki Toda, Satoshi Nakamura
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama-cho, Ikoma-shi, Nara, Japan{miura.akiba.lr9, neubig, ssakti, tomoki, s-nakamura}@is.naist.jp

Abstract
Pivot translation allows for translation of
language pairs with little or no parallel
data by introducing a third language forwhich data exists. In particular, the triangulation method, which translates by combining source-pivot and pivot-target translation models into a source-target model,
is known for its high translation accuracy.
However, in the conventional triangulation
method, information of pivot phrases isforgotten and not used in the translation
process. In this paper, we propose a novel
approach to remember the pivot phrases inthe triangulation stage, and use a pivot language model as an additional information
source at translation time. Experimental results on the Europarl corpus showed
gains of 0.4-1.2 BLEU points in all tested
combinations of languages1 .

1Introduction

In statistical machine translation (SMT) (Brown etal., 1993), it is known that translation with models trained on larger parallel corpora can achieve
greater accuracy (Dyer et al., 2008). Unfortunately, large bilingual corpora are not readily
available for many language pairs, particularly
those that don’t include English. One effective solution to overcome the scarceness of bilingual datais to introduce a pivot language for which parallel
data with the source and target languages exists(de Gispert and Mariño, 2006).
Among various methods using pivot languages,
the triangulation method (Cohn and Lapata, 2007;
Utiyama and Isahara, 2007; Zhu et al., 2014),
which translates by combining source-pivot andpivot-target translation models into a source-target
1
Code to replicate the experiments can be found athttps://github.com/akivajp/acl2015

Annäherung	

approach	approccio	

Ansatz	approximation	accesso	Zufahrt	entrance	ravvicinamento	

(a) Triangulation (de-en-it)
Annäherung	(via: approach)	

approccio	

Annäherung	
ravvicinamento	(via: approach, approximation)	
・ ・ ・	(b) Traditional Triangulated Phrases
Annäherung	Annäherung	Annäherung	〈approccio, approach〉	
〈ravvicinamento, approach〉	
〈ravvicinamento, approximation〉	
・ ・ ・	(c) Proposed Triangulated Phrases
Figure 1: An example of (a) triangulation and theresulting phrases in the (b) traditional method offorgetting pivots and (c) our proposed method ofremembering pivots.

model, has been shown to be one of the most effective approaches. However, word sense ambiguity
and interlingual differences of word usage cause
difficulty in accurately learning correspondences
between source and target phrases.
Figure 1 (a) shows an example of three words
in German and Italian that each correspond to theEnglish polysemic word “approach.” In such acase, finding associated source-target phrase pairs
and estimating translation probabilities properly
becomes a complicated problem. Furthermore, inthe conventional triangulation method, information about pivot phrases that behave as bridges between source and target phrases is lost after learning phrase pairs, as shown in Figure 1 (b).
To overcome these problems, we propose anovel triangulation method that remembers thepivot phrase connecting source and target in therecords of phrase/rule table, and estimates a joint
translation probability from the source to target

and pivot simultaneously. We show an example
in Figure 1 (c). The advantage of this approachis that generally we can obtain rich monolingual
resources in pivot languages such as English, andSMT can utilize this additional information to improve the translation quality.
To utilize information about the pivot language
at translation time, we train a Multi-SynchronousContext-free Grammar (MSCFG) (Neubig et al.,2015), a generalized extension of synchronous
CFGs (SCFGs) (Chiang, 2007), that can generate strings in multiple languages at the same time.
To create the MSCFG, we triangulate source-pivot
and pivot-target SCFG rule tables not into a single
source-target SCFG, but into a source-target-pivot
MSCFG rule table that remembers the pivot. During decoding, we use language models over boththe target and the pivot to assess the naturalness ofthe derivation. We perform experiments on pivot
translation of Europarl proceedings, which showthat our method indeed provide significant gains inaccuracy (of up to 1.2 BLEU points), in all combinations of 4 languages with English as a pivot
language.

2

Translation Formalisms

2.1 Synchronous Context-free Grammars
First, we cover SCFGs, which are widely usedin machine translation, particularly hierarchicalphrase-based translation (Hiero; Chiang (2007)).
In SCFGs, the elementary structures are rewrite
rules with aligned pairs of right-hand sides:⟩⟨(1)
X → s, twhere X is the head of the rewrite rule, and s and t
are both strings of terminals and non-terminals inthe source and target side respectively. Each stringin the right side tuple has the same number of indexed non-terminals, and identically indexed nonterminals correspond to each-other. For example,
a synchronous rule could take the form of:
X → ⟨X0 of the X1 , X1 的 X0 ⟩ .

(2)In the SCFG training method proposed by
Chiang (2007), SCFG rules are extracted basedon parallel sentences and automatically obtained
word alignments. Each extracted rule is scoredwith phrase translation probabilities in both directions ϕ(s|t) and ϕ(t|s), lexical translation probabilities in both directions ϕlex (s|t) and ϕlex (t|s),a word penalty counting the terminals in t, and aconstant phrase penalty of 1.
At translation time, the decoder searches forthe target sentence that maximizes the derivation
probability, which is defined as the sum of thescores of the rules used in the derivation, and thelog of the language model probability over the target strings. When not considering an LM, it is possible to efficiently find the best translation for an
input sentence using the CKY+ algorithm (Chappelier et al., 1998). When using an LM, the expanded search space is further reduced based on alimit on expanded edges, or total states per span,through a procedure such as cube pruning (Chiang, 2007).
2.2 Multi-Synchronous CFGs
MSCFGs (Neubig et al., 2015) are a generalizationof SCFGs that are be able to generate sentences in
multiple target languages simultaneously. The single target side string t in the SCFG production ruleis extended to have strings for N target languages:⟨⟩
X → s, t1 , ..., tN .
(3)
Performing multi-target translation withMSCFGs is quite similar to translating using
standard SCFGs, with the exception of the expanded state space caused by having one LM
for each target. Neubig et al. (2015) propose asequential search method, that ensures diversity inthe primary target search space by first expanding
with only primary target LM, then additionallyexpands the states for other LMs, a strategy wealso adopt in this work.
In the standard training method for MSCFGs,
the multi-target rewrite rules are extracted frommultilingual line-aligned corpora by applying an
extended version of the standard SCFG rule extraction method, and scored with features that consider the multiple targets. It should be noted that
this training method requires a large amount ofline-aligned training data including the source andall target languages. This assumption breaks downwhen we have little parallel data, and thereby we
propose a method to generate MSCFG rules by
triangulating 2 SCFG rule tables in the following
section.

3 Pivot Translation Methods
Several methods have been proposed for SMT using pivot languages. These include cascade methods that consecutively translate from source topivot then pivot to target (de Gispert and Mariño,2006), synthetic data methods that machinetranslate the training data to generate a pseudoparallel corpus (de Gispert and Mariño, 2006),
and triangulation methods that obtain a sourcetarget phrase/rule table by merging source-pivot
and pivot-target table entries with identical pivot
language phrases (Cohn and Lapata, 2007). In particular, the triangulation method is notable for producing higher quality translation results than otherpivot methods (Utiyama and Isahara, 2007), so we
use it as a base for our work.
3.1 Traditional Triangulation Method
In the triangulation method by Cohn and Lapata(2007), we first train source-pivot and pivot-target
rule tables, then create rules:⟨⟩
X → s, t(4)

if there exists a pivot phrase p such that the pair⟨s,
⟨ p⟩⟩ is in source-pivot table TSP and the pairp, t is in pivot-target table TP T . Source-target
table TST is created by calculation of the translation probabilities using phrase translation probabilities ϕ(·) and lexical translation probabilities
ϕlex (·) for all connected phrases according to thefollowing equations (Cohn and Lapata, 2007):
( )
ϕ t|s
( )
ϕ s|t
( )
ϕlex t|s(ϕlex s|t)=∑p∈TSP ∩TP T=∑

p∈TSP ∩TP T=∑

p∈TSP ∩TP T

=

∑

( )
ϕ t|p ϕ (p|s) ,

(5)

( )
ϕ (s|p) ϕ p|t ,

(6)

( )
ϕlex t|p ϕlex (p|s) , (7)( )ϕlex (s|p) ϕlex p|t . (8)p∈TSP ∩TP T

The equations (5)-(8) are based on the( memo)ϕt|p,srylesschannelmodel,
which
assumes=()( )ϕ t|p and ϕ s|p, t = ϕ (s|p). Unfortunately,
these equations are not accurate due to polysemy
and disconnects in the grammar of the languages.
As a result, pivot translation is significantly more
ambiguous than standard translation.
3.2 Proposed Triangulation Method
To help reduce this ambiguity, our proposed triangulation method remembers the corresponding
pivot phrase as additional information to be utilized for disambiguation. Specifically, instead of
marginalizing over the pivot phrase p, we create an

MSCFG rule for the tuple of the connected sourcetarget-pivot phrases such as:⟩⟨
X → s, t, p .

(9)

The advantage of translation with these rules isthat they allow for incorporation of additional features over the pivot sentence such as a strong pivot
LM.
In addition to the equations (5)-(8), we also estimate translation probabilities ϕ(t, p|s), ϕ(s|p, t)that consider both target and pivot phrases at thesame time according to:)( )(ϕ t, p|s = ϕ t|p ϕ (p|s) ,()ϕ s|p, t = ϕ (s|p) .

(10)
(11)

Translation probabilities between source and pivot
phrases ϕ(p|s), ϕ(s|p), ϕlex (p|s), ϕlex (s|p) can
also be used directly from the source-pivot rule table. This results in 13 features for each MSCFG
rule: 10 translation probabilities, 2 word penalties
counting the terminals in t and p, and a constantphrase penalty of 1.
It should be noted that remembering the pivot
results in significantly larger rule tables. To savecomputational resources, several pruning methods
are conceivable. Neubig et al. (2015) show that aneffective pruning method in the case of a main target T1 with the help of target T2 is the T1 -pruning
method, namely, using L candidates of t1 with thehighest translation probability ϕ(t1 |s) and selecting t2 with highest ϕ(t1 , t2 |s) for each t1 . We follow this approach, using the L best t, and the corresponding 1 best p .

4 Experiments
4.1 Experimental Setup
We evaluate the proposed triangulation method
through pivot translation experiments on the Europarl corpus, which is a multilingual corpus including 21 European languages (Koehn, 2005)
widely used in pivot translation work. In ourwork, we perform translation among German (de),Spanish (es), French (fr) and Italian (it), with English (en) as the pivot language. To prepare thedata for these 5 languages, we first use the GaleChurch alignment algorithm (Gale and Church,1993) to retrieve a multilingual line-aligned corpus of about 900k sentences, then hold out 1,500
sentences each for tuning and test. In our basic

Source

de
es
fr
it

Target
es
fr
it
de
fr
it
de
es
it
de
es
fr

DirectCascade

27.10
25.65
23.04
20.11
33.48
27.82
19.69
34.36
28.48
19.09
31.99
31.39

25.05
23.86
20.76
18.52
27.00
22.57
18.01
27.26
22.73
14.03
25.64
25.87

Tri. SCFG(baseline)
25.31
24.12
21.27
18.77
29.54
25.11
18.73
30.31
25.31
17.35
28.85
28.48

BLEU Score [%]Tri. MSCFG
Tri. MSCFG-PivotLM+PivotLM 100k
25.38
25.52
24.16
24.25
21.42
† 21.65
18.97
19.08† 29.87† 29.91
25.01
25.18
18.77
18.87
30.53
† 30.7325.50† 25.72† 17.99‡ 18.17
28.83
29.01
28.40
28.63

Tri. MSCFG+PivotLM 2M† 25.75† 24.58‡ 22.29† 19.40† 29.95‡ 25.64† 19.19‡ 31.00‡ 26.22‡ 18.52† 29.31† 29.02

Table 1: Results for each method. Bold indicates the highest BLEU score in pivot translation, anddaggers indicate statistically significant gains over Tri. SCFG († : p < 0.05, ‡ : p < 0.01)

training setup, we use 100k sentences for training both the TMs and the target LMs. We assume that in many situations, a large amount ofEnglish monolingual data is readily available andtherefore, we train pivot LMs with different data
sizes up to 2M sentences.
As a decoder, we use Travatar (Neubig, 2013),
and train SCFG TMs with its Hiero extraction
code. Translation results are evaluated by BLEU
(Papineni et al., 2002) and we tuned to maximize BLEU scores using MERT (Och, 2003). For
trained and triangulated TMs, we use T1 rule pruning with a limit of 20 rules per source rule. For
decoding using MSCFG, we adopt the sequential
search method.
We evaluate 6 translation methods:
Direct: Translating with a direct SCFG trained on
the source-target parallel corpus (not using apivot language) for comparison.
Cascade: Cascading source-pivot and pivottarget translation systems.
Tri. SCFG: Triangulating source-pivot andpivot-target SCFG TMs into a source-target
SCFG TM using the traditional method.
Tri. MSCFG: Triangulating source-pivot andpivot-target SCFG TMs into a sourcetarget-pivot MSCFG TM in our approach.
-PivotLM indicates translating without apivot LM and +PivotLM 100k/2M indicatesa pivot LM trained using 100k/2M sentences
respectively.

4.2 Experimental Results
The result of experiments using all combinationsof pivot translation tasks for 4 languages via English is shown in Table 1. From the results, we cansee that the proposed triangulation method considering pivot LMs outperforms the traditional triangulation method for all language pairs, and translation with larger pivot LMs improves the BLEU
scores. For all languages, the pivot-remembering
triangulation method with the pivot LM trainedwith 2M sentences achieves the highest score of
the pivot translation methods, with gains of 0.41.2 BLEU points from the baseline method. Thisshows that remembering the pivot and using it
to disambiguate results is consistently effective in
improving translation accuracy.
We can also see that the MSCFG triangulated
model without using the pivot LM slightly outperforms the standard SCFG triangulation method forthe majority of language pairs. It is conceivablethat the additional scores of translation probabilities with pivot phrases are effective features thatallow for more accurate rule selection.
Finally, we show an example of a translated sentence for which pivot-side ambiguity is resolved in
the proposed triangulation method:
Input (German): ich bedaure , daß es keine
gemeinsame annäherung gegeben hat .
Reference (Italian): sono spiacente del mancato
approccio comune .
Tri. SCFG: mi rammarico per il fatto che non si
ravvicinamento comune . (BLEU+1: 13.84)
Tri. MSCFG+PivotLM 2M:

mi dispiace che non esiste un approccio comune . (BLEU+1: 25.10)
i regret that there is no common approach .
(Generated English Sentence)
The derivation uses an MSCFG rule connecting“approccio” to “approach” in the pivot, and we
can consider that appropriate selection of English
words according to the context contributes to selecting relevant vocabulary in Italian.

5

Conclusion

In this paper, we have proposed a method for pivot
translation using triangulation of SCFG rule tables into an MSCFG rule table that remembers thepivot, and performing translation with pivot LMs.
In experiments, we found that these models are
effective in the case when a strong pivot LM exists. In the future, we plan to explore more refined
methods to devising effective intermediate expressions, and improve estimation of probabilities fortriangulated rules.

Acknowledgements
The authors thank anonymous reviewers for helpful suggestions. This work was supported in partby the Microsoft CORE project.

References
Peter F. Brown, Vincent J.Della Pietra, Stephen
A. Della Pietra, and Robert L. Mercer. 1993.
The Mathematics of Statistical Machine Translation:
Parameter Estimation. Computational Linguistics,
19:263–312.
Jean-Cédric Chappelier, Martin Rajman, et al. 1998. A
Generalized CYK Algorithm for Parsing Stochastic
CFG. TAPD, 98(133-137):5.
David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228.
Trevor Cohn and Mirella Lapata. 2007. MachineTranslation by Triangulation: Making Effective Useof Multi-Parallel Corpora. In Proc. ACL, pages 728–
735, June.
Adrià de Gispert and José B. Mariño. 2006. CatalanEnglish Statistical Machine Translation without Parallel Corpus: Bridging through Spanish. In Proc.
of LREC 5th Workshop on Strategies for developing
machine translation for minority languages.
Christopher Dyer, Aaron Cordova, Alex Mont, andJimmy Lin. 2008. Fast, easy, and cheap: construction of statistical machine translation models with
MapReduce. In Proc. WMT, pages 199–207.

William A Gale and Kenneth W Church. 1993. A
program for aligning sentences in bilingual corpora.
Computational linguistics, 19(1):75–102.
Philipp Koehn. 2005. Europarl: A parallel corpus forstatistical machine translation. In MT summit, volume 5, pages 79–86.
Graham Neubig, Philip Arthur, and Kevin Duh.
2015. Multi-Target Machine Translation with MultiSynchronous Context-free Grammars. In Proc.
NAACL.
Graham Neubig. 2013. Travatar: A Forest-to-String
Machine Translation Engine based on Tree Transducers. In Proc. ACL Demo Track, pages 91–96.
Franz Josef Och. 2003. Minimum Error Rate Training
in Statistical Machine Translation. In Proc. ACL,
pages 160–167.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: a method for automatic
evaluation of machine translation. In Proc. ACL,
pages 311–318.
Masao Utiyama and Hitoshi Isahara. 2007. A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation. In Proc. NAACL, pages
484–491.
Xiaoning Zhu, Zhongjun He, Hua Wu, Conghui Zhu,
Haifeng Wang, and Tiejun Zhao. 2014. Improving
Pivot-Based Statistical Machine Translation by Pivoting the Co-occurrence Count of Phrase Pairs. In
Proc. EMNLP.