Statistical Machine Translation with Terminology

Tsuyoshi Okita, Andy Way
CNGL / School of Computing, Dublin City University,{tokita,away}@computing.dcu.ie

Abstract
This paper considers a scenario which isslightly different from Statistical MachineTranslation (SMT) in that we are given
almost perfect knowledge about bilingual terminology, considering the situation when a Japanese patent is applied
to or granted by the Japanese Patent Office (JPO). Technically, we incorporate
bilingual terminology into Phrase-based
SMT (PB-SMT) focusing on the statistical properties of them. The first modification is made on the word aligner
which incorporates knowledge about terminology as prior knowledge. The second modification is made both on thelanguage modeling and the translation
modeling which reflect the hierarchicalstructure of bilingual terminology, that is
the non-compound characteristics of thephrases, using the Pitman-Yor processbased smoothing methods. Using 200k
JP–EN NTCIR corpus, our experimental
results show that the overall improvementof this method was 1.10 BLEU point absolute and 5% relative.

1 Introduction
This paper examines the situation when aJapanese patent is applied to or granted by theJapanese Patent Office (JPO) without being translated into English. Our target is to translate aJapanese patent from Japanese into English. Twonotable characteristics in this context are that weassume that we know perfect knowledge aboutbilingual terminology in the training / development / test corpora and that we know their correspondences in these corpora as well. In practicalsituation, this would be quite realistic: it is often
possible by borrowing human knowledge to findout the correct terminology in English although it
may be difficult for an ordinary Japanese to construct a correct English sentence.
We intend to assist this by Statistical MachineTranslation (SMT) (Brown et al., 1993; Marcu
and Wong, 2002; Chiang, 2005; Koehn, 2010).
Without loss of generality, we limit ourselves todiscuss it with Phrase-Based SMT (PB-SMT).
PB-SMT consists of translation modeling, which
is word alignment (Brown et al., 1993) followedby phrase extraction (Och and Ney, 2003a), language modeling (Stolcke, 2002), Minimum-Error
Rate Training (MERT) (Och and Ney, 2003b), anddecoding (Koehn et al., 2007). The keys to handle
this situation will be summarized by two issues
below.
Firstly, the prior knowledge about bilingual terminology may not be effectively employed by theconventional word aligner (Och and Ney, 2003a)in translation modeling process. Bilingual terminology, in general, consists of n-to-m correspondences between the source terminology and thetarget terminology. Since word alignment aims
at obtaining 1-to-n or n-to-1 correspondences, it
is in fact problematic to obtain such correspondences after phrase extraction. Naive way to incorporate this is to do a grouping of terminology
into one word in both sides. Since a word aligner
does not recognize these word correspondences,
there is no guarantee that a word aligner detects
these word correspondences.
41Secondly, the prior knowledge about bilingual
terminology effects the output of translation modeling. It is a well known fact that since the relative frequency or maximum likelihood estimate
does not consider zero frequencies, the less frequent items will obtain with larger probability. It
is likely that the prior knowledge about bilingual
terminology will emphasize this phenomenon further since the resulted phrase pairs become lessfrequent due to this prior knowledge. Foster et al.
(Foster, 06) applied various smoothing techniqueto translation model.
This paper is organized as follows. Section
2 introduces an MWE-sensitive word aligner for
the first problem. In Section 3 we mention
the smoothing technique based on the hierarchical Pitman-Yor process, which solves the second
problem. Experimental results are presented in
Section 4 by combining these techniques. Section 5 concludes and provides avenues for further
research.

2 Translation Modeling with Prior
Knowledge
As a word aligner, we use an MWE-sensitive
word aligner to incorporate prior knowledge aboutbilingual terminology (Okita et al., 2010). Here,
we do not need to extract MWEs, but such MWEs
are already assumed to be given as bilingul terminology.
The EM algorithm-based word aligner uses
maximum likelihood in its M-step.
Our
method replaces this maximum likelihood estimate (shown in Equation (1)) with the MAP(Maximum A Posteriori) estimate (shown inEquation (2)), which is a basic Bayesian machinelearning method. Let t be a lexical translation
probability t(e|f ); note that often t is omitted inword alignment literature but for our purposes thisneeds to be explicit.
EEXH :

q(z; x) =p(z|x; θ)arg maxtXq(z|x) log p(x, z; t)x,z+ log p(t)(2)

Then, the prior log p(t), a probability used to reflect the degree of prior belief about the occurrences of the events, can embed prior knowledge
about MWEs.
A prior for IBM Model 1 considers all possible alignments exhaustively in E-Step as in thedefinition of EM algorithm (while IBM Model
3 and 4 only sample a neighborhood alignments
around the best alignment). Let us give information about alignment link between e and f
by T = {(sentID, ti , tj , posi , posj ), . . . , } into
prior. The prior p(t) = p(t; e, f, T ) for given
word e and f in a sentence is defined simply 1
if they have alignment link, 0 if they are not connected, and uniform if their link is not known:(ei = ti , fj = tj )
 10
(ei = ti , fj 6= tj )p(t; ei , fi , T ) =0(ei 6= ti , fj = tj )uniform(ei 6= ti , fj 6= tj )Then we embed this prior in the M-step of EM algorithm where we replaced its likelihood estimate
with MAP estimate (Okita et al., 2010). Although
this is for the case of IBM Model 1, IBM Models
3 and 4 are essentially the same except that they
are not proper. Due to the space problems here,
further details can be found in (Okita et al., 2010).

3

Language Model and Translation
Model Smoothing

This section describes the statistical smoothing
method based on hierarchical Pitman-Yor processes, which is a nonparametric generalizationof the Dirichlet distribution that produces powerlaw distributions (Teh, 2006; Goldwater et al.,2006). Hence, this smoothing method of hierarchical Pitman-Yor processes does a smoothingtask under the prior knowledge that the underlying distribution has power-law properties.

3.1 Language Model Smoothingt′ = arg max Q(t, told ) =t
X
Various pieces of research have been carriedarg maxq(z|x) log p(x, z; t) (1) out in which hierarchical Pitman-Yor processestx,zhave been applied to language models (Hierarchi′
MAPM: t = arg max Q(t, told ) + log p(t) = cal Pitman-Yor Language Model (HPYLM) (Teh,tMMLE :

42

2006; Mochihashi and Sumita, 2007; Huang andRenals, 2009)). This model is shown to be superior to the interpolated Kneser-Ney methods(Kneser and Ney, 1995) and comparable to themodified Kneser Ney methods in terms of perplexity. (Okita and Way, 2010) empirically verified that HPYLM improves BLEU score as wellalthough it employed a slight modification on
the decoding process. Following descriptions are
based on various literatures (Teh, 2006; Mochihashi and Sumita, 2007; Mochihashi et al., 2009;
Okita and Way, 2010).
HPYLM: Generative Model The Pitman-Yor
process (Pitman, 1995) is defined as a threeparametric distribution P Y (d, θ, G0 ) where d denotes a discount parameter, θ a strength parameter,
and G0 a base distribution. One nice property isthat the Pitman-Yor process is known to produce a
power-law distribution: the more words have been
assigned to a draw from G0 , the more likely subsequent words will be assigned to the draw, while
the more we draw from G0 , the more likely a newword will be assigned to a new draw from G0 .
HPYLM is constructed in the following way
encoding such nice property of the power-law distribution. A graphical model of HPYLM is shownin Figure 1. Firstly, the Pitman-Yor process isplaced as a prior in the generative model. For agiven a context u, let Gu (w) be the probability of
the current word taking value w. Using a PitmanYor process as the prior for Gu [Gu (w)]w∈W as in(3):
Gu |d|u| , θ|u| , Gπ(u) ∼ P Y (d|u| , θ|u| , Gπ(u) ) (3)where π(u) is a function whose parameter is acontext u, the discount and strength parameters
are functions of the length |u| of the context, while
the mean vector is Gπ(u) , the vector of probabilities of the current word given all but the earliest
word in the context.
Secondly, π(u) is defined as the suffix of u consisting of all but the earliest word in Equation (3)as (Teh, 2006). This signifies that u is n-gramwords and π(u) is (n-1)-gram words; this induction of Equation (3) makes an n-gram hierarchy.
Thirdly, as the last sentence suggests, such aprior of the Pitman-Yor processes is placed re-G_0theta_0

G_o

d_0

theta_u

d_u

G_u

X_ui
i=1,...,n_unFigure 1: Graphical model of hierarchical PitmanYor process.
cursively over Gπ(u) in the generative model as
in Equation (4): Gu |d|u| , θ|u| , Gπ(u) ∼ P Y (d|u| , θ|u| , Gπ(u) )...
G∅ |d0 , θ0 , G0 ∼ P Y (d0 , θ0 , G0 )This is repeated until we get to G0 , the vector
of probabilities over the current word given theempty context ∅. G0 is the global mean vector, given a uniform value of G0 = 1/V for allw ∈ W.
HPYLM: Inference One procedure to do an inference in order to generate words drawn from G
is called Chinese restaurant process, which iteratively marginalizes out G. Note that when the
vocabulary is finite, P Y (d, θ, G0 ) has no known
analytic form.
We assume the language modelling. Let h be
an n-gram context; for example in 3-gram, this is
h = {w1 , w2 }. A Chinese restaurant contains an
infinite number of tables t, each with infinite seating capacity. Customers, which are the n-gramcounts c(w|h), enter the restaurant and seat themselves over the tables 1, . . . , thw· . The first customer sits at the first available table, while eachof the subsequent customers sits at an occupied
table with probability proportional to the numberof customers already sitting there chwk − d, or ata new unoccupied table with probability proportional to θ + d · th· as is shown in (5):
w|h ∼

43



chwk − d (1 ≤ k ≤ thw ).
θ + d · th· (k = new).

(5)

(4)

3.2

Translation Model Smoothing

An n-gram is often defined as a subsequence of n
items from a given sequence where items can bephonemes, syllables, letters, words or base pairs.
Although we can extend this definition of n-gramto the one which includes ‘phrases’, let us use thedifferent term ‘n-phrase-gram’ instead in this paper, in order not to mix up with the n-gram forwords. Fig. 2 shows a typical example of phrase
extraction process. In this process, under the consistency constrained, phrase pairs are extracted
which is depicted in the center. Note that this figure is depicted separating the source and the target
sides.
Fig. 3 shows the same figure if we depict themin pairs. The lowest column includes only 1phrase-grams, the second lowest column includes

house

dass erdasser

im hausbleibthaus

bleibtmichael
geht
davon
aus

,

im haus bleibt

imgeht davon aus

dass
er
im
haus

in the housein theassumesdass er im haus bleibt

mchaelassumes
that
he
will
stay
in
the
house

michaelwill stay in the housewill stayhethat hemichael geht davon aus

that

that he will stay in the housemichael geht davon aus , dassmichaelmichael assumes

michael assumes that

where p(w|h′ ) is the same probability using a (n1)-gram context h′ . The case when thw = 1 corresponds to an interpolated Kneser-Ney smoothing(Kneser and Ney, 1995).
Implementation of this inference procedure relates to the Markov chain Monte Carlo sampling.
The simplest way is to build a Gibbs sampler
which randomly selects n-gram words, draws abinary decision as to which (n − 1)-gram words
originated from, and updates the language modelaccording to the new lower-order n-grams (Goldwater et al., 2006). A blocked Gibbs sampler is
proposed by Mochihashi et al. (Mochihashi et al.,2009), which is originally proposed for segmentation. This algorithm is an iterative procedure,
which randomly selects a n-gram word, removes
the “sentence” data of this n-gram word, and updates by adding a new “sentence” according to the
new n-grams. This procedure is expected to mix
rapidly compared to the simple Gibbs sampler.

michael geht davon aus , dass er

michael assumes that heθ + d · th·c(w|h) − d · thw
p(w|h) =+θ + c(h)θ + c(h)′p(w|h )
(6)

michael geht davon aus, dass er im haus bleibt

michael assumes that he will stay in the house

where chwk is the number of P
customers seated attable k until now, and th· = w thw is the total
number of tables in h.
Hence, the predictive distribution of n-gramprobability in HPYLM is recursively calculated asin Equation (6):Figure 2: A toy example of phrase extraction process. Resulted phrase pairs can be described as alattice structure.
2-phrase-grams, and so on. The line connecting
two nodes indicates parent-child relations. Then,
this becomes the lattice structure of the generated
phrase pairs. These generated phrase pairs may
have several paths to yield the whole sentences.
As is similar with HPYLM, we can limit this byconsidering the suffix of a sequence, meaning that
we can process a sequence always from left-toright. Hence, although the natural lattice would
include the dashed lines, the dashed lines can beeliminated if we impose constraint that we should
always read the suffix of this sequence from leftto-right. This constraint makes the resulted structure a tree. If the resulted structure is a tree, we
can employ the same strategy with HPYLM. The
predictive distribution can be calculated by Equation (6) with the replacement of n-grams with nphrase-grams.

4

Experimental Results

Our baseline was a standard log-linear PB-SMT
system based on Moses. The GIZA++ implementation (Och and Ney, 2003a) of IBM Model
4 was used for word alignment. For phrase extraction the grow-diag-final heuristics described in(Och and Ney, 2003a) was used to derive the refined alignment. We then performed MERT process which optimizes the BLEU metric, while a
5-gram language model was derived with Kneser44

Lattice structure (Tree structure if we only accept real lines)
michael assumes that he will stay in the house|| michael geht davon aus, dass er im haus bleibt

michael assumes that he || michael geht davon aus , dass erthat he will stay in the house ||michael assumes that || michael geht davon aus , dasswill stay in the house ||dass er im haus bleibt

im haus bleibtthat he || dass er
michael assumes || michael geht davon aus
in the house || im haus2−phrase−grammichael || michaelthat || dassassumes || geht davon aus

he || erin the || im1−phrase−gramhouse || haus

will stay || bleibt

Figure 3:
Ney smoothing trained with SRILM on the English side of the training data. We used Moses fordecoding.
We used NTCIR-8 patent corpus for JP–EN(Fujii et al., 2010). We randomly selected 200k
sentence pairs as training corpus. For JP–EN
patent corpus, we used 1.2k sentence for development set while we used a test set prepared for
NTCIR-8 evaluation campaign. We prepared terminology in this way: we extracted bilingual terminology by heuristic MWE-extraction strategy
which we described in (Okita et al., 2010). Then,
we corrected these extracted terminology by handinspecting the corpora.
JP–EN
Base
MWE only
Smoothing only
Both

BLEU
21.68
22.48
22.44
22.78

Table 1: Results for 200k JP–EN sentences
Table 1 shows our results. The improvement of
BLEU by the modification on word aligner was
0.80 BLEU point absolute and 3.6% relative, themodification on language model and on translation model by the hierarchical Pitman-Yor process
was 0.76 BLEU point absolute and 3.5% relative.
Finally, the overall method was 1.10 BLEU point
absolute and 5.0% relative.

5 Conclusion and Further Studies
This paper considered a scenario in SMT that weare given the perfect knowledge about bilingual
terminology. This scenario consisted of two mod-ifications of PB-SMT. Firstly, we modified a wordaligner in order to incorporate prior knowledge
about bilingual terminology. Secondly, we employed the statistical smoothing technique both on
language model and translation model. We obtained the improvement of 1.10 BLEU point absolute and 5.0% relative for this settings.
There are several avenues for further research.
Firstly, this paper considers the situation where
we have prior knowledge about bilingual terminology. Although we discussed how to incorporate this prior knowledge into a word aligner,
we did not discuss how to incorporate this priorknowledge into a decoder. In fact, this seems to
be a lucky situation since the stack decoding algorithm (Koehn, 2010) was capable of handling
this situation although the search space of decoding process was reduced. More generic case ofprior knowledge may be worth trying. For example, suppose that we have bilingual sentence patterns a priori. In this case, it will raise the necessity of modifying the decoding algorithm in order
to incorporate such prior knowledge. Secondly,
this paper considers the in-domain prior knowledge about terminology in terms of training corpus. it would be interesting to see whether theapproach of hierarchical Pitman-Yor process may
work as well for the out-of-domain prior knowledge although our view is that this may be far beyond the reach since this approach can be applied
to a smoothing task, but not to a domain adaptation task.

6

Acknowledgments

This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of45

the Centre for Next Generation Localisation(http://www.cngl.ie) at Dublin City University. We would also like to thank the Irish Centre for High-End Computing.

References
Bishop, Christopher M. 2006. Pattern Recognition
and Machine Learning. Springer. Cambridge, UK
Brown, Peter F., Vincent J. D. Pietra, Stephen A. D.
Pietra, Robert L. Mercer. 1993. The Mathematics
of Statistical Machine Translation: Parameter Estimation. Computational Linguistics. 19(2), pp. 263–
311.
Callison-Burch, Chris, David Talbot and Miles Osborne. 2004. Statistical Machine Translation with
Word- and Sentence-Aligned Parallel Corpora. In
Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL’04),
Main Volume. Barcelona, Spain, pp. 175–182.
Chiang, David. 2005. A hierarchical phrase-basedmodel for statistical machine translation. In Proceedings of the 41nd Annual Meeting of the Association for Computational Linguistics (ACL05). pp.
263-270.
Foster, George and Roland Kuhn and Howard Johnson,
2006. Phrasetable Smoothing for Statistical Machine Translation. In Proceedings of the Empirical
Methods in Natural Language Processing (EMNLP
2006), pp. 53– 61.
Fujii, Atsushi, Masao Utiyama, Mikio Yamamoto,Takehito Utsuro, Terumasa Ehara, Hiroshi Echizenya, Sayori Shimohata. 2010. Overview of the
Patent Translation Task at the NTCIR-8 Workshop.
Proceedings of the 8th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answeringand Cross-lingual Information Access, pp. 293–302.
Gale, William, and Ken Church. 1991. A Program for
Aligning Sentences in Bilingual Corpora. In Proceedings of the 29th Annual Meeting of the Association for Computational Linguistics. Berkeley CA,
pp. 177–184.
Goldwater, Sharon, Tom L. Griffiths, and Marc Johnson. “Contextual dependencies in unsupervised
word segmentation”. In Proceedings of Conference on Computational Linguistics / Association
for Computational Linguistics (COLING-ACL06).
Sydney, Australia. July, 2006. pp. 673–680.
Huang, Songang and Steve Renals. “Hierarchical
Bayesian Language Models for Conversational

Speech Recognition”. IEEE Transactions on Audio,
Speech and Language Processing, 2009. 18:8, pp.
1941–1954.
Kneser, Reinhard and Herman Ney, 1995. Improved
backing-off for m-gram language modeling. Proceedings of the IEEE International Conference on
Acoustics, Speech and Signal Processing, vol. 1,
Detroit, MI, 1995, pp. 181–184.
Koehn, Philipp, Franz Och, Daniel Marcu. 2003. Statistical Phrase-Based Translation. In Proceedingsof the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. Edmonton,
Canada. pp. 115–124.
Koehn, Philipp. 2005. Europarl: A Parallel Corpus
for Statistical Machine Translation. In Conference
Proceedings: the tenth Machine Translation Summit. Phuket, Thailand, pp.79-86.
Koehn, Philipp, H. Hoang, A. Birch, C. CallisonBurch, M. Federico, N. Bertoldi, B. Cowan,
W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar,
A. Constantin, and E. Herbst, 2007. Moses: Open
source toolkit for Statistical Machine Translation.
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion
Volume Proceedings of the Demo and Poster Sessions, Prague, Czech Republic, pp. 177–180.
Koehn, Philipp. 2010. Statistical Machine Translation. Cambridge University Press. Cambridge, UK.
Kupiec, Julian. 1993. An Algorithm for finding NounPhrase Correspondences in Bilingual Corpora. In
Proceedings of the 31st Annual Meeting of Association for Computational Linguistics. Columbus. OH.
pp. 17–22.
Marcu, Daniel and William Wong. 2002. A PhraseBased, Joint Probability Model for Statistical Machine Translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-2002), Philadelphia,
PA. pp.133–139.
McLachlan, Geoffrey J. and Thriyambakam Krishnan,
1997. The EM Algorithm and Extensions. Wiley
Series in probability and statistics. New York, NY.
Mochihashi, Daichi, T. Yamada and N. Ueda.
“Bayesian Unsupervised Word Segmentation with
Nested Pitman-Yor Language Modeling”. In Proceedings of Joint Conference of the 47th Annual
Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AsianFederation of Natural Language Processing (ACLIJCNLP 2009), Singapore, August, 2009. pp. 100–
108.
46

Mochihashi, Daichi and Eiichiro Sumita. “The Infinite
Markov Model”. In Proceedings of the 20th Neural
Information Processing Systems (NIPS 2007), Vancouver, 2007. pp. 1017–1024.
Moore, Robert C.. 2003. Learning Translations ofNamed-Entity Phrases from Parallel Corpora. In
Proceedings of the 11th Conference of the European
Chapter of the Association for Computational Linguistics. Budapest, Hungary. pp. 259–266.
Moore, Robert C.. 2004. On Log-Likelihood-Ratios
and the Significance of Rare Events. In Proceedingsof the 2004 Conference on Empirical Methods in
Natural Language Processing (EMNLP). Barcelona,
Spain, pp. 333–340.

Teh, Yee Whye. “A hierarchical Bayesian language
model based on Pitman-Yor processes”. In Proceedings of Joint Conference of the 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, 2006. pp. 985–992.
Utiyama, Masao and Hitoshi Isahara. 2003. Reliable
Measures for Aligning Japanese-English News Articles and Sentences, In Proceedings of the 41st Annual Meeting of the Association for Computational
Linguistics. Sapporo, Japan, pp. 72–79.
Vogel, Stephan, Hermann Ney, Christoph Tillmann
1996. HMM-Based Word Alignment in Statistical Translation. In Proceedings of the 16th International Conference on Computational Linguistics.
Copenhagen, Denmark, pp. 836–841.

Och, Franz and Herman Ney. 2003. A SystematicComparison of Various Statistical Alignment Models. Computational Linguistics. 29(1), pp. 19–51.
Och, Franz. 2003 “Minimum Error Rate Training in
Statistical Machine Translation. Proceedings of the
41st Annual Meeting of the Association for Computational Linguistics, Sapporo, Japan, 2003, pp. 160–
167.
Okita, Tsuyoshi, Alfredo Maldonado Guerra, Yvette
Graham, and Andy Way. Multi-Word ExpressionSensitive Word Alignment. In Proceedings of the
Fourth International Workshop On Cross Ling ual
Information Access (CLIA2010, collocated with
COLING2010), Beijing, China. pp. 26–34.
Okita, Tsuyoshi and Andy Way. Hierarchical PitmanYor Language Model in Machine Translation. In
Proceedings of the International Conference on
Asian Language Processing (IALP 2010), Harbin,
China, December, 2010.
Pitman, Jim. “Exchangeable and partially Exchangeable Random Partitions”. Probability Theory and
Related Fields, Vol. 102, pp. 145-158, 1995.
Resnik, Philip and I. Dan Melamed, 1997. SemiAutomatic Acquisition of Domain-Specific Translation Lexicons. Proceedings of the 5th Applied Natural Language Processing Conference. Washington,
DC., pp. 340–347.
Stolcke, Andreas. 2002 SRILM – An extensible language modeling toolkit. Proceedings of the International Conference on Spoken Language Processing,
Denver, CO, pp. 901–904.
Talbot, David. 2005. Constrained EM for parallel textalignment, Natural Language Engineering, 11(3):pp. 263–277.

47