Joint Space Neural Probabilistic Language Model for Statistical Machine
      Translation
Authors: Tsuyoshi Okita
ICLR 2013

Abstract: A neural probabilistic language model (NPLM) provides an idea to achieve the better perplexity than n-gram language model and their smoothed language models. This paper investigates application area in bilingual NLP, specifically Statistical Machine Translation (SMT). We focus on the perspectives that NPLM has potential to open the possibility to complement potentially `huge' monolingual resources into the `resource-constraint' bilingual resources. We introduce an ngram-HMM language model as NPLM using the non-parametric Bayesian construction. In order to facilitate the application to various tasks, we propose the joint space model of ngram-HMM language model. We show an experiment of system combination in the area of SMT. One discovery was that our treatment of noise improved the results 0.20 BLEU points if NPLM is trained in relatively small corpus, in our case 500,000 sentence pairs, which is often the case due to the long training time of NPLM.

arXiv:1301.3614v3  [cs.CL]  21 Apr 2017
Joint Space Neural Probabilistic Language Model for
Statistical Machine T ranslation
Tsuyoshi Okita
Dublin City University
Glasnevin, Dublin 9
Ireland
tokita@computing.dcu.ie
Abstract
A neural probabilistic language model (NPLM) provides an id ea to achieve the
better perplexity than n-gram language model and their smoo thed language mod-
els. This paper investigates application area in bilingual NLP , speciﬁcally Sta-
tistical Machine Translation (SMT). W e focus on the perspec tives that NPLM
has potential to open the possibility to complement potenti ally ‘huge’ monolin-
gual resources into the ‘resource-constraint’ bilingual r esources. W e introduce an
ngram-HMM language model as NPLM using the non-parametric B ayesian con-
struction. In order to facilitate the application to variou s tasks, we propose the
joint space model of ngram-HMM language model. W e show an exp eriment of
system combination in the area of SMT . One discovery was that our treatment
of noise improved the results 0.20 BLEU points if NPLM is trai ned in relatively
small corpus, in our case 500,000 sentence pairs, which is of ten the case due to
the long training time of NPLM.
1 Introduction
A neural probabilistic language model (NPLM) [3, 4] and the d istributed representations [25] pro-
vide an idea to achieve the better perplexity than n-gram lan guage model [47] and their smoothed
language models [26, 9, 48]. Recently, the latter one, i.e. s moothed language model, has had a lot
of developments in the line of nonparametric Bayesian metho ds such as hierarchical Pitman-Y or
language model (HPYLM) [48] and Sequence Memoizer (SM) [51, 20], including an application to
SMT [36, 37, 38]. A NPLM considers the representation of data in order to make the probability
distribution of word sequences more compact where we focus o n the similar semantical and syntac-
tical roles of words. For example, when we have two sentences
“The cat is walking in the bedroom”
and “ A dog was running in a room” , these sentences can be more compactly stored than the n-gra m
language model if we focus on the similarity between (the, a) , (bedroom, room), (is, was), and (run-
ning, walking). Thus, a NPLM provides the semantical and syn tactical roles of words as a language
model. A NPLM of [3] implemented this using the multi-layer n eural network and yielded 20% to
35% better perplexity than the language model with the modiﬁ ed Kneser-Ney methods [9].
There are several successful applications of NPLM [41, 11, 4 2, 10, 12, 14, 43]. First, one category
of applications include POS tagging, NER tagging, and parsi ng [12, 7]. This category uses the
features provided by a NPLM in the limited window size. It is o ften the case that there is no such
long range effects that the decision cannot be made beyond th e limited windows which requires to
look carefully the elements in a long distance. Second, the o ther category of applications include
Semantic Role Labeling (SRL) task [12, 14]. This category us es the features within a sentence. A
typical element is the predicate in a SRL task which requires the information which sometimes in
a long distance but within a sentence. Both of these approach es do not require to obtain the best
tag sequence, but these tags are independent. Third, the ﬁna l category includes MER T process [42]
1
and possibly many others where most of them remain undevelop ed. The objective of this learning
in this category is not to search the best tag for a word but the best sequence for a sentence. Hence,
we need to apply the sequential learning approach. Although most of the applications described in
[11, 10, 12, 14] are monolingual tasks, the application of th is approach to a bilingual task introduces
really astonishing aspects, which we can call “creative wor ds” [50], automatically into the traditional
resource constrained SMT components. For example, the trai ning corpus of word aligner is often
strictly restricted to the given parallel corpus. However, a NPLM allows this training with huge
monolingual corpus. Although most of this line has not been e ven tested mostly due to the problem
of computational complexity of training NPLM, [43] applied this to MER T process which reranks
the n-best lists using NPLM. This paper aims at different tas k, a task of system combination [1,
29, 49, 15, 13, 35]. This category of tasks employs the sequen tial method such as Maximum A
Posteriori (MAP) inference (V iterbi decoding) [27, 44, 33] on Conditional Random Fields (CRFs) /
Markov Random Fields (MRFs).
Although this paper discusses an ngram-HMM language model w hich we introduce as one model of
NPLM where we borrow many of the mechanism from inﬁnite HMM [1 9] and hierarchical Pitman-
Y or LM [48], one main contribution would be to show one new app lication area of NPLM in SMT .
Although several applications of NPLM have been presented, there have been no application to the
task of system combination as far as we know .
The remainder of this paper is organized as follows. Section 2 describes ngram-HMM language
model while Section 3 introduces a joint space model of ngram -HMM language model. In Section
4, our intrinsic experimental results are presented, while in Section 5 our extrinsic experimental
results are presented. W e conclude in Section 5.
2 Ngram-HMM Language Model
Generative model Figure 1 depicted an example of ngram-HMM language model, i. e. 4-gram-
HMM language model in this case, in blue (in the center). W e co nsider a Hidden Markov
Model (HMM) [40, 21, 2] of size K which emits n-gram word sequence wi, . . . , w i−K+1 where
hi, . . . , h i−K+1 denote corresponding hidden states. The arcs from wi−3 to wi, · · ·, wi−1 to wi
show the back-off relations appeared in language model smoo thing, such as Kneser-Ney smoothing
[26], Good-Turing smoothing [24], and hierarchical Pitman -Y or LM smoothing [48].
0
G
G
G
u
0
d
d
0
0
u
k
H
α
ϕ θu
θ
π
βγ
k=1,...,T u=1,...,K
h h h h
w w w w
i−3 i−2 i−1 i
i−3 i−2 i−1 i
Figure 1: Figure shows a graphical representation of the 4-g ram HMM language model.
In the left side in Figure 1, we place one Dirichlet Process pr ior DP (α, H), with concentration pa-
rameter α and base measure H, for the transition probabilities going out from each hidde n state.
This construction is borrowed from the inﬁnite HMM [2, 19]. T he observation likelihood for the
hidden word ht are parameterized as in wt|ht ∼ F (φht ) since the hidden variables of HMM is lim-
ited in its representation power where φht denotes output parameters. This is since the observations
can be regarded as being generated from a dynamic mixture mod el [19] as in (1), the Dirichlet priors
2
on the rows have a shared parameter.
p(wi|hi−1 = k) =
K∑
hi=1
p(hi|hi−1 = k)p(wi|hi)
=
K∑
hi=1
πk,h i p(wi|φhi ) (1)
In the right side in Figure 1, we place Pitman-Y or prior PY, wh ich has advantage in its power-law
behavior as our target is NLP , as in (2):
wi|w1:i−1 ∼ PY(di, θ i, G i) (2)
where α is a concentration parameter, θ is a strength parameter, and Gi is a base measure. This
construction is borrowed from hierarchical Pitman-Y or lan guage model [48].
Inference W e compute the expected value of the posterior distribution of the hidden variables with
a beam search [19]. This blocked Gibbs sampler alternate sam ples the parameters (transition matrix,
output parameters), the state sequence, hyper-parameters , and the parameters related to language
model smoothing. As is mentioned in [19], this sampler has ch aracteristic in that it adaptively
truncates the state space and run dynamic programming as in ( 3):
p(ht|w1:t, u 1:t) = p(wt|ht)
∑
ht−1:ut<π (ht−1,ht)
p(ht−1|w1:t−1, u 1:t−1) (3)
where ut is only valid if this is smaller than the transition probabil ities of the hidden word sequence
h1, . . . , h K . Note that we use an auxiliary variable ui which samples for each word in the sequence
from the distribution ui ∼ Uniform(0, π (hi−1,h i)). The implementation of the beam sampler con-
sists of preprocessing the transition matrix π and sorting its elements in descending order.
Initialization First, we obtain the parameters for hierarchical Pitman-Y o r process-based language
model [48, 23], which can be obtained using a block Gibbs samp ling [32].
Second, in order to obtain a better initialization value h for the above inference, we perform the
following EM algorithm instead of giving the distribution o f h randomly. This EM algorithm in-
corporates the above mentioned truncation [19]. In the E-st ep, we compute the expected value of
the posterior distribution of the hidden variables. For eve ry position hi, we send a forward message
α (hi−n+1:i−1) in a single path from the start to the end of the chain (which is the standard forward
recursion in HMM; Hence we use α ). Here we normalize the sum of α considering the truncated
variables ui−n+1:i−1.
α (hi−n+2:i) =
∑ α (hi−n+1:i−1)∑ α (ui−n+1:i−1)P (wi|hi)
∑
α (ui−n+1:i−1)P (hi|hi−n+1:i−1) (4)
Then, for every position hj , we send a message β(hi−n+2:i, h j ) in multiple paths from the start to
the end of the chain as in (5),
β(hi−n+2:i, h j ) =
∑ α (hi−n+1:i−1)∑ α (ui−n+1:i−1)P (wi|hi)
∑
β(hi−n+1:i−1, h j )P (hi|hi−n+1:i−1) (5)
This step aims at obtaining the expected value of the posteri or distribution (Similar construction to
use expectation can be seen in factored HMM [22]). In the M-st ep, using this expected value of
the posterior distribution obtained in the E-step to evalua te the expectation of the logarithm of the
complete-data likelihood.
3 Joint Space Model
In this paper, we mechanically introduce a joint space model . Other than the ngram-HMM language
model obtained in the previous section, we will often encoun ter the situation where we have another
hidden variables h1 which is irrelevant to h0 which is depicted in Figure 2. Suppose that we have
3
the ngram-HMM language model yielded the hidden variables s uggesting semantic and syntactical
role of words. Adding to this, we may have another hidden vari ables suggesting, say, a genre ID.
This genre ID can be considered as the second context which is often not closely related to the ﬁrst
context. This also has an advantage in this mechanical const ruction that the resulted language model
often has the perplexity smaller than the original ngram-HM M language model. Note that we do
not intend to learn this model jointly using the universal cr iteria, but we just concatenate the labels
by different tasks on the same sequence. By this formulation , we intend to facilitate the use of this
language model.
h h h h
0 0 0 0
1111
i−3 i−2 i−1 i
Figure 2: Figure shows the joint space 4-gram HMM language mo del.
It is noted that those two contexts may not be derived in a sing le learning algorithm. For example,
language model with the sentence context may be derived in th e same way with that with the word
context. In the above example, a hidden semantics over sente nce is not a sequential object. Hence,
this can be only considering all the sentence are independen t. Then, we can obtain this using, say,
LDA.
4 Intrinsic Evaluation
W e compared the perplexity of ngram-HMM LM (1 feature), ngra m-HMM LM (2 features, the same
as in this paper and genre ID is 4 class), modiﬁed Kneser-Ney s moothing (irstlm) [18], and hierar-
chical Pitman Y or LM [48]. W e used news2011 English testset. W e trained LM using Europarl.
ngram-HMM (1 feat) ngram-HMM (2 feat) modiﬁed Kneser-Ney hi erarchical PY
Europarl 1500k 114.014 113.450 118.890 118.884
T able 1: T able shows the perplexity of each language model.
5 Extrinsic Evaluation: T ask of System Combination
W e applied ngram-HMM language model to the task of system com bination. For given multiple
Machine Translation (MT) outputs, this task essentially co mbines the best fragments among given
MT outputs to recreate a new MT output. The standard procedur e consists of three steps: Minimum
Bayes Risk decoding, monolingual word alignment, and monot onic consensus decoding. Although
these procedures themselves will need explanations in orde r to understand the following, we keep
the main text in minimum, moving some explanations (but not s ufﬁcient) in appendices. Note that
although this experiment was done using the ngram-HMM langu age model, any NPLM may be
sufﬁcient for this purpose. In this sense, we use the term NPL M instead of ngram-HMM language
model.
Features in Joint Space The ﬁrst feature of NPLM is the semantically and syntactical ly similar
words of roles, which can be derived from the original NPLM. W e introduce the second feature in
this paragraph, which is a genre ID.
The motivation to use this feature comes from the study of dom ain adaptation for SMT where it be-
comes popular to consider the effect of genre in testset. Thi s paper uses Latent Dirichlet Allocation
4
(LDA) [5, 46, 6, 45, 33] to obtain the genre ID via (unsupervis ed) document classiﬁcation since our
interest here is on the genre of sentences in testset. And the n, we place these labels on a joint space.
LDA represents topics as multinomial distributions over th e W unique word-types in the corpus and
represents documents as a mixture of topics. Let C be the number of unique labels in the corpus.
Each label c is represented by a W -dimensional multinomial distribution φc over the vocabulary.
For document d, we observe both the words in the document w(d) as well as the document labels
c(d). Given the distribution over topics θd, the generation of words in the document is captured by
the following generative model.The parameters α and β relate to the corpus level, the variables θd
belong to the document level, and ﬁnally the variables zdn and wdn correspond to the word level,
which are sampled once for each word in each document.
Using topic modeling in the second step, we propose the overa ll algorithm to obtain genre IDs for
testset as in (5).
1. Fix the number of clusters C, we explore values from small to big where the optimal value
will be searched on tuning set.
2. Do unsupervised document classiﬁcation (or LDA) on the so urce side of the tuning and test
sets.
(a) For each label c ∈ { 1, . . . C }, sample a distribution over word-types φc ∼
Dirichlet(·|β)
(b) For each document d ∈ { 1, . . . , D }
i. Sample a distribution over its observed labels θd ∼ Dirichlet(·|α )
ii. For each word i ∈ { 1, . . . , N W
d }
A. Sample a label z(d)
i ∼ Multinomial(θd)
B. Sample a word w(d)
i ∼ Multinomial(φc) from the label c = z(d)
i
3. Separate each class of tuning and test sets (keep the origi nal index and new index in the
allocated separated dataset).
4. (Run system combination on each class.)
5. (Reconstruct the system combined results of each class pr eserving the original index.)
Modiﬁed Process in System Combination Given a joint space of NPLM, we need to specify
in which process of the task of system combination among thre e processes use this NPLM. W e
only discuss here the standard system combination using con fusion-network. This strategy takes the
following three steps (V ery brief explanation of these thre e is available in Appendix):
• Minimum Bayes Risk decoding [28] (with Minimum Error Rate Tr aining (MER T) process
[34])
ˆEMBR
best = argminE′∈E R(E′) = argminE′∈E
∑
E′∈EE
L(E, E ′)P (E|F )
= argminE′∈E
∑
E′∈EE
(1 − BLEUE(E′))P (E|F )
• Monolingual word alignment
• (Monotone) consensus decoding (with MER T process)
Ebest = arg max
e
I∏
i=1
φ(i|¯ei)pLM (e)
Similar to the task of n-best reranking in MER T process [43], we consider the reranking of nbest
lists in the third step of above, i.e. (monotone) consensus d ecoding (with MER T process). W e do
not discuss the other two processes in this paper.
On one hand, we intend to use the ﬁrst feature of NPLM, i.e. the semantically and syntactically
similar role of words, for paraphrases. The n-best rerankin g in MER T process [43] alternate the
5
probability suggested by word sense disambiguation task us ing the feature of NPLM, while we
intend to add a sentence which replaces the words using NPLM. On the other hand, we intend to
use the second feature of NPLM, i.e. the genre ID, to split a si ngle system combination system into
multiple system combination systems based on the genre ID cl usters. In this perspective, the role of
these two feature can be seen as independent. W e conducted fo ur kinds of settings below .
(A) —First Feature: N-Best Reranking in Monotonic Consensu s Decoding without Noise –
NPLM plain In the ﬁrst setting for the experiments, we used the ﬁrst feat ure without considering
noise. The original aim of NPLM is to capture the semanticall y and syntactically similar words
in a way that a latent word depends on the context. W e will be ab le to get variety of words if we
condition on the ﬁxed context, which would form paraphrases in theory.
W e introduce our algorithm via a word sense disambiguation ( WSD) task which selects the right
disambiguated sense for the word in question. This task is ne cessary due to the fact that a text is
natively ambiguous accommodating with several different m eanings. The task of WSD [14] can be
written as in (6):
P (synseti|featuresi, θ ) = 1
Z(features)
∏
m
g(synseti, k )f(feature
k
i ) (6)
where k ranges over all possible features, f(featurek
i ) is an indicator function whose value is 1 if
the feature exists, and 0 otherwise, g(synseti, k ) is a parameter for a given synset and feature, θ is a
collection of all these parameters in g(synseti, k ), and Z is a normalization constant. Note that we
use the term “synset” as an analogy of the W ordNet [30]: this i s equivalent to “sense” or “meaning”.
Note also that NPLM will be included as one of the features in t his equation. If features include
sufﬁcient statistics, a task of WSD will succeed. Otherwise , it will fail. W e do reranking of the
outcome of this WSD task.
On the one hand, the paraphrases obtained in this way have att ractive aspects that can be called
“a creative word” [50]. This is since the traditional resour ce that can be used when building a
translation model by SMT are constrained on parallel corpus . However, NPLM can be trained on
huge monolingual corpus. On the other hand, unfortunately i n practice, the notorious training time
of NPLM only allows us to use fairly small monolingual corpus although many papers made an
effort to reduce it [31]. Due to this, we cannot ignore the fac t that NPLM trained not on a huge
corpus may be affected by noise. Conversely, we have no guara ntee that such noise will be reduced
if we train NPLM on a huge corpus. It is quite likely that NPLM h as a lot of noise for small corpora.
Hence, this paper also needs to provide the way to overcome di fﬁculties of noisy data. In order to
avoid this difﬁculty, we limit the paraphrase only when it in cludes itself in high probability.
(B)— First Feature: N-Best Reranking in Monotonic Consensu s Decoding with Noise – NPLM
dep In the second setting for our experiment, we used the ﬁrst fea ture considering noise. Although
we modiﬁed a suggested paraphrase without any intervention in the above algorithm, it is also pos-
sible to examine whether such suggestion should be adopted o r not. If we add paraphrases and the
resulted sentence has a higher score in terms of the modiﬁed d ependency score [39] (See Figure 3),
this means that the addition of paraphrases is a good choice. If the resulted score decreases, we do
not need to add them. One difﬁculty in this approach is that we do not have a reference which allows
us to score it in the usual manner. For this reason, we adopt th e naive way to deploy the above and
we deploy this with pseudo references . (This formulation is equivalent that we decode these input s
by MBR decoding.) First, if we add paraphrases and the result ed sentence does not have a very bad
score, we add these paraphrases since these paraphrase are n ot very bad ( naive way). Second, we
do scoring between the sentence in question with all the other candidates (pseudo references ) and
calculate an average of them. Thus, our second algorithm is t o select a paraphrase which may not
achieve a very bad score in terms of the modiﬁed dependency sc ore using NPLM.
(C) — Second Feature: Genre ID — DA (Domain Adaptation) In the third setting of our ex-
periment, we used only the second feature. As is mentioned in the explanation about this feature,
we intend to splits a single module of system combination int o multiple modules of system combi-
6
S
NP NP VP
Yesterday John
resigned
V
SUBJ          PRED    john
                    NUM     sg
                    PERS     3
PRED         resign
TENSE       past
ADJ            ([PRED yesterday])
S
NP VP
V NPJohn
resigned yesterday
SUBJ PRED   john
NUM    sg
PERS    3
PRED   resign
TENSE   past
ADJ          ([PRED yesterday])
Different structure Same representation
c−structure f−structure
in c−structure in f−structure
Figure 3: By the modiﬁed dependency score [39], the score of t hese two sentences, “John resigned
yesterday” and “Y esterday John resigned”, are the same. Fig ure shows c-structure and f-structure of
two sentences using Lexical Functional Grammar (LFG) [8].
nation according to the genre ID. Hence, we will use the modul e of system combination tuned for
the speciﬁc genre ID, 1 .
(D) — First and Second Feature — COMBINED In the fourth setting we used both features.
In this setting, (1) we used modules of system combination wh ich are tuned for the speciﬁc genre
ID, and (2) we prepared NPLM whose context can be switched bas ed on the speciﬁc genre of the
sentence in test set. The latter was straightforward since t hese two features are stored in joint space
in our case.
Experimental Results ML4HMT -2012 provides four translation outputs ( s1 to s4) which are
MT outputs by two RBMT systems, A P E RT IU M and L U CY , PB-SMT (M O S E S ) and HPB-SMT
(M O S E S ), respectively. The tuning data consists of 20,000 sentenc e pairs, while the test data con-
sists of 3,003 sentence pairs.
Our experimental setting is as follows. W e use our system com bination module [16, 17, 35], which
has its own language modeling tool, MER T process, and MBR dec oding. W e use the BLEU metric
as loss function in MBR decoding. W e use TER P2 as alignment metrics in monolingual word
alignment. W e trained NPLM using 500,000 sentence pairs fro m English side of EN-ES corpus of
EU RO PA RL 3 .
The results show that the ﬁrst setting of NPLM-based paraphr ased augmentation, that is NPLM
plain, achieved 25.61 BLEU points, which lost 0.39 BLEU poin ts absolute over the standard sys-
tem combination. The second setting, NPLM dep, achieved sli ghtly better results of 25.81 BLEU
points, which lost 0.19 BLEU points absolute over the standa rd system combination. Note that
the baseline achieved 26.00 BLEU points, the best single sys tem in terms of BLEU was s4 which
achieved 25.31 BLEU points, and the best single system in ter ms of METEOR was s2 which
achieved 0.5853. The third setting achieved 26.33 BLEU poin ts, which was the best among our
four settings. The fourth setting achieved 25.95, which is a gain lost 0.05 BLEU points over the
standard system combination.
Other than our four settings where these settings differ whi ch features to use, we run several differ-
ent settings of system combination in order to understand th e performance of four settings. Standard
system combination using BLEU loss function (line 5 in T able 2), standard system combination
using TER loss function (line 6), system combination whose b ackbone is unanamously taken from
the RBMT outputs (MT input s2 in this case; line 11), and syste m combination whose backbone is
selected by the modiﬁed dependency score (which has three va riations in the ﬁgure; modDep preci-
1 E.g., we translate newswire with system combination module tuned with newswire tuning set, while we
translate medical text with system combination module tune d with medical text tuning set.
2 http://www .cs.umd.edu/∼ snover/terp
3 http://www .statmt.org/europarl
7
sion, recall and Fscore; line 12, 13 and 14). One interesting characteristics is that the s2 backbone
(line 11) achieved the best score among all of these variatio ns. Then, the score of the modiﬁed
dependency measure-selected backbone follows. From these runs, we cannot say that the runs re-
lated to NPLM, i.e. (A), (B) and (D), were not particularly su ccessful. The possible reason for this
was that our interface with NPLM was only limited to paraphra ses, which was not very successfuly
chosen by reranking.
NIST BLEU METEOR WER PER
MT input s1 6.4996 0.2248 0.5458641 64.2452 49.9806
MT input s2 6.9281 0.2500 0.5853446
62.9194 48.0065
MT input s3 7.4022 0.2446 0.5544660 58.0752 44.0221
MT input s4 7.2100 0.2531
0.5596933 59.3930 44.5230
standard system combination (BLEU) 7.6846 0.2600 0.564394 4 56.2368 41.5399
standard system combination (TER) 7.6231 0.2638 0.5652795 56.3967 41.6092
(A) NPLM plain 7.6041 0.2561 0.5593901 56.4620 41.8076
(B) NPLM dep 7.6213 0.2581 0.5601121 56.1334 41.7820
(C) DA 7.7146 0.2633 0.5647685 55.8612 41.7264
(D) COMBINED 7.6464 0.2595 0.5610121 56.0101 41.7702
s2 backbone 7.6371 0.2648 0.5606801 56.0077 42.0075
modDep precision 7.6670 0.2636 0.5659757 56.4393 41.4986
modDep recall 7.6695 0.2642 0.5664320 56.5059 41.5013
modDep Fscore 7.6695 0.2642 0.5664320 56.5059 41.5013
T able 2: This table shows single best performance, the perfo rmance of the standard system combina-
tion (BLEU and TER loss functions), the performance of four s ettings in this paper ((A), . . . ,(D)), the
performance of s2 backboned system combination, and the per formance of the selection of sentences
by modiﬁed dependency score (precision, recall, and F-scor e each).
Conclusion and Perspectives
This paper proposes a non-parametric Bayesian way to interp ret NPLM, which we call ngram-
HMM language model. Then, we add a small extension to this by c oncatenating other context
in the same model, which we call a joint space ngram-HMM langu age model. The main issues
investigated in this paper was an application of NPLM in bili ngual NLP , speciﬁcally Statistical
Machine Translation (SMT). W e focused on the perspectives t hat NPLM has potential to open the
possibility to complement potentially ‘huge’ monolingual resources into the ‘resource-constraint’
bilingual resources. W e compared our proposed algorithms a nd others. One discovery was that
when we use a fairly small NPLM, noise reduction may be one way to improve the quality. In our
case, the noise reduced version obtained 0.2 BLEU points bet ter.
Further work would be to apply this NPLM in various other task s in SMT: word alignment, hierar-
chical phrase-based decoding, and semantic incorporated M T systems in order to discover the merit
of ‘depth’ of architecture in Machine Learning.
References
[1] B ANGALOR E , S., B OR DEL , G., AND RIC C AR DI , G. Computing consensus translation from multiple machine translation systems. In Proceedings of the IEEE
Automatic Speech Recognition and Understanding W orkshop ( ASRU) (2001), 350–354.
[2] B EAL , M. J. V ariational algorithms for approximate bayesian inf erence. PhD Thesis at Gatsby Computational Neuroscience Unit, Univ ersity College London
(2003).
[3] B ENGIO , Y ., D UC HAR ME , R., AND VINC ENT , P . A neural probabilistic language model. In Proceedings of Neural Information Systems (2000).
[4] B ENGIO , Y ., S C HW ENK , H., S EN ´EC AL , J.-S., M OR IN , F., AND GAUV AIN , J.-L. Neural probabilistic language models. Innovations in Machine Learning:
Theory and Applications Edited by D. Holmes and L. C. Jain (2005).
[5] B LEI , D., N G, A. Y ., AND JOR DAN , M. I. Latent dirichlet allocation. Journal of Machine Learning Research 3 (2003), 9931022.
[6] B LEI , D. M. Introduction to probabilistic topic models. Communications of the ACM (2011).
[7] B OR DES , A., G LOROT , X., W ESTON , J., AND BENGIO , Y . T owards open-text semantic parsing via multi-task lear ning of structured embeddings. CoRR
abs/1107.3663 (2011).
[8] B R ESNAN , J. Lexical functional syntax. Blackwell (2001).
[9] C HEN , S., AND GOODMAN , J. An empirical study of smoothing techniques for language modeling. T echnical report TR-10-98 Harvard University (1998).
8
[10] C OLLOB ERT , R. Deep learning for efﬁcient discriminative parsing. In Proceedings of the 14th International Conference on Arti ﬁcial Intelligence and Statistics
(AISTATS) (2011).
[11] C OLLOB ERT , R., AND WESTON , J. A uniﬁed architecture for natural language processing: Deep neural networks with multitask learning. In International
Conference on Machine Learning (ICML 2008) (2008).
[12] C OLLOB ERT , R., W ESTON , J., B OTTOU , L., K AR LEN , M., K A VUKC UOGLU , K., AND KUKSA , P . Natural language processing (almost) from scratch. Journal
of Machine Learning Research 12 (2011), 2493–2537.
[13] D ENERO , J., C HIANG , D., AND KNIGHT , K. Fast consensus decoding over translation forests. In proceedings of the Joint Conference of the 47th Annual
Meeting of the ACL and the 4th International Joint Conferenc e on Natural Language Processing of the AFNLP (2009), 567–575.
[14] D ESC HAC HT , K., B ELDER , J. D., AND MOENS , M.-F. The latent words language model. Computer Speech and Language 26 (2012), 384–409.
[15] D U, J., H E, Y ., P ENKALE , S., AND WAY, A. MaTrEx: the DCU MT System for WMT 2009. In Proceedings of the Third EACL W orkshop on Statistical
Machine Translation (2009), 95–99.
[16] D U, J., AND WAY, A. An incremental three-pass system combination framewor k by combining multiple hypothesis alignment methods. International Journal
of Asian Language Processing 20 , 1 (2010), 1–15.
[17] D U, J., AND WAY, A. Using terp to augment the system combination for smt. In Proceedings of the Ninth Conference of the Association fo r Machine Translation
(AMTA2010) (2010).
[18] F EDER IC O , M., B ERTOLDI , N., AND CETTOLO , M. Irstlm: an open source toolkit for handling large scale l anguage models. Proceedings of Interspeech
(2008).
[19] G AEL , J. V ., V LAC HOS , A., AND GHAHR AMANI , Z. The inﬁnite hmm for unsupervised pos tagging. The 2009 Conference on Empirical Methods on Natural
Language Processing (EMNLP 2009) (2009).
[20] G ASTHAUS , J., W OOD , F., AND TEH , Y . W . Lossless compression based on the sequence memoizer. DCC 2010 (2010).
[21] G HAHR AMANI , Z. An introduction to hidden markov models and bayesian net works. International Journal of P attern Recognition and Artiﬁcia l Intelligence
15, 1 (2001), 9–42.
[22] G HAHR AMANI , Z., J OR DAN , M. I., AND SMYTH , P . Factorial hidden markov models. Machine Learning (1997).
[23] G OLDW ATER , S., G R IFFITHS , T. L., AND JOHNSON , M. Contextual dependencies in unsupervised word segmenta tion. In Proceedings of Conference on
Computational Linguistics / Association for Computationa l Linguistics (COLING-ACL06) (2006), 673–680.
[24] G OOD , I. J. The population frequencies of species and the estimat ion of population paramters. Biometrika 40 , (3-4) (1953), 237–264.
[25] H INTON , G. E., M CCLELLAND , J. L., AND RUMELHART , D. Distributed representations. P arallel Distributed Processing: Explorations in the Micr ostructure
of Cognition(Edited by D.E. Rumelhart and J.L. McClelland) MIT Press 1 (1986).
[26] K NESER , R., AND NEY, H. Improved backing-off for n-gram language modeling. In Proceedings of the IEEE International Conference on Acou stics, Speech
and Signal Processing (1995), 181–184.
[27] K OLLER , D., AND FR IEDMAN , N. Probabilistic graphical models: Principles and techni ques. MIT Press (2009).
[28] K UMAR , S., AND BYR NE , W . Minimum Bayes-Risk word alignment of bilingual texts. In Proceedings of the Empirical Methods in Natural Language
Processing (EMNLP 2002) (2002), 140–147.
[29] M ATUSOV, E., U EFFING , N., AND NEY, H. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment.
In Proceedings of the 11st Conference of the European Chapte r of the Association for Computational Linguistics (EACL) (2006), 33–40.
[30] M ILLER , G. A. W ordnet: A lexical database for english. Communications of the ACM 38 , 11 (1995), 39–41.
[31] M NIH , A., AND TEH , Y . W . A fast and simple algorithm for training neural probab ilistic language models. In Proceedings of the International Conference on
Machine Learning (2012).
[32] M OC HIHASHI , D., Y AMADA , T., AND UEDA , N. Bayesian unsupervised word segmentation with nested pi tman-yor language modeling. In Proceedings of
Joint Conference of the 47th Annual Meeting of the Associati on for Computational Linguistics and the 4th International Joint Conference on Natural Language
Processing of the Asian F ederation of Natural Language Proc essing (ACL-IJCNLP 2009) (2009), 100–108.
[33] M UR PHY, K. P . Machine learning: A probabilistic perspective. The MIT Press (2012).
[34] O C H, F., AND NEY, H. A systematic comparison of various statistical alignme nt models. Computational Linguistics 29 , 1 (2003), 19–51.
[35] O KITA , T., AND V AN GENAB ITH , J. Minimum bayes risk decoding with enlarged hypothesis sp ace in system combination. In Proceedings of the 13th
International Conference on Intelligent T ext Processing a nd Computational Linguistics (CICLING 2012). LNCS 7182 P ar t II. A. Gelbukh (Ed.) (2012), 40–51.
[36] O KITA , T., AND WAY, A. Hierarchical pitman-yor language model in machine tran slation. In Proceedings of the International Conference on Asian Lan guage
Processing (IALP 2010) (2010).
[37] O KITA , T., AND WAY, A. Pitman-Yor process-based language model for Machine Tr anslation. International Journal on Asian Language Processing 21 , 2
(2010), 57–70.
[38] O KITA , T., AND WAY, A. Given bilingual terminology in statistical machine tra nslation: Mwe-sensitve word alignment and hierarchical pi tman-yor process-
based translation model smoothing. In Proceedings of the 24th International Florida Artiﬁcial Intelligence Research Society Conference (FLAIRS-24) (2011),
269–274.
[39] O W C ZAR ZAK , K., V AN GENAB ITH , J., AND WAY, A. Evaluating machine translation with LFG dependencies. Machine Translation 21 , 2 (2007), 95–119.
[40] R AB INER , L. R. A tutorial on hidden markov models and selected applic ations in speech recognition. Proceedings of the IEEE 77 , 2 (1989), 257–286.
[41] S C HW ENK , H. Continuous space language models. Computer Speech and Language 21 (2007), 492–518.
[42] S C HW ENK , H. Continuous space language models for statistical machi ne translation. The Prague Bulletin of Mathematical Linguistics 83 (2010), 137–146.
[43] S C HW ENK , H., R OUSSEAU , A., AND ATTIK , M. Large, pruned or continuous space language models on a gp u for statistical machine translation. In Proceeding
of the NAACL workshop on the Future of Language Modeling (2012).
[44] S ONTAG , D. Approximate inference in graphical models using LP rela xations. Massachusetts Institute of T echnology (Ph.D. thesis) (2010).
[45] S ONTAG , D., AND ROY, D. M. The complexity of inference in latent dirichlet alloc ation. In Advances in Neural Information Processing Systems 24 (NI PS)
(2011).
[46] S TEYVER S , M., AND GR IFFITHS , T. Probabilistic topic models. Handbook of Latent Semantic Analysis. Psychology Press (2007).
[47] S TOLC KE , A. SRILM – An extensible language modeling toolkit. In Proceedings of the International Conference on Spoken La nguage Processing (2002),
901–904.
[48] T EH , Y . W . A hierarchical bayesian language model based on pitma n-yor processes. In Proceedings of the 44th Annual Meeting of the Association for
Computational Linguistics (ACL-06), Prague, Czech Republ ic (2006), 985–992.
[49] T ROMB LE , R., K UMAR , S., O C H, F., AND MAC HER EY, W . Lattice minimum bayes-risk decoding for statistical ma chine translation. Proceedings of the 2008
Conference on Empirical Methods in Natural Language Proces sing (2008), 620–629.
[50] V EALE , T. Exploding the creativity myth: The computational found ations of linguistic creativity. London: Bloomsbury Academic (2012).
[51] W OOD , F., A R C HAMB EAU , C., G ASTHAUS , J., J AMES , L., AND TEH , Y . W . A stochastic memoizer for sequence data. In Proceedings of the 26th International
Conference on Machine Learning (2009), 1129–1136.
9


Peer review:
[Review]:
To quote the authors, this paper introduces a n-gram-HMM language
model as neural probabilistic language model (NPLM) using the
non-parametric Bayesian construction. This article is really confused
and describes a messy mix of different approaches. At the end, it is
very hard to understand what the author wanted to do and what he has
done. This paper can be improved in many ways before it could be
published: authors must clarify their motivations; the interaction
between neural and HMM models could be describe more precisely;


In the reading order: 

In the introduction, authors mistake the MERT process with the
translation process. While MERT is used to tune the weights of a
log-linear combination of models in order to optimize BLEU for
instance, the NPLM are used as an additionnal model to re-rank nbest
lists. Moreover, the correct citation for MERT is the ACL 2003 paper
of F. Och.

In section 2, the authors introduce a HMM language model. A lot of
questions remain: What does the hidden states intend to capture ?
What is the motivation ? What is the joint distribution associated to
the graphical model of figure 1 ? How the word n-gram distributions
are related to the hidden states ?

In section 3, the authors enhance their HMM LM with an additional row
of hidden states (joint space HMM). At this point the overall goal of
the paper is for me totally unclear. 

For the following experimental sections, a lot of information on the
set put is missing. Experiments cannot be reproduced given based on
the content of that paper. For example, the intrinsic evaluation
introduces ngram-HMM with one or two features. The very confused
explanation of these features is provided further in the
article. Authors do not describe the data-set (there are a lot of
europarl version), nor the order of the LMs under consideration.

In section 5, the following sentence puzzled me: 'Note that although
this experiment was done using the ngram-HMM language model, any NPLM
may be sufficient for this purpose. In this sense, we use the term
NPLM instead of ngram-HMM language model.' Moreover, the first feature
is derived from a NPLM, but how this NPLM is learnt, on which dataset,
what are the parameters, the order of the model and how this feature
is derived. I could not find the answers in this article. The rest of
the paper is more and more unclear. At the end, authors shows a BLEU
improvement of 0.2 on a system combination task. While I don't
understand the models used, the gain is really small and I wonder if
it is significant. For comparison's sake, MBR decoding usually provide
a BLEU improvement of at least 0.2.