Joint Space Neural Probabilistic Language Model for
Statistical Machine Translation
Tsuyoshi Okita,
School of Computing
Dublin City University
Glasnevin, Dublin 9, Ireland
tokita@computing.dcu.ie

Abstract
A neural probabilistic language model (NPLM) provides an idea to achieve thebetter perplexity than n-gram language model and their smoothed language models. This paper investigates application area in bilingual NLP, specifically Statistical Machine Translation (SMT). We focus on the perspectives that NPLM
has potential to open the possibility to complement potentially huge monolingual resources into the resource-constraint bilingual resources. We introduce anngram-HMM language model as NPLM using the non-parametric Bayesian construction. In order to facilitate the application to various tasks, we propose thejoint space model of ngram-HMM language model. We show an experiment ofsystem combination in the area of SMT. One discovery was that our treatmentof noise improved the results 0.20 BLEU points if NPLM is trained in relatively
small corpus, in our case 500,000 sentence pairs, which is often the case due tothe long training time of NPLM.

1Introduction

A neural probabilistic language model (NPLM) [3, 4] and the distributed representations [25] provide an idea to achieve the better perplexity than n-gram language model [47] and their smoothedlanguage models [26, 9, 48]. Recently, the latter one, i.e. smoothed language model, has had a
lot of developments in the line of nonparametric Bayesian methods such as hierarchical Pitman-Yor
language model (HPYLM) [48] and Sequence Memoizer (SM) [51, 20], including an application to
SMT [36, 37, 38]. A NPLM considers the representation of data in order to make the probability
distribution of word sequences more compact where we focus on the similar semantical and syntactical roles of words. For example, when we have two sentences The cat is walking in the bedroom
and A dog was running in a room, these sentences can be more compactly stored than the n-gram
language model if we focus on the similarity between (the, a), (bedroom, room), (is, was), and(run- ning, walking). Thus, a NPLM provides the semantical and syntactical roles of words as alanguage model. A NPLM of [3] implemented this using the multi-layer neural network and yielded
2035There are several successful applications of NPLM [41, 11, 42, 10, 12, 14, 43]. First, one
category of applications include POS tagging, NER tagging, and parsing [12, 7]. This category uses
the features provided by a NPLM in the limited window size. It is often the case that there is no suchlong range effects that the decision cannot be made beyond the limited windows which requires tolook carefully the elements in a long distance. Second, the other category of applications include
Semantic Role Labeling (SRL) task [12, 14]. This category uses the features within a sentence. A
typical element is the predicate in a SRL task which requires the information which sometimes in along distance but within a sentence. Both of these approaches do not require to obtain the best tag
sequence, but these tags are independent. Third, the final category includes MERT process [42] andpossibly many others where most of them remain undeveloped. The objective of this learning

1

054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074

in this category is not to search the best tag for a word but the best sequence for a sentence. Hence,
we need to apply the sequential learning approach. Although most of the applications described in[11, 10, 12, 14] are monolingual tasks, the application of this approach to a bilingual task introduces
really astonishing aspects, which we can call “creative words” [50], automatically into the traditional
resource constrained SMT components. For example, the training corpus of word aligner is often
strictly restricted to the given parallel corpus. However, a NPLM allows this training with hugemonolingual corpus. Although most of this line has not been even tested mostly due to the problem
of computational complexity of training NPLM, [43] applied this to MERT process which reranks
the n-best lists using NPLM. This paper aims at different task, a task of system combination [1,29, 49, 15, 13, 35]. This category of tasks employs the sequential method such as Maximum A
Posteriori (MAP) inference (Viterbi decoding) [27, 44, 33] on Conditional Random Fields (CRFs) /Markov Random Fields (MRFs).
Although this paper discusses an ngram-HMM language model which we introduce as one model of
NPLM where we borrow many of the mechanism from infinite HMM [19] and hierarchical PitmanYor LM [48], one main contribution would be to show one new application area of NPLM in SMT.
Although several applications of NPLM have been presented, there have been no application to thetask of system combination as far as we know.
The remainder of this paper is organized as follows. Section 2 describes ngram-HMM language
model while Section 3 introduces a joint space model of ngram-HMM language model. In Section
4, our intrinsic experimental results are presented, while in Section 5 our extrinsic experimental
results are presented. We conclude in Section 5.

075
076
077
078
079
080
081
082
083
084

2 Ngram-HMM Language Model
Generative model Figure 1 depicted an example of ngram-HMM language model, i.e. 4-gramHMM language model in this case, in blue (in the center). We consider a Hidden Markov
Model (HMM) [40, 21, 2] of size K which emits n-gram word sequence wi , . . . , wi−K+1 where
hi , . . . , hi−K+1 denote corresponding hidden states. The arcs from wi−3 to wi , · · · , wi−1 to wishow the back-off relations appeared in language model smoothing, such as Kneser-Ney smoothing[26], Good-Turing smoothing [24], and hierarchical Pitman-Yor LM smoothing [48].

085
086
087
088
089
090
091
092

γ

β

α

πk

hi−3

hi−2

h i−1

hi

H

ϕ

w
i−3

wi−2

wi−1

wi

G0
θ0
G0

d0

093
094
095
096

k=1,...,TGu
u=1,...,K

θu
du

097
098
099
100

Figure 1: Figure shows a graphical representation of the 4-gram HMM language model.

101
102
103
104
105
106
107

In the left side in Figure 1, we place one Dirichlet Process prior DP(α, H), with concentration parameter α and base measure H, for the transition probabilities going out from each hidden state.
This construction is borrowed from the infinite HMM [2, 19]. The observation likelihood for thehidden word ht are parameterized as in wt |ht ∼ F (φht ) since the hidden variables of HMM is limited in its representation power where φht denotes output parameters. This is since the observations
can be regarded as being generated from a dynamic mixture model [19] as in (1), the Dirichlet priors
2

108
109
110
111
112

on the rows have a shared parameter.
p(wi |hi−1 = k) =

=

119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161

K
X

πk,hi p(wi |φhi )

(1)

hi =1

115

118

p(hi |hi−1 = k)p(wi |hi )

hi =1

113
114
116
117

K
X

In the right side in Figure 1, we place Pitman-Yor prior PY, which has advantage in its power-law
behavior as our target is NLP, as in (2):wi |w1:i−1∼

PY(di , θi , Gi )(2)

where α is a concentration parameter, θ is a strength parameter, and Gi is a base measure. This
construction is borrowed from hierarchical Pitman-Yor language model [48].
Inference We compute the expected value of the posterior distribution of the hidden variables witha beam search [19]. This blocked Gibbs sampler alternate samples the parameters (transition matrix,
output parameters), the state sequence, hyper-parameters, and the parameters related to language
model smoothing. As is mentioned in [19], this sampler has characteristic in that it adaptively
truncates the state space and run dynamic programming as in (3):Xp(ht−1 |w1:t−1 , u1:t−1 )(3)p(ht |w1:t , u1:t ) = p(wt |ht )ht−1 :ut <π (ht−1 ,ht )where ut is only valid if this is smaller than the transition probabilities of the hidden word sequenceh1 , . . . , hK . Note that we use an auxiliary variable ui which samples for each word in the sequence
from the distribution ui ∼ Uniform(0, π (hi−1 ,hi ) ). The implementation of the beam sampler consists of preprocessing the transition matrix π and sorting its elements in descending order.
Initialization First, we obtain the parameters for hierarchical Pitman-Yor process-based language
model [48, 23], which can be obtained using a block Gibbs sampling [32].
Second, in order to obtain a better initialization value h for the above inference, we perform thefollowing EM algorithm instead of giving the distribution of h randomly. This EM algorithm incorporates the above mentioned truncation [19]. In the E-step, we compute the expected value ofthe posterior distribution of the hidden variables. For every position hi , we send a forward messageα(hi−n+1:i−1 ) in a single path from the start to the end of the chain (which is the standard forward
recursion in HMM; Hence we use α). Here we normalize the sum of α considering the truncatedvariables ui−n+1:i−1 .
PXα(hi−n+1:i−1 )Pα(hi−n+2:i ) =P (wi |hi )α(ui−n+1:i−1 )P (hi |hi−n+1:i−1 )(4)α(ui−n+1:i−1 )Then, for every position hj , we send a message β(hi−n+2:i , hj ) in multiple paths from the start tothe end of the chain as in (5),PXα(hi−n+1:i−1 )β(hi−n+2:i , hj ) = P
P (wi |hi )β(hi−n+1:i−1 , hj )P (hi |hi−n+1:i−1 ) (5)α(ui−n+1:i−1 )This step aims at obtaining the expected value of the posterior distribution (Similar construction touse expectation can be seen in factored HMM [22]). In the M-step, using this expected value ofthe posterior distribution obtained in the E-step to evaluate the expectation of the logarithm of thecomplete-data likelihood.

3 Joint Space ModelIn this paper, we mechanically introduce a joint space model. Other than the ngram-HMM language
model obtained in the previous section, we will often encounter the situation where we have anotherhidden variables h1 which is irrelevant to h0 which is depicted in Figure 2. Suppose that we have
3

162
163
164
165
166
167
168
169

the ngram-HMM language model yielded the hidden variables suggesting semantic and syntacticalrole of words. Adding to this, we may have another hidden variables suggesting, say, a genre ID.
This genre ID can be considered as the second context which is often not closely related to the firstcontext. This also has an advantage in this mechanical construction that the resulted language model
often has the perplexity smaller than the original ngram-HMM language model. Note that we do
not intend to learn this model jointly using the universal criteria, but we just concatenate the labels
by different tasks on the same sequence. By this formulation, we intend to facilitate the use of thislanguage model.

170
171

h1i−3

h1i−2

1
hi−1

0

0

0

hi1

172
173
174

0

175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195

Figure 2: Figure shows the joint space 4-gram HMM language model.
It is noted that those two contexts may not be derived in a single learning algorithm. For example,
language model with the sentence context may be derived in the same way with that with the word
context. In the above example, a hidden semantics over sentence is not a sequential object. Hence,
this can be only considering all the sentence are independent. Then, we can obtain this using, say,
LDA.

4 Intrinsic Evaluation
We compared the perplexity of ngram-HMM LM (1 feature), ngram-HMM LM (2 features, the sameas in this paper and genre ID is 4 class), modified Kneser-Ney smoothing (irstlm) [18], and hierarchical Pitman Yor LM [48]. We used news2011 English testset. We trained LM using Europarl.
Europarl 1500k

ngram-HMM (1 feat)114.014

ngram-HMM (2 feat)113.450

modified Kneser-Ney
118.890hierarchical PY
118.884

196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215

Table 1: Table shows the perplexity of each language model.

5 Extrinsic Evaluation: Task of System Combination
We applied ngram-HMM language model to the task of system combination. For given multiple
Machine Translation (MT) outputs, this task essentially combines the best fragments among given
MT outputs to recreate a new MT output. The standard procedure consists of three steps: Minimum
Bayes Risk decoding, monolingual word alignment, and monotonic consensus decoding. Althoughthese procedures themselves will need explanations in order to understand the following, we keep
the main text in minimum, moving some explanations (but not sufficient) in appendices. Note that
although this experiment was done using the ngram-HMM language model, any NPLM may besufficient for this purpose. In this sense, we use the term NPLM instead of ngram-HMM language
model.
Features in Joint Space The first feature of NPLM is the semantically and syntactically similarwords of roles, which can be derived from the original NPLM. We introduce the second feature inthis paragraph, which is a genre ID.
The motivation to use this feature comes from the study of domain adaptation for SMT where it becomes popular to consider the effect of genre in testset. This paper uses Latent Dirichlet Allocation
4

216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244

(LDA) [5, 46, 6, 45, 33] to obtain the genre ID via (unsupervised) document classification since ourinterest here is on the genre of sentences in testset. And then, we place these labels on a joint space.
LDA represents topics as multinomial distributions over the W unique word-types in the corpus andrepresents documents as a mixture of topics. Let C be the number of unique labels in the corpus.
Each label c is represented by a W -dimensional multinomial distribution φc over the vocabulary.
For document d, we observe both the words in the document w(d) as well as the document labels
c(d) . Given the distribution over topics θd , the generation of words in the document is captured by
the following generative model.The parameters α and β relate to the corpus level, the variables θdbelong to the document level, and finally the variables zdn and wdn correspond to the word level,
which are sampled once for each word in each document.
Using topic modeling in the second step, we propose the overall algorithm to obtain genre IDs fortestset as in (5).
1. Fix the number of clusters C, we explore values from small to big where the optimal value
will be searched on tuning set.
2. Do unsupervised document classification (or LDA) on the source side of the tuning and test
sets.
(a) For each label c ∈ {1, . . . C}, sample a distribution over word-types φc ∼Dirichlet(·|β)(b) For each document d ∈ {1, . . . , D}
i. Sample a distribution over its observed labels θd ∼ Dirichlet(·|α)
ii. For each word i ∈ {1, . . . , NdW }(d)∼ Multinomial(θd )(d)∼ Multinomial(φc ) from the label c = zi

A. Sample a label zi

B. Sample a word wi(d)

3. Separate each class of tuning and test sets (keep the original index and new index in the
allocated separated dataset).
4. (Run system combination on each class.)

245
246

5. (Reconstruct the system combined results of each class preserving the original index.)

247

Modified Process in System Combination Given a joint space of NPLM, we need to specify
in which process of the task of system combination among three processes use this NPLM. We
only discuss here the standard system combination using confusion-network. This strategy takes thefollowing three steps (Very brief explanation of these three is available in Appendix):

248
249
250
251
252
253
254
255
256
257

• Minimum Bayes Risk decoding [28] (with Minimum Error Rate Training (MERT) process
[34])
X
MBR
Êbest= argminE ′ ∈E R(E ′ ) = argminE ′ ∈E
L(E, E ′ )P (E|F )=argminE ′ ∈E

261
262
263
264
265
266
267
268
269

(1 − BLEUE (E ′ ))P (E|F )E ′ ∈EE

258
259
260

E ′ ∈EEX• Monolingual word alignment• (Monotone) consensus decoding (with MERT process)
Ebest

=

arg maxeIYφ(i|ēi )pLM (e)i=1Similar to the task of n-best reranking in MERT process [43], we consider the reranking of nbest
lists in the third step of above, i.e. (monotone) consensus decoding (with MERT process). We do
not discuss the other two processes in this paper.
On one hand, we intend to use the first feature of NPLM, i.e. the semantically and syntacticallysimilar role of words, for paraphrases. The n-best reranking in MERT process [43] alternate the
5

270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323

probability suggested by word sense disambiguation task using the feature of NPLM, while we
intend to add a sentence which replaces the words using NPLM. On the other hand, we intend touse the second feature of NPLM, i.e. the genre ID, to split a single system combination system into
multiple system combination systems based on the genre ID clusters. In this perspective, the role ofthese two feature can be seen as independent. We conducted four kinds of settings below.
(A) —First Feature: N-Best Reranking in Monotonic Consensus Decoding without Noise –NPLM plain In the first setting for the experiments, we used the first feature without considering
noise. The original aim of NPLM is to capture the semantically and syntactically similar words
in a way that a latent word depends on the context. We will be able to get variety of words if we
condition on the fixed context, which would form paraphrases in theory.
We introduce our algorithm via a word sense disambiguation (WSD) task which selects the rightdisambiguated sense for the word in question. This task is necessary due to the fact that a text isnatively ambiguous accommodating with several different meanings. The task of WSD [14] can be
written as in (6):P (synseti |featuresi , θ)

=

Y
k
1
g(synseti , k)f (featurei )Z(features) m(6)

where k ranges over all possible features, f (featureki ) is an indicator function whose value is 1 if
the feature exists, and 0 otherwise, g(synseti , k) is a parameter for a given synset and feature, θ is acollection of all these parameters in g(synseti , k), and Z is a normalization constant. Note that we
use the term “synset” as an analogy of the WordNet [30]: this is equivalent to “sense” or “meaning”.
Note also that NPLM will be included as one of the features in this equation. If features include
sufficient statistics, a task of WSD will succeed. Otherwise, it will fail. We do reranking of theoutcome of this WSD task.
On the one hand, the paraphrases obtained in this way have attractive aspects that can be called“a creative word” [50]. This is since the traditional resource that can be used when building atranslation model by SMT are constrained on parallel corpus. However, NPLM can be trained onhuge monolingual corpus. On the other hand, unfortunately in practice, the notorious training timeof NPLM only allows us to use fairly small monolingual corpus although many papers made aneffort to reduce it [31]. Due to this, we cannot ignore the fact that NPLM trained not on a huge
corpus may be affected by noise. Conversely, we have no guarantee that such noise will be reduced
if we train NPLM on a huge corpus. It is quite likely that NPLM has a lot of noise for small corpora.
Hence, this paper also needs to provide the way to overcome difficulties of noisy data. In order toavoid this difficulty, we limit the paraphrase only when it includes itself in high probability.
(B)— First Feature: N-Best Reranking in Monotonic Consensus Decoding with Noise – NPLM
dep In the second setting for our experiment, we used the first feature considering noise. Although
we modified a suggested paraphrase without any intervention in the above algorithm, it is also possible to examine whether such suggestion should be adopted or not. If we add paraphrases and theresulted sentence has a higher score in terms of the modified dependency score [39] (See Figure 3),
this means that the addition of paraphrases is a good choice. If the resulted score decreases, we do
not need to add them. One difficulty in this approach is that we do not have a reference which allows
us to score it in the usual manner. For this reason, we adopt the naive way to deploy the above andwe deploy this with pseudo references. (This formulation is equivalent that we decode these inputs
by MBR decoding.) First, if we add paraphrases and the resulted sentence does not have a very badscore, we add these paraphrases since these paraphrase are not very bad (naive way). Second, we
do scoring between the sentence in question with all the other candidates (pseudo references) andcalculate an average of them. Thus, our second algorithm is to select a paraphrase which may notachieve a very bad score in terms of the modified dependency score using NPLM.
(C) — Second Feature: Genre ID — DA (Domain Adaptation) In the third setting of our experiment, we used only the second feature. As is mentioned in the explanation about this feature,
we intend to splits a single module of system combination into multiple modules of system combi6

324
325

c−structure

326

NP
John

f−structure
SUBJ

PRED john
NUM sgPERS 3
PRED resign
TENSE past
ADJ([PRED yesterday])

S

327
328

VP

V
resigned

329
330

NP
yesterday

Same representationin f−structure

Different structurein c−structure

331

S

332
333

NP
NP
Yesterday John

SUBJ
VP
PRED
TENSE
ADJ

V
resigned

334

PRED john
NUM sgPERS 3
resignpast([PRED yesterday])

335
336
337
338
339

Figure 3: By the modified dependency score [39], the score of these two sentences, “John resigned
yesterday” and “Yesterday John resigned”, are the same. Figure shows c-structure and f-structure oftwo sentences using Lexical Functional Grammar (LFG) [8].

340
341
342
343
344
345
346
347
348
349

nation according to the genre ID. Hence, we will use the module of system combination tuned forthe specific genre ID, 1 .
(D) — First and Second Feature — COMBINED In the fourth setting we used both features.
In this setting, (1) we used modules of system combination which are tuned for the specific genre
ID, and (2) we prepared NPLM whose context can be switched based on the specific genre of thesentence in test set. The latter was straightforward since these two features are stored in joint spacein our case.

350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373

Experimental Results ML4HMT-2012 provides four translation outputs (s1 to s4) which are
MT outputs by two RBMT systems, APERTIUM and L UCY, PB-SMT (M OSES ) and HPB-SMT(M OSES ), respectively. The tuning data consists of 20,000 sentence pairs, while the test data consists of 3,003 sentence pairs.
Our experimental setting is as follows. We use our system combination module [16, 17, 35], which
has its own language modeling tool, MERT process, and MBR decoding. We use the BLEU metricas loss function in MBR decoding. We use TER P2 as alignment metrics in monolingual wordalignment. We trained NPLM using 500,000 sentence pairs from English side of EN-ES corpus of
E UROPARL3 .
The results show that the first setting of NPLM-based paraphrased augmentation, that is NPLM
plain, achieved 25.61 BLEU points, which lost 0.39 BLEU points absolute over the standard system combination. The second setting, NPLM dep, achieved slightly better results of 25.81 BLEU
points, which lost 0.19 BLEU points absolute over the standard system combination. Note thatthe baseline achieved 26.00 BLEU points, the best single system in terms of BLEU was s4 which
achieved 25.31 BLEU points, and the best single system in terms of METEOR was s2 which
achieved 0.5853. The third setting achieved 26.33 BLEU points, which was the best among ourfour settings. The fourth setting achieved 25.95, which is again lost 0.05 BLEU points over the
standard system combination.
Other than our four settings where these settings differ which features to use, we run several different settings of system combination in order to understand the performance of four settings. Standard
system combination using BLEU loss function (line 5 in Table 2), standard system combination
using TER loss function (line 6), system combination whose backbone is unanamously taken fromthe RBMT outputs (MT input s2 in this case; line 11), and system combination whose backbone isselected by the modified dependency score (which has three variations in the figure; modDep preci-

374
375
376
377

1

E.g., we translate newswire with system combination module tuned with newswire tuning set, while wetranslate medical text with system combination module tuned with medical text tuning set.
2
http://www.cs.umd.edu/∼snover/terp3
http://www.statmt.org/europarl

7

378
379
380
381
382
383
384

sion, recall and Fscore; line 12, 13 and 14). One interesting characteristics is that the s2 backbone(line 11) achieved the best score among all of these variations. Then, the score of the modified
dependency measure-selected backbone follows. From these runs, we cannot say that the runs related to NPLM, i.e. (A), (B) and (D), were not particularly successful. The possible reason for thiswas that our interface with NPLM was only limited to paraphrases, which was not very successfuly
chosen by reranking.

385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403

MT input s1
MT input s2
MT input s3
MT input s4
standard system combination (BLEU)
standard system combination (TER)(A) NPLM plain(B) NPLM dep(C) DA(D) COMBINED
s2 backbone
modDep precision
modDep recall
modDep Fscore

NIST
6.49966.9281
7.40227.21007.68467.62317.6041
7.62137.7146
7.6464
7.6371
7.66707.66957.6695BLEU
0.22480.25000.24460.25310.26000.26380.25610.25810.26330.25950.26480.2636
0.26420.2642

METEOR
0.5458641
0.5853446
0.5544660
0.5596933
0.5643944
0.5652795
0.5593901
0.5601121
0.5647685
0.5610121
0.5606801
0.5659757
0.5664320
0.5664320

WER
64.2452
62.919458.0752
59.3930
56.2368
56.3967
56.462056.1334
55.861256.0101
56.0077
56.439356.5059
56.5059

PER
49.9806
48.006544.022144.5230
41.539941.6092
41.8076
41.782041.7264
41.770242.0075
41.4986
41.501341.5013

Table 2: This table shows single best performance, the performance of the standard system combination (BLEU and TER loss functions), the performance of four settings in this paper ((A),. . .,(D)), theperformance of s2 backboned system combination, and the performance of the selection of sentencesby modified dependency score (precision, recall, and F-score each).

404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419

Conclusion and Perspectives
This paper proposes a non-parametric Bayesian way to interpret NPLM, which we call ngramHMM language model. Then, we add a small extension to this by concatenating other context
in the same model, which we call a joint space ngram-HMM language model. The main issues
investigated in this paper was an application of NPLM in bilingual NLP, specifically Statistical
Machine Translation (SMT). We focused on the perspectives that NPLM has potential to open thepossibility to complement potentially ‘huge’ monolingual resources into the ‘resource-constraint’bilingual resources. We compared our proposed algorithms and others. One discovery was thatwhen we use a fairly small NPLM, noise reduction may be one way to improve the quality. In ourcase, the noise reduced version obtained 0.2 BLEU points better.
Further work would be to apply this NPLM in various other tasks in SMT: word alignment, hierarchical phrase-based decoding, and semantic incorporated MT systems in order to discover the meritof ‘depth’ of architecture in Machine Learning.

420
421
422
423
424
425
426
427
428
429
430
431

References[1] BANGALORE , S., B ORDEL , G., AND R ICCARDI , G. Computing consensus translation from multiple machine translation systems. In Proceedings of the IEEE
Automatic Speech Recognition and Understanding Workshop (ASRU) (2001), 350–354.
[2] B EAL , M. J. Variational algorithms for approximate bayesian inference. PhD Thesis at Gatsby Computational Neuroscience Unit, University College London(2003).
[3] B ENGIO , Y., D UCHARME , R., AND V INCENT, P. A neural probabilistic language model. In Proceedings of Neural Information Systems (2000).
[4] B ENGIO , Y., S CHWENK , H., S EN ÉCAL , J.-S., M ORIN , F., AND G AUVAIN , J.-L. Neural probabilistic language models. Innovations in Machine Learning:
Theory and Applications Edited by D. Holmes and L. C. Jain (2005).
[5] B LEI , D., N G , A. Y., AND J ORDAN , M. I. Latent dirichlet allocation. Journal of Machine Learning Research 3 (2003), 9931022.
[6] B LEI , D. M. Introduction to probabilistic topic models. Communications of the ACM (2011).
[7] B ORDES , A., G LOROT, X., W ESTON , J., AND B ENGIO , Y. Towards open-text semantic parsing via multi-task learning of structured embeddings. CoRR
abs/1107.3663 (2011).
[8] B RESNAN , J. Lexical functional syntax. Blackwell (2001).
[9] C HEN , S., AND G OODMAN , J. An empirical study of smoothing techniques for language modeling. Technical report TR-10-98 Harvard University (1998).

8

432
433

[10] C OLLOBERT, R. Deep learning for efficient discriminative parsing. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics(AISTATS) (2011).

434[11] C OLLOBERT, R., AND W ESTON , J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International
Conference on Machine Learning (ICML 2008) (2008).

435
436

[12] C OLLOBERT, R., W ESTON , J., B OTTOU , L., K ARLEN , M., K AVUKCUOGLU , K., AND K UKSA , P. Natural language processing (almost) from scratch. Journalof Machine Learning Research 12 (2011), 2493–2537.

437
438

[13] D E N ERO , J., C HIANG , D., AND K NIGHT, K. Fast consensus decoding over translation forests. In proceedings of the Joint Conference of the 47th Annual
Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (2009), 567–575.

439[15] D U , J., H E , Y., P ENKALE , S., AND WAY, A. MaTrEx: the DCU MT System for WMT 2009. In Proceedings of the Third EACL Workshop on Statistical
Machine Translation (2009), 95–99.

440
441
442
443
444
445
446
447
448
449

[14] D ESCHACHT, K., B ELDER , J. D., AND M OENS , M.-F. The latent words language model. Computer Speech and Language 26 (2012), 384–409.

[16] D U , J., AND WAY, A. An incremental three-pass system combination framework by combining multiple hypothesis alignment methods. International Journalof Asian Language Processing 20, 1 (2010), 1–15.
[17] D U , J., AND WAY, A. Using terp to augment the system combination for smt. In Proceedings of the Ninth Conference of the Association for Machine Translation(AMTA2010) (2010).
[18] F EDERICO , M., B ERTOLDI , N., AND C ETTOLO , M. Irstlm: an open source toolkit for handling large scale language models. Proceedings of Interspeech(2008).
[19] G AEL , J. V., V LACHOS , A., AND G HAHRAMANI , Z. The infinite hmm for unsupervised pos tagging. The 2009 Conference on Empirical Methods on Natural
Language Processing (EMNLP 2009) (2009).
[20] G ASTHAUS , J., W OOD , F., AND T EH , Y. W. Lossless compression based on the sequence memoizer. DCC 2010 (2010).
[21] G HAHRAMANI , Z. An introduction to hidden markov models and bayesian networks. International Journal of Pattern Recognition and Artificial Intelligence15, 1 (2001), 9–42.
[22] G HAHRAMANI , Z., J ORDAN , M. I., AND S MYTH , P. Factorial hidden markov models. Machine Learning (1997).

450[23] G OLDWATER , S., G RIFFITHS , T. L., AND J OHNSON , M. Contextual dependencies in unsupervised word segmentation. In Proceedings of Conference on
Computational Linguistics / Association for Computational Linguistics (COLING-ACL06) (2006), 673–680.

451
452

[24] G OOD , I. J. The population frequencies of species and the estimation of population paramters. Biometrika 40, (3-4) (1953), 237–264.

453
454

[25] H INTON , G. E., M C C LELLAND , J. L., AND RUMELHART, D. Distributed representations. Parallel Distributed Processing: Explorations in the Microstructureof Cognition(Edited by D.E. Rumelhart and J.L. McClelland) MIT Press 1 (1986).
[26] K NESER , R., AND N EY, H. Improved backing-off for n-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speechand Signal Processing (1995), 181–184.

455[27] KOLLER , D., AND F RIEDMAN , N. Probabilistic graphical models: Principles and techniques. MIT Press (2009).

456
457

[28] K UMAR , S., AND B YRNE , W. Minimum Bayes-Risk word alignment of bilingual texts. In Proceedings of the Empirical Methods in Natural Language
Processing (EMNLP 2002) (2002), 140–147.

458[29] M ATUSOV, E., U EFFING , N., AND N EY, H. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment.
In Proceedings of the 11st Conference of the European Chapter of the Association for Computational Linguistics (EACL) (2006), 33–40.

459
460

[31] M NIH , A., AND T EH , Y. W. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the International Conference on
Machine Learning (2012).

461
462

[32] M OCHIHASHI , D., YAMADA , T., AND U EDA , N. Bayesian unsupervised word segmentation with nested pitman-yor language modeling. In Proceedings ofJoint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language
Processing of the Asian Federation of Natural Language Processing (ACL-IJCNLP 2009) (2009), 100–108.

463[33] M URPHY, K. P. Machine learning: A probabilistic perspective. The MIT Press (2012).

464
465

[34] O CH , F., AND N EY, H. A systematic comparison of various statistical alignment models. Computational Linguistics 29, 1 (2003), 19–51.

466
467
468
469
470
471
472
473

[30] M ILLER , G. A. Wordnet: A lexical database for english. Communications of the ACM 38, 11 (1995), 39–41.

[35] O KITA , T., AND VAN G ENABITH , J. Minimum bayes risk decoding with enlarged hypothesis space in system combination. In Proceedings of the 13th
International Conference on Intelligent Text Processing and Computational Linguistics (CICLING 2012). LNCS 7182 Part II. A. Gelbukh (Ed.) (2012), 40–51.
[36] O KITA , T., AND WAY, A. Hierarchical pitman-yor language model in machine translation. In Proceedings of the International Conference on Asian Language
Processing (IALP 2010) (2010).
[37] O KITA , T., AND WAY, A. Pitman-Yor process-based language model for Machine Translation. International Journal on Asian Language Processing 21, 2(2010), 57–70.
[38] O KITA , T., AND WAY, A. Given bilingual terminology in statistical machine translation: Mwe-sensitve word alignment and hierarchical pitman-yor processbased translation model smoothing. In Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference (FLAIRS-24) (2011),
269–274.
[39] OWCZARZAK , K., VAN G ENABITH , J., AND WAY, A. Evaluating machine translation with LFG dependencies. Machine Translation 21, 2 (2007), 95–119.
[40] R ABINER , L. R. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE 77, 2 (1989), 257–286.
[41] S CHWENK , H. Continuous space language models. Computer Speech and Language 21 (2007), 492–518.

474[42] S CHWENK , H. Continuous space language models for statistical machine translation. The Prague Bulletin of Mathematical Linguistics 83 (2010), 137–146.

475
476

[43] S CHWENK , H., ROUSSEAU , A., AND ATTIK , M. Large, pruned or continuous space language models on a gpu for statistical machine translation. In Proceedingof the NAACL workshop on the Future of Language Modeling (2012).

477
478
479
480
481
482
483
484
485

[44] S ONTAG , D. Approximate inference in graphical models using LP relaxations. Massachusetts Institute of Technology (Ph.D. thesis) (2010).
[45] S ONTAG , D., AND ROY, D. M. The complexity of inference in latent dirichlet allocation. In Advances in Neural Information Processing Systems 24 (NIPS)(2011).
[46] S TEYVERS , M., AND G RIFFITHS , T. Probabilistic topic models. Handbook of Latent Semantic Analysis. Psychology Press (2007).
[47] S TOLCKE , A. SRILM – An extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing (2002),
901–904.
[48] T EH , Y. W. A hierarchical bayesian language model based on pitman-yor processes. In Proceedings of the 44th Annual Meeting of the Association for
Computational Linguistics (ACL-06), Prague, Czech Republic (2006), 985–992.
[49] T ROMBLE , R., K UMAR , S., O CH , F., AND M ACHEREY, W. Lattice minimum bayes-risk decoding for statistical machine translation. Proceedings of the 2008
Conference on Empirical Methods in Natural Language Processing (2008), 620–629.
[50] V EALE , T. Exploding the creativity myth: The computational foundations of linguistic creativity. London: Bloomsbury Academic (2012).
[51] W OOD , F., A RCHAMBEAU , C., G ASTHAUS , J., JAMES , L., AND T EH , Y. W. A stochastic memoizer for sequence data. In Proceedings of the 26th International
Conference on Machine Learning (2009), 1129–1136.

9