A Binarized Neural Network Joint Model for Machine Translation
Jingyi Zhang1,2 , Masao Utiyama1 , Eiichro Sumita1
Graham Neubig2 , Satoshi Nakamura2
1
National Institute of Information and Communications Technology,
3-5Hikaridai, Keihanna Science City, Kyoto 619-0289, Japan
2
Graduate School of Information Science, Nara Institute of Science and Technology,
Takayama, Ikoma, Nara 630-0192, Japan
jingyizhang/mutiyama/eiichiro.sumita@nict.go.jp
neubig/s-nakamura@is.naist.jp
P(ti=1)P(ti=2)

Abstract
m-wordsource context

The neural network joint model (NNJM),
which augments the neural network language model (NNLM) with an m-wordsource context window, has achieved large
gains in machine translation accuracy, butalso has problems with high normalization
cost when using large vocabularies. Training the NNJM with noise-contrastive estimation (NCE), instead of standard maximum likelihood estimation (MLE), can reduce computation cost. In this paper, we
propose an alternative to NCE, the binarized NNJM (BNNJM), which learns a binary classifier that takes both the context
and target words as input, and can be efficiently trained using MLE. We compare
the BNNJM and NNJM trained by NCE
on various translation tasks.

1

ti-n+1~ti-1
P(ti=N)(a)m-wordsource context

P(ti is correct)
P(ti is wrong)

ti-n+1~ti(b)

Figure 1: (a) the traditional NNJM and (b) the proposed BNNJM

Introduction

Neural network translation models, which learn
mappings over real-valued vector representations
in high-dimensional space, have recently achievedlarge gains in translation accuracy (Hu et al., 2014;
Devlin et al., 2014; Sundermeyer et al., 2014;
Auli et al., 2013; Schwenk, 2012; Sutskever et al.,2014; Bahdanau et al., 2015).
Notably, Devlin et al. (2014) proposed a neuralnetwork joint model (NNJM), which augments then-gram neural network language model (NNLM)
with an m-word source context window, as shownin Figure 1a. While this model is effective, thecomputation cost of using it in a large-vocabulary
SMT task is quite expensive, as probabilities need
to be normalized over the entire vocabulary. To
solve this problem, Devlin et al. (2014) presented a technique to train the NNJM to be selfnormalized and avoided the expensive normalization cost during decoding. However, they also

note that this self-normalization technique sacrifices neural network accuracy, and the training
process for the self-normalized neural network isvery slow, as with standard maximum likelihood
estimation (MLE).
To remedy the problem of long training times
in the context of NNLMs, Vaswani et al. (2013)used a method called noise contrastive estimation(NCE). Compared with MLE, NCE does not require repeated summations over the whole vocabulary and performs nonlinear logistic regression todiscriminate between the observed data and artificially generated noise.
This paper proposes an alternative framework ofbinarized NNJMs (BNNJM), which are similar to
the NNJM, but use the current target word not as
the output, but as the input of the neural network,
estimating whether the target word under examination is correct or not, as shown in Figure 1b.
Because the BNNJM uses the current target word
as input, the information about the current target
word can be combined with the context word information and processed in the hidden layers.
The BNNJM learns a simple binary classifier,given the context and target words, therefore it
can be trained by MLE very efficiently. “Incorrect” target words for the BNNJM can be generated in the same way as NCE generates noise

2094
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2094–2099,
Lisbon, Portugal, 17-21 September 2015. c 2015 Association for Computational Linguistics.

for the NNJM. We present a novel noise distribution based on translation probabilities to train the
NNJM and the BNNJM efficiently.

2

Neural Network Joint Model
|T |

|S|

Let T = t1 be a translation of S = s1 . The
NNJM (Devlin et al., 2014) defines the following
probability,
P (T |S) =

Y|T |

i=1



a +(m−1)/2

P ti |saii −(m−1)/2 , ti−1
i−n+1



(1)

where target word ti is affiliated with source word
sai . Affiliation ai is derived from the word alignments using heuristics1 . To estimate these probabilities, the NNJM uses m source context words
and n − 1 target history words as input to a neuralnetwork and performs estimation of unnormalized
probabilities p (ti |C) before normalizing over allwords in the target vocabulary V ,i |C)P (ti |C) = p(tPZ(C) 0Z (C) =p (ti |C)(2)

ti 0 ∈V

where C stands for source and target context
words as in Equation 1.
The NNJM can be trained on a word-aligned
parallel corpus using standard MLE, but the costof normalizing over the entire vocabulary to calculate the denominator in Equation 2 is quite large.
Devlin et al. (2014)’s self-normalization technique
can avoid normalization cost during decoding, but
not during training.
NCE can be used to train NNLM-style models(Vaswani et al., 2013) to reduce training times.
NCE creates a noise distribution q (ti ), selects knoise samples ti1 , ..., tik for each ti and introduces
a random variable v which is 1 for training examples and 0 for noise samples,1
i |C)
P (v = 1, ti |C) = 1+k
· p(t
Z(C)
k
P (v = 0, ti |C) = 1+k· q (ti ) .

NCE trains the model to distinguish training
data from noise by maximize the conditional likelihood,
L = log P (v = 1|C, ti ) +

k
P

j=1

log P (v = 0|C, tik ).

The normalization cost can be avoided by usingp (ti |C) as an approximation of P (ti |C).2
1

If ti aligns to exactly one source word, ai is the index ofthis source word; If ti aligns to multiple source words, ai is
the index of the aligned word in the middle; If ti is unaligned,
they inherit its affiliation from the closest aligned word.
2
The theoretical properties of self-normalization techniques, including NCE and Devlin et al. (2014)’s method,
are investigated by Andreas and Klein (2015).

3

Binarized NNJM

In this paper, we propose a new framework of thebinarized NNJM (BNNJM), which is similar to
the NNJM but learns not to predict the next word
given the context, but solves a binary classification problem by adding a variable v ∈ {0, 1} thatstands for whether the current target word ti is correctly/wrongly produced in terms of source cona +(m−1)/2
text words saii −(m−1)/2 and target history words
i−1
ti−n+1,

a +(m−1)/2

P v|saii −(m−1)/2 , ti−1
i−n+1 , ti .
The BNNJM is learned by a feedforward
neural networko with m + n inputs
nai +(m−1)/2 i−1
sai −(m−1)/2 , ti−n+1 , ti and two outputs forv = 1/0.
Because the BNNJM uses the current target
word as input, the information about the current
target word can be combined with the context
word information and processed in the hidden layers. Thus, the hidden layers can be used to learn
the difference between correct target words andnoise in the BNNJM, while in the NNJM the hidden layers just contain information about context
words and only the output layer can be used to discriminate between the training data and noise, giving the BNNJM more power to learn this classification problem.
We can use the BNNJM probability in translation as an approximation for the NNJM as below,

a +(m−1)/2

P ti |saii −(m−1)/2 , ti−1
i−n+1
a +(m−1)/2≈ P v = 1|saii −(m−1)/2 , ti−1
i−n+1 , ti .
As a binary classifier, the gradient for a single example in the BNNJM can be calculated
efficiently by MLE without it being necessary
to calculate the softmax over the full vocabulary. On the other hand, we need to create“positive” and “negative” examples for the classifier. Positive examples can be extracted directly from the word-aligned
parallel corpus asDE
a +(m−1)/2 i−1
saii −(m−1)/2 , ti−n+1, ti ; Negative examples canbe generated for each positive example in thesame
way that NCE generates
noise data asDE
ai +(m−1)/2 i−10
sai −(m−1)/2 , ti−n+1 , ti , where ti 0 ∈ V \ {ti }.

4

Noise Sampling

4.1

Unigram Noise

Vaswani et al. (2013) adopted the unigram probability distribution (UPD) to sample noise for train-

2095

我
I

会

安排 人

带

你

choose to combine these target/source words as a
new target/source word.3

转转will arrange for someone to take you roundFigure 2: A parallel sentence pair.
ing NNLMs with NCE,0i )q (ti 0 ) = Poccur(toccur(t 00 )ti 00 ∈Vi

where occur (ti 0 ) stands for how many times ti 0
occurs in the training corpus.
4.2

Translation Model NoiseIn this paper, we propose a noise distribution specialized for translation models, such as the NNJM
or BNNJM.
Figure 2 gives a Chinese-to-English parallel
sentence pair with word alignments to demonstrate the intuition behind our method. Focusing
on sai =“安 排”, this is translated into ti =“arrange”. For this positive example, UPD is allowed
to sample any arbitrary noise, such as ti 0 = “banana”. However, in this case, noise ti 0 = “banana”
is not useful for model training, as constraints on
possible translations given by the phrase table ensure that “安排” will never be translated into “banana”. On the other hand, noise ti 0 = “arranges”
and “arrangement” are both possible translationsof “安排” and therefore useful training data, that
we would like our model to penalize.
Based on this intuition, we propose the useof another noise distribution that only uses ti 0
that are possible translations of sai , i.e., ti 0 ∈U (sai ) \ {ti }, where U (sai ) contains all targetwords aligned to sai in the parallel corpus.
Because U (sai ) may be quite large and contain many wrong translations caused by wrongalignments, “banana” may actually be includedin U (“安 排”). To mitigate the effect of uncommon examples, we use a translation probability distribution (TPD) to sample noise ti 0 fromU (sai ) \ {ti } as follows,q (ti 0 |sai ) = P

align(sai ,ti 0 )

(

align
ti 00 ∈U (sai )sai ,ti 00

5

Experiments

5.1

Setting

We evaluated the effectiveness of the proposed approach for Chinese-to-English (CE), Japanese-toEnglish (JE) and French-to-English (FE) translation tasks. The datasets officially provided for thepatent machine translation task at NTCIR-9 (Gotoet al., 2011) were used for the CE and JE tasks.
The development and test sets were both provided
for the CE task while only the test set was provided
for the JE task. Therefore, we used the sentencesfrom the NTCIR-8 JE test set as the developmentset. Word segmentation was done by BaseSeg(Zhao et al., 2006) for Chinese and Mecab4 forJapanese. For the FE language pair, we used standard data for the WMT 2014 translation task. The
training sets for CE, JE and FE tasks contain 1M,
3M and 2M sentence pairs, respectively.
For each translation task, a recent version ofMoses HPB decoder (Koehn et al., 2007) with thetraining scripts was used as the baseline (Base).
We used the default parameters for Moses, anda 5-gram language model was trained on the target side of the training corpus using the IRSTLM
Toolkit5 with improved Kneser-Ney smoothing.
Feature weights were tuned by MERT (Och,2003).
The word-aligned training set was used to learn
the NNJM and the BNNJM.6 For both NNJM and
BNNJM, we set m = 7 and n = 5. The NNJM
was trained by NCE using UPD and TPD as noise
distributions. The BNNJM was trained by standard MLE using UPD and TPD to generate negative examples.
The number of noise samples for NCE was set
to be 100. For the BNNJM, we used only one negative example for each positive example in eachtraining epoch, as the BNNJM needs to calculate3)

where align (sai , ti 0 ) is how many times ti 0 isaligned to sai in the parallel corpus.
Note that ti could be unaligned, in which case
we assume that it is aligned to a special null word.
Noise for unaligned words is sampled according to
the TPD of the null word. If several target/sourcewords are aligned to one source/target word, we

The processing for multiple alignments helps sample
more useful negative examples for TPD, and had little effect on the translation performance when UPD was used as
the noise distribution for the NNJM and the BNNJM in our
preliminary experiments.
4
http://sourceforge.net/projects/mecab/files/
5
http://hlt.fbk.eu/en/irstlm
6
Both the NNJM and the BNNJM had one hidden layer,
100 hidden nodes, input embedding dimension 50, output
embedding dimension 50. A small set of training data wasused as validation data. The training process was stopped
when validation likelihood stopped increasing.

2096

NNJM
BNNJM

UPD
TPD
UPD
TPD

CE
E
T
20
22
4
14
16
11

E
19
6
12
9

JE

T
49
34

FE
E
T
20
28
4
11
22
9

S: 该(this) 移动(movement) 持续(continued) 到(until)
寄生虫(parasite) 由(by) 两(two) 个 舌(tongues) 部 21
彼此(each other) 接触(contact) 时(where) 的 点(point)
接触(touched) 。
R: this movement is continued until the parasite is
touched by the point where the two tongues 21 contact
each other .
T1 : the mobile continues to the parasite from the two
tongue 21 contacts the points of contact with each other .
T2 : this movement is continued until the parasite by two
tongue 21 contact points of contact with each other .

Table 1: Epochs (E) and time (T) in minutes perepoch for each task.
Base
NNJM
BNNJM

UPD
TPD
UPD
TPD

CE
32.95
34.36+
34.60+
32.89
35.05+*

JE
30.13
31.30+
31.50+
30.04
31.42+

FE
24.56
24.68
24.80
24.50
25.84+*

Table 3: Translation examples. Here, S: source;
R: reference; T1 uses NNJM; T2 uses BNNJM.
该− >the移动− >mobile
持续− >continues
到− >toSUM该− >this移动− >movementnull− >is持续− >continued
到− >until
SUM

Table 2: Translation results. The symbol + and *represent significant differences at the p < 0.01
level against Base and NNJM+UPD, respectively.
Significance tests were conducted using bootstrap
resampling (Koehn, 2004).
the whole neural network (not just the output layer
like the NNJM) for each noise sample and thus
noise computation is more expensive. However,for different epochs, we resampled the negative
example for each positive example, so the BNNJM
can make use of different negative examples.
5.2

Results and Discussion

Table 1 shows how many epochs these two models
needed and the training time for each epoch on a
10-core 3.47GHz Xeon X5690 machine.7 Translation results are shown in Table 2.
We can see that using TPD instead of UPD
as a noise distribution for the NNJM trained byNCE can speed up the training process significantly, with a small improvement in performance.
But for the BNNJM, using different noise distributions affects translation performance significantly.
The BNNJM with UPD does not improve overthe baseline system, likely due to the small number of noise samples used in training the BNNJM,
while the BNNJM with TPD achieves good performance, even better than the NNJM with TPD
on the Chinese-to-English and French-to-English
translation tasks.
From Table 2, the NNJM does not improvetranslation performance significantly on the FEtask. Note that the baseline BLEU for the FE
7

The decoding time for the NNJM and the BNNJM weresimilar, since the NNJM trained by NCE uses p (ti |C) as
an approximation of P (ti |C) without normalization and the
BNNJM only needs to be normalized over two output neurons.

NNJM
1.681
-4.506-1.5502.510-1.865-2.414-1.5270.006-0.292-6.846-11.075

BNNJM-0.126-3.758-0.130-0.220-4.236-0.649-0.200-0.055-0.249-0.186-1.341

Table 4: Scores for different translations.
task is lower than CE and JE tasks, indicating thatlearning is harder for the FE task than CE and JE
tasks. The validation perplexities of the NNJM
with UPD for CE, JE and FE tasks are 4.03, 3.49
and 8.37. Despite these difficult learning circumstances and lack of large gains for the NNJM, the
BNNJM improves translations significantly for the
FE task, suggesting that the BNNJM is more robust to difficult translation tasks that are hard for
the NNJM.
Table 3 gives Chinese-to-English translation examples to demonstrate how the BNNJM (with
TPD) helps to improve translations over the
NNJM (with TPD). In this case, the BNNJM helps
to translate the phrase “该 移 动 持 续 到” better. Table 4 gives translation scores for these twotranslations calculated by the NNJM and the BNNJM. Context words are used for predictions but
not shown in the table.
As can be seen, the BNNJM prefers T2 while
the NNJM prefers T1 . Among these predictions,
the NNJM and the BNNJM predict the translation
for “到” most differently. The NNJM clearly predicts that in this case “到” should be translated into
“to” more than “until”, likely because this example rarely occurs in the training corpus. However,the BNNJM prefers “until” more than “to”, which2097

demonstrates the BNNJM’s robustness to less frequent examples.
5.3

NNJM
BNNJM
Imp. (%)

Analysis for JE Translation Results

Finally, we examine the translation results to explore why the BNNJM with TPD did not outperform the NNJM with TPD for the JE translationtask, as it did for the other translation tasks. We
found that using the BNNJM instead of the NNJM
on the JE task did improve translation quality significantly for infrequent words, but not for frequent words.
First, we describe how we estimate translation
quality for infrequent words. Suppose we have atest set S, a reference set R and a translation set T
with I sentences,
Si (1 ≤ i ≤ I) , Ri (1 ≤ i ≤ I) , Ti (1 ≤ i ≤ I)

Ti contains J individual words,
To (Wij ) is how many times Wij occurs in Ti and
Ro (Wij ) is how many times Wij occurs in Ri .
The general 1-gram translation accuracy (Papineni et al., 2002) is calculated as,
I P
J
P

Pg =

i=1 j=1

min(To (Wij ),Ro (Wij ))I P
JPTo (Wij )i=1 j=1

This general 1-gram translation accuracy does notdistinguish word frequency.
We use a modified 1-gram translation accuracy
that weights infrequent words more heavily,
I P
J
P

Pc =

i=1 j=1

min(To (Wij ),Ro (Wij ))·I P
JP
i=1 j=1

1
Occur Wij()

To (Wij )where Occur (Wij ) is how many times Wij occurs in the whole reference set. Note Pc will not
be 1 even in the case of completely accurate translations, but it can approximately reflect infrequent
word translation accuracy, since correct frequent
word translations contribute less to Pc .
Table 5 shows Pg and Pc for different translation tasks. It can be seen that the BNNJM improves infrequent word translation quality similarly for all translation tasks, but improves general translation quality less for the JE task than theother translation tasks. We conjecture that the reason why the BNNJM is less useful for frequent
word translations on the JE task is the fact that
the JE parallel corpus has less accurate function
word alignments than other language pairs, as the

CE

Pc
5.79
5.97
3.1

Pg
68.2
68.4
0.29

JE

Pc
4.15
4.30
3.6

Pg
61.2
61.7
0.81

FE

Pc
6.70
6.86
2.4

Table 5: 1-gram precisions and improvements.
grammatical features of Japanese and English arequite different.8 Wrong function word alignments
will make noise sampling less effective and therefore lower the BNNJM performance for function
word translations. Although wrong word alignments will also make noise sampling less effective for the NNJM, the BNNJM only uses one
noise sample for each positive example, so wrong
word alignments affect the BNNJM more than the
NNJM.

6

Wij ∈ W ords (Ti )

Pg
70.3
70.9
0.85

Related Work

Xu et al. (2011) proposed a method to use binaryclassifiers to learn NNLMs. But they also usedthe current target word in the output, similarly toNCE. The BNNJM uses the current target word as
input, so the information about the current target
word can be combined with the context word information and processed in hidden layers.
Mauser et al. (2009) presented discriminative
lexicon models to predict target words. They
train a separate classifier for each target word, as
these lexicon models use discrete representationsof words and different classifiers do not share features. In contrast, the BNNJM uses real-valued
vector representations of words and shares features, so we train one classifier and can use thesimilarity information between words.

7

Conclusion

This paper proposes an alternative to the NNJM,
the BNNJM, which learns a binary classifier thattakes both the context and target words as input
and combines all useful information in the hidden
layers. We also present a novel noise distribution
based on translation probabilities to train the BNNJM efficiently. With the improved noise sampling method, the BNNJM can achieve comparable performance with the NNJM and even improve
the translation results over the NNJM on Chineseto-English and French-to-English translations.
8
Infrequent words are usually content words and frequentwords are usually function words.

2098

References
Jacob Andreas and Dan Klein. 2015. When and why
are log-linear models self-normalizing? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages
244–249.
Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation
modeling with recurrent neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1044–
1054.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In International Conference on Learning Representations.
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas
Lamar, Richard Schwartz, and John Makhoul. 2014.
Fast and robust neural network joint models for statistical machine translation. In Proceedings of the
52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
1370–1380.
Isao Goto, Bin Lu, Ka Po Chow, Eiichiro Sumita, andBenjamin K Tsou. 2011. Overview of the patentmachine translation task at the NTCIR-9 workshop.
In Proceedings of The 9th NII Test Collection for IR
Systems Workshop Meeting, pages 559–578.
Yuening Hu, Michael Auli, Qin Gao, and Jianfeng Gao.
2014. Minimum translation modeling with recurrent
neural networks. In Proceedings of the 14th Conference of the European Chapter of the Association for
Computational Linguistics, pages 20–29.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses:
Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of
the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster
Sessions, pages 177–180.

the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic
evaluation of machine translation. In Proceedingsof 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
Holger Schwenk. 2012. Continuous space translation
models for phrase-based statistical machine translation. In Proceedings of International Conference on
Computational Linguistics : Posters, pages 1071–
1080.
Martin Sundermeyer, Tamer Alkhouli, Joern Wuebker,
and Hermann Ney. 2014. Translation modelingwith bidirectional recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 14–25.
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014.
Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112.
Ashish Vaswani, Yinggong Zhao, Victoria Fossum,and David Chiang. 2013. Decoding with largescale neural language models improves translation.
In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages
1387–1392.
Puyang Xu, Asela Gunawardana, and Sanjeev Khudanpur. 2011. Efficient subsampling for training complex language models. In Proceedings of the 2011
Conference on Empirical Methods in Natural Language Processing, pages 1128–1136.
Hai Zhao, Chang-Ning Huang, and Mu Li. 2006. An
improved Chinese word segmentation system withconditional random field. In Proceedings of the Fifth
SIGHAN Workshop on Chinese Language Processing, pages 162–165.

Philipp Koehn. 2004. Statistical significance tests formachine translation evaluation. In Proceedings ofthe 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395.
Arne Mauser, Saša Hasan, and Hermann Ney. 2009.
Extending statistical machine translation with discriminative and trigger-based lexicon models. In
Proceedings of the 2009 Conference on Empirical
Methods in Natural Language Processing, pages
210–218.
Franz Josef Och. 2003. Minimum error rate training
in statistical machine translation. In Proceedings of2099