Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16)

A Bilingual Graph-Based Semantic Model for Statistical Machine Translation
Rui Wang,1 Hai Zhao,1,2⇤ Sabine Ploux,3 ⇤ Bao-Liang Lu,1,2 and Masao Utiyama4
1
Department of Computer Science and Eng.
2
Key Lab of Shanghai Education Commission for Intelligent Interaction and Cognitive Eng.
Shanghai Jiao Tong University, Shanghai, China
3
Centre National de la Recherche Scientifique, CNRS-L2C2, France
4
National Institute of Information and Communications Technology, Kyoto, Japanwangrui.nlp@gmail.com, {zhaohai, blu}@cs.sjtu.edu.cn, sploux@isc.cnrs.fr, mutiyama@nict.go.jp
Abstractal., 1990], in which the lexicon is organized conceptually asa set of terms associated with a partition into synsets1 , though
words organized in this way are not conveniently represented
as vectors.
Word embedding for vector representation is usually built
in two-steps. The first is to determine the detailed context
related to a given word. The second is to summary the relationship between word and its context into lower dimensions.
For context determination: 1) The first category is to extract the word or word relations from the entire text, which
is usually regarded as document level processing, such as
bag-of-word, LSA, and LDA. 2) The second category is touse sliding window, such as n-grams, skip-grams and otherlocal co-occurrence relation [Mikolov et al., 2013a; Zou etal., 2013; Levy and Goldberg, 2014; Pennington et al., 2014;
Vulić and Moens, 2015]. 3) The third category, which has
been seldom considered, uses much more sophisticated graphstyle context. Ploux and Ji [2003] describe a graph based semantic matching model using bilingual lexicons and monolingual synonyms2 . They later represent words using individual monolingual co-occurrences [Ji and Ploux, 2003]. Saluja
et al. [2014] propose a graph method to generate translation
candidates using monolingual co-occurrences.
For relationship summarising, neural networks are very
popular for bilingual word embeddings and SMT [Mikolov etal., 2013a; Wang et al., 2013; Zou et al., 2013; Zhang et al.,2014; Gao et al., 2014; Wang et al., 2014; Lauly et al., 2014;
Wang et al., 2015; 2016b]. There are also some works which
use matrix factorization [Ploux and Ji, 2003; Pennington etal., 2014; Shi et al., 2015] and canonical correlation analysis[Faruqui and Dyer, 2014; Lu et al., 2015] for word embedding.
Sense gives more exact meaning formulization than theword form itself. However, most of existing methods embed words as vectors, instead of sense information. Motivated by these inconveniences, we propose Bilingual Contexonym Cliques (BCCs), which are extracted from bilingual Point-wise Mutual Information (PMI) based word cooccurrence graph. BCC plays a role of minimal unit for bilingual sense representation. Correspondence Analysis (CA) is

Most existing bilingual embedding methods for
Statistical Machine Translation (SMT) suffer fromtwo obvious drawbacks. First, they only focus
on simple context such as word count and cooccurrence in document or sliding window to build
word embedding, ignoring latent useful information from selected context. Second, word sense
but not word form is supposed to be the minimal
semantic unit while most existing works are still
for word representation. This paper presents Bilingual Graph-based Semantic Model (BGSM) to alleviate such shortcomings. By means of maximum complete sub-graph (clique) for context selection, BGSM is capable of effectively modelingword sense representation instead of the word form
itself. The proposed model is applied to phrase pairtranslation probability estimation and generationfor SMT. The empirical results show that BGSM
can enhance SMT both in performance (up to +1.3
BLEU) and efficiency in comparison against existing methods.

1

Introduction

Continuous representations of words onto multi-dimensional
vectors enhance traditional natural language processing[Zhao et al., 2010; Zhao and Kit, 2011; Zhao et al., 2013;
Zhang and Zhao, 2013], especially Statistical Machine Translation (SMT), by measuring similarities of words using distances of corresponding vectors [Bengio et al., 2003; Mikolov
et al., 2013b; Wang et al., 2016a]. Most of early works arederived from cognitive processing such as WordNet [Miller et⇤Corresponding authors. R. Wang, H. Zhao and B. L. Lu
were partially supported by Cai Yuanpei Program (CSC No.
201304490199 and No. 201304490171), National Natural Science
Foundation of China (No. 61170114 and No. 61272248), National
Basic Research Program of China (No. 2013CB329401), Major Basic Research Program of Shanghai Science and Technology Committee (No. 15JC1400103), Art and Science Interdisciplinary Fundsof Shanghai Jiao Tong University (No. 14JCRZ04), and Key Projectof National Society Science Foundation of China (No. 15-ZDA041).

1
2

2950

synset is a small group of synonyms labeled as a concept.
http://dico.isc.cnrs.fr

then used for summarizing BCC-word matrix into lower dimension vectors for word representation. This work extends
previous monolingual method [Ploux and Ji, 2003], whichneeds bilingual lexicons or synonyms for bilingual mappingand has been never used for SMT.
The remaining of this paper is organized as follows: theproposed Bilingual Graph-based Semantic Model (BGSM)
is introduced in Section 2, and then it is applied to phrasetranslation probability estimation in Section 3 and phrase pairgeneration in Section 4 to enhance SMT. The experiments andanalysis are given in Section 5. Section 6 summarizes this
work.

2

Bilingual Graph-based Semantic Model

2.1

Graph Constructing

graph built from a very large corpus (such as millions of sentences) without any pruning. In addition, not all of the nodes
are useful for word representations, because some nodes do
not have any connection with input contextual words (sparsity). These nodes actually have not a direct impact overclique extraction. For a word n and its contextual words{n1 , n2 , ..., ni , ...nt } as input3 , only the co-occurrence nodes
nij of each ni (including n itself) are indeed useful and thenactually extracted. The set of nodes {nij } with their weighted
edges form an extracted graph Gextracted for further cliques
extraction. So the number of nodes in the extracted graphGextracted , |Nextracted |, is given by,|Nextracted | =Formally, words are considered as nodes (vertices) and cooccurrence relationships of words are considered as edges ofgraph. An edge-weighted graph derived from a bilingual corpus is formalized as, G = {W, E}, where W is node set andE is edge set which is weighted by co-occurrence relationshipintroduced as follows.
For a given bilingual parallel corpus, each source sentence SF = (wf1 , wf2 , ..., wfk ) and its corresponding target sentence SE = (we1 , we2 , ..., wel ) are combined together to construct a Bilingual Sentence (BS) =(wf1 , wf2 , ..., wfk , we1 , we2 , ..., wel ).
For words (either
source or target word) wi and wj , if they are in the same BS,
they are called co-occurrences for each other and marked as
ni and nj in the graph G. As node ni in graph is always referred to word wi , we will not distinguish them throughout this paper. The Edge Weight (EW ) connecting nodes ni
and nj is defined by a PMI score,

8i,j{nij } .

In practice, the |Nextracted | is much smaller than |V | (vocabulary size of bilingual corpus). For a typical corpus(IWSLT in Section 5.1), |Nextracted | is around 371.2 on average and |V | is 162.3K. Thus the clique extraction in practiceis quite efficient as it works over a quite small sized graph.
Clique extraction may follow a standard routine in [Luceand Perry, 1949]. As the clique in this paper is to represent
a fine grained bilingual sense of a word, it is called Bilingual
Contexonym Clique (BCC). Similar but more fine grained
than synset (a small group of synonyms labeled as concept)
defined in WordNet [Miller et al., 1990], the BCC now is theminimal unit for bilingual meaning representation.
Taking the word work e and readers e as an example (without context), two groups of BCCs (in alphabetical order) are
given in Table 1. It shows that different word senses can be
distinguished by BCCs. The BCCs containing employees e,
travail f (work) and unemployed e may indicate the meaningof job, while the BCCs containing readers e may indicate themeaning of literature.

Co(ni , nj ),(1)
f r(ni ) ⇥ f r(nj )where Co(ni , nj ) is the co-occurrence counting of ni and nj
and f r(n) stands for how many times n occurs in corpus.
For nearly all languages, stop words such as of, a, the inEnglish or de, une, la in French have a wide distribution, andthis results in that most nodes in the graph are unnecessarily
connected with these stop word nodes. A filter with an EW
threshold is thus set to prune these poorly connected edges.
will be tuned using development data, to let the resulted
graph keep the more useful edges based on the empirical result of each individual task.
EW =

2.2

[

Words

work eContext-Dependent Clique Extraction

A clique defined in graph theory means a maximum, complete sub-graph [Luce and Perry, 1949]. For a subset of nodes
with edges in the whole graph, if every two nodes in the subset are connected to each other, this subset of nodes form aclique. Suppose that both N1 and N2 are subsets of N in G.
If N1 ⇢ N2 , then N1 cannot be a clique (maximality).
Graph problems, such as finding all cliques from a graph,
are mostly associated with high computational complexity(Clique Problem). The Clique Problem related to our model
has been shown NP-complete [Karp, 1972], and it is time consuming or even impossible to find the cliques from the whole

readers e

BCCs{employees e, travail f (work), unemployed e, work e }{heures f (hours), travaillent f (to work,
third-person plural form), travailler f(work), week e, work e }{readers e, work e }...
{informations f(information),journaux f (newspapers), online e, readers e}{journaux f (newspapers), lire f (read),newspaper e, presse f (press), readers e,
reading e}{readers e, work e}...

Table 1: BCC examples. Suffixes ‘ e’ and ‘ f ’ are used toindicate English or French, respectively. The English words
in parentheses are corresponding translations.
3
For SMT task, the words in aligned phrases are used as context,
please refer to Section 3 for details.

2951

Note that edges pruning is static, and nodes selection
is dynamic depending on input word sequence. The proposed node selection and clique extraction follow [Plouxand Ji, 2003], except that we use bilingual co-occurrencegraph rather than monolingual synonym or hypo(hypero)nym
graphs. BCCs can be regarded as loose synsets, as only
strongly related words can be nodes in a clique that possess full connections, and different senses will naturally resultin roughly different cliques from our empirical observations,though noise or improper connections also exist at the sametime. To obtain concise semantic vector representations, adimension reduction will be performed.

2.3

Semantic Spatial Representation

Correspondence Analysis (CA) [Benzcri, 1973] assesses theextent of matching between two variables. It determines thefirst n factors of a system of orthogonal axes that capture thegreatest amount of variance in the matrix. The first axis (orfactor) captures the largest variations, the second axis captures the second largest, and so on.
To project words onto lower dimensional semantic space,
CA is conducted over the clique-word matrix constructedfrom the relation between BCCs and words. An initial correspondence matrix M = {mij } is built, where mij = 1 if
the BCC in row i contains the word in column j, and 0 if not.
Normalized correspondence matrix P = {pij } is directly derived from M , where pij = mij /NM , and NM is grand totalof all the elements in M . Let the row and column marginal totals of P be r and c which are the vectors of row and column
masses, respectively, and Dr and Dc be the diagonal matricesof row and column masses. Coordinates of the row and column profiles with respect to principal axes are computed byusing the Singular Value Decomposition (SVD) as follows.
Principal coordinates of rows F and columns G:
1

Figure 1: Spatial map of work e (without context) as input(The green lines indicate the borderline of clusters).

1

F = Dr 2 U ⌃, G = Dc 2 V ⌃,where U , V and ⌃ (diagonal matrix of singular values in descending order) are from the matrix of standardized residuals
S and the SVD,
1

1

S = U ⌃V ⇤ = Dr 2 (P rc⇤ )Dc 2 ,where ⇤ denotes conjugate transpose and U ⇤ U = V ⇤ V = I.
By above processes, CA projects BCCs (F) and words(G) onto semantic geometric coordinates as vectors. Inertia
2/NM is to measure semantic variations of principal axes
for F and G:
2

/NM =

X X (pijijlire_fri c j ) 2.
ri c jFollowing standard setting of CA [Benzcri, 1973], top
principal dimensions (axes) of vectors are chosen for word
and clique representation. Bilingual Graph-based Semantic
Model (BGSM) is consequently constructed from these principal dimensions. In short, a word with its context are used
as input of BGSM, and vectors of the word4 and its bilingual
co-occurrences are output.
4

Figure 2: Spatial map of work e with context readers e as
input.

In fact both BCCs and words can be represented as vectors.

2952

To visualize the results, top two dimensions are chosen andillustrated into spatial map. We illustrate the spatial relationship between BCCs and words in the same map, instead of thespatial map of words only. BCCs are represented by points
and words by regions. Label of word is approximately (toavoid overlapping) placed at the barycentre of region (grayline) delineated by a set of BCCs that contain the word. BCCs
are clustered [Ward Jr, 1963] into three groups (green line).
We only present several typical words, and the words too far
from most of other words are discarded.
Figures 1 and 2 illustrate the spatial representation of all
the co-occurrence words when work e is input without context and with readers e as context using BGSM, respectively.
The following observations can be obtained from them:
1) Several good translation pairs are shown, such as(work e, travailler f ), (readers e, lecteurs f ), (reading e,lire f ) and (book e, livre f ).
2) For work e as input, the word work e itself is placed at
the center and the other words are mainly clustered into threesemantic groups: employment (the right), evaluation of job(the upper left), and literary works5 (the bottom). However,
we cannot determine which sense of work e belongs to.
3) For work e+readers e as input, we can identify that thesense of work e by the words close to both of work e andreaders e, such as book e, print e and paper f.

3(PF , PE ), we consider two strategies for selecting the context words:
Strategy-A: only the source words in PF are used as thecontextual words, {wf1 , wf2 , ..., wfk }.
Strategy-B: both the source words in PF and target words
in all the aligned PE are used as its contextual words,{wf1 , wf2 , ..., wfk , we1 , we2 , ..., wel }.
Word wf i (or wei ) is represented as vector Vwfi (or Vwej )(The co-occurrence word wco can also be represented as vector Vwco ). Note that all the source and target words for thesame source phrase PF are represented as vectors in the same
geometric coordinates. Some words may not belong to any
BCC (partially because the graph is pruned). These unknown
words would be represented as a default vector.

3.2

Because the numbers of word alignments in phrase pairs aredifferent, Normalized Euclidean Distance (N ED) is adopted
to measure the distance between source and target phrases
incorporated with IBM word-alignment model:

N ED(PF , PE ) =Phrase Translation Probability Estimation

Bilingual Phrase Semantic Representation

Sim(PEi |PF ) =The phrase table of phrase-based SMT model can be simply
formalized as6 ,(PF , PE , scores, word-alignment),sP

2align(i,j) ED (Vwfi , Vwej )Pi,j align(wfi , wej ),(3)

where ED(Vwfi , Vwej ) stands for Euclidean Distance between word vectors Vwfi and Vwej , align(i,j) is from thePword-alignment model in Eq. (2), and i,j align(wfi , wej )is total number of word alignments between PF and PE .
As there are usually multiple PE that are aligned to PF ,NPE is noted as the number of aligned PE . To let thesimilarity score Sim(PE |PF ) be a probability distribution,
Sim(PE |PF ) =1 if NPE = 1; otherwise, Sim(PEi |PF ) is
given by,

BGSM represents words as vectors dynamically on various
geometric coordinates according to contextual words. For
each word in a source phrase of SMT, its contextual words are
fixed, so all the translation candidate target words can be represented as vectors in the same geometric coordinates. This
makes it possible to apply BGSM into phrase-based SMT for
selecting translated phrase candidates.

3.1

Semantic Similarity MeasurementP

j N ED(NPE

2(PF , PEj ) N ED2 (PF , PEi )P.
1) ⇥ j N ED2 (PF , PEj )(4)

Using the same pipeline, Sim(PF |PE ) can also be calculated. Both Sim(PE |PF ) and Sim(PF |PE ) can be added
as features for SMT decoding.

(2)

where PF (wf1 , wf2 , ..., wfk ) and PE (we1 , we2 , ..., wel ) aresource and its aligned target phrases, respectively, and scoresindicate various feature scores including direct translation
probability, lexical weighting and phrase penalty. The phrase
length is limited to 7, which is the default setting for phrasebased SMT.
BGSM represents words in phrase table as six-dimension
vectors. It should be noted that the clique extraction depends
on contextual words, and CA then projects clique-word matrix onto corresponding semantic geometric coordinates accordingly. So the same contextual words should be used for
all the words in PF and all its aligned PE , in order to represent them on the same geometric coordinates. For each word
wfi or wej (where 1  i  k, 1  j  l), in phrase pair4

Bilingual Phrase Generation (BPG)

A few phrases are outside corpus but share the similar meaning as those inside the corpus. Take the source French phrase
la bonne réponse as example, the corresponding aligned target English phrase the right answer is in the corpus andphrase table. The other phrases, such as the correct answer or
the right response, may not be in the corpus or phrase table,
however, they are also good candidates for translation.
Since the BGSM can be used to represent words as vectors
and measure their similarities by computing vector distance,
it is possible to generate new (maybe better) phrases with similar meaning as the original one for phrase table.

4.1

5

Part of borderline of this cluster is overlapped by some words.
The alignment is from standard IBM alignment model [Berger
et al., 1994].

Phrase Pair Generation

As mentioned in Section 3, for each word w (source or target), both of the source words in PF and target words are

6

2953

used as its contextual words (Strategy-B). Word w and its cooccurrence words {wco } are represented as vectors.
For an aligned word pair (wfi , wej ), we find a new translation replacement we0 j in {wco } to help generate new phrases.
For either source phrase PF or target phrase PE , each word
inside will be tentatively replaced by the nearest word in its
corresponding co-occurrence according to word vector distance (here, Euclidean distance is actually adopted.). However, only one word replacement with the minimal distancefor either phrase will be chosen and implemented to generate
two new phrases PE0 and PF0 , respectively7 .
Sim(PE0 |PF ) and Sim(PF0 |PE ) can be calculated using
Eqs. (3) and (4). They are also being the phrase transition
probabilities for the generated (PF , PE0 ) and (PF0 , PE ), respectively, as no such probabilities exist in the original phrase
table. The updated lexical weighting lex(PE0 |PF ) and inverse
lexical weighting lex(PF0 |PE ) are computed by IBM model[Berger et al., 1994].
The generated phrases are filled-up [Bisazza et al., 2011]
into original phrase table. That is, a penalty score is added
as feature: for original phrase pairs, the penalty is set as one;for the generated ones the penalty is set as natural logarithm
base e (= 2.71828...). All scores weights in phrase table will
be further tuned using MERT [Och, 2003].

4.2

Corpustraining
dev
test

5.2

N ED(PF , PE0 ),(5)
N ED(PF , PE )is used to measure the usefulness of generated phrase pairs.
A threshold " is set up to keep the most useful generated
phrase pair only. Namely, for a source phrase PF , only thePE0 whose DR(PE0 , PE ) smaller than " are selected as thegenerated phrase pair (PF , PE0 ). Using the same pipeline, thesize of generated source candidate phrases is also tuned. For
SMT task, the threshold is tuned according to SMT performance on development data.
DR(PE0 , PE ) =5.1

Setting up

NIST
2.4M
1.6K
1.3K

Baseline Systems

The same basic settings for the IWSLT-2014, NTCIR-9 and
NIST08 translation baseline systems are complied. The standard Moses phrase-based SMT system is applied [Koehn etal., 2007] together with GIZA++ [Och and Ney, 2004] foralignment, SRILM [Stolcke, 2002] for language modelingand MERT for tuning (we run MERT three times and record
the average BLEU score on test data). The paired bootstrap
re-sampling test9 is performed. Significant tests are done foreach round of test. Marks at the right of BLEU scores indicate
whether our proposed methods are significantly better/worse
than the corresponding baseline (‘++/’: significantly better/worse at significance level ↵ = 0.01; ‘+/ ’: ↵ = 0.05).
All the experiments in this paper are conducted on the samemachine with 2.70GHz CPU.
As the proposed BGSM is a bilingual word embedding
method and applied to SMT as new features, we only compare with most related bilingual word embedding or generation methods for SMT. For phrase pair translation probability
estimation task, two typical neural network based bilingual
embedding methods, Continuous Space Translation Model(CSTM10 ) [Schwenk, 2012] and [Zou et al., 2013], are select as baselines. The embedding of each method is added asfeatures to the phrase-based SMT baseline, with all the othersetting the same. For bilingual phrase generation methods,
CSTM is used as the same way as [Schwenk, 2012]’s generation method. We also compare with [Saluja et al., 2014],
which uses graph method for translation candidate generation11 .

Using the phrase generation approach, a lot of new phrase
pairs can be generated. We need to select the most reasonable ones inside them. The Distance Ratio (DR) of normalized distance in Eq. (4) between the generated phrase pair
(PF , PE0 ) and the original phrase pair (PF , PE ),Experiments

NCTIR
1.0M
2.0K
2.0K

Table 2: Sentence statistics on parallel corpora.

Prase-table Size Tuning

5

IWSLT
186.8K
0.9K
1.6K

5.3

Results and Analysis

Only the best performed results (for both the baselines andproposed methods) on development data are chosen to beevaluated on test data and shown. The parameters for BGSM
are set as follows: 1) Vector dimensions are 6; 2) Thresholdfor edge weight pruning EW in Eq. (1) is 3 ⇥ 10 4 ; 3)Threshold " for phrase table tuning DR in Eq. (5) is 1.31.

To evaluate BGSM in various language and domain SMT systems, Corpora of IWSLT-2014 French to English (EN) [Cettolo et al., 2012], NTCIR-9 Chinese to English [Goto et al.,2011] and NISTOpenMT088 are chosen.

of [Galley et al., 2008] that were used by [Zou et al., 2013]. NIST
Eval 2006 is used as development data and NIST Eval 2008 as testdata.
9
The implementation of our system follows http://www.ark.cs.
cmu.edu/MT
10
The recommended settings of CSTM [Schwenk, 2012] are followed. That is, phrase length limit is set as 7, shared 320-dimension
projection layer for each word (that is 2240 for 7 words), 768dimension projection layer, 512-dimension hidden layer. The dimensions of input/output layers are the same as the size of vocabularies of source/target words.
11
The implementation and settings in [Saluja et al., 2014] are followed except morphological generation.

7

Two or more words can be replaced, but it may lead to serious
sense bias, and the experiments also show that replacing more than
two words does not perform well.
8
Zou et al. [2013] only released their word vectors rather thantheir code (http://ai.stanford.edu/⇠wzou/mt/), so we have to conduct experiments on NIST08 Chinese-English translation task as
they did for comparison. The training data consists of part of NIST
OpenMT06, United Nations Parallel Text (1993-2007) and corpora

2954

Phrase Pair Translation Probability Estimation Results

Methods
CSTM
BGSM-A
BGSM-B

Table 3 indicates that BGSM can improve SMT performance
up to +0.85 BLEU, and outperforms CSTM or Zou’s methods up to +0.67 BLEU. Besides, the Strategy-B performs better than Strategy-A, which attributes to more contextual (bothtarget and source) information used for the former while only
source for the latter.

Baseline+Zou+CSTM+BGSM-A+BGSM-B

IWSLT
31.80
N/A
32.19
32.32+
32.61++

NTCIR
32.19
N/A
32.37
32.56
33.04++

6

NIST
30.12
30.36
30.25
30.38
30.44+‘Baseline + BPG’ indicates adding the generated phrase pairs
to the original phrase table. ‘BPG + BGSM’ indicates adding
the generated phrase pairs, as well as replacing the translation
probabilities in original phrase table with the Sim(PE |PF )and Sim(PF |PE ) calculated by BGSM (Section 4.1).

IWSLT

NTCIR

Phrase Pairs
9.8M
23.1M
31.5M
25.6M
25.6M
71.8M
297.8M
341.3M
312.6M
312.6M

Conclusion

Existing word embedding methods usually only consider simple context such as document or sliding window for word
relationship building and later word representation. Instead,this paper focuses on sense representation in terms of bilingual background. Using a graph constructed from a bilingual corpus, Bilingual Contexonym Clique (BCC) is proposed for better sense characterization. A BCC-word matrix is then built from dynamic sense-sensitive context in
the graph and correspondence analysis is to summarize thematrix into lower dimensions as Bilingual Graph Semantic
Model (BGSM).
BGSM word embedding is applied to phrase pair translation probability estimation and generation. The experimental
results show that the proposed model can enhance phrasebased SMT decoding and achieve a significant improvementwith high computational efficiency. It also outperforms theexisting related word embedding methods for SMT.

Bilingual Phrase Pair Generation Results

Methods
Baseline+CSTM+Saluja+BPG+BPG+BGSM
Baseline+CSTM+Saluja+BPG+BPG+BGSM

Calculating Time17.1 Minutes
8.9 Minutes
15.6 Minutes

Table 5: CPU time on IWSLT-2014.

Table 3: Phrase pair translation probability estimation results(BLEU).

CorporaTraining Time59.5 Hours
1.1 Hours
1.1 Hours

References

BLEU
31.80
32.19
32.35
32.37
33.13++
32.19
32.42
32.68
32.54+
33.47++[Bengio et al., 2003] Y. Bengio, R. Ducharme, P. Vincent,and C. Janvin. A neural probabilistic language model.
Journal of Machine Learning Research, 2003.
[Benzcri, 1973] J.-P. Benzcri. L’analyse des correspondances. In L’Analyse des Donnes, 1973.
[Berger et al., 1994] A. L. Berger, P. F. Brown, S. A.
D. Pietra, V. J. D. Pietra, J. R. Gillett, J. D. Lafferty, R. L.
Mercer, H. Printz, and L. Ureš. The Candide system formachine translation. In HLT, 1994.
[Bisazza et al., 2011] A. Bisazza, N. Ruiz, M. Federico, and
B. Kessler. Fill-up versus interpolation methods forphrase-based smt adaptation. In IWSLT, 2011.
[Cettolo et al., 2012] M. Cettolo, C. Girardi, and M. Federico. Wit3 : Web inventory of transcribed and translated
talks. In EAMT, 2012.
[Faruqui and Dyer, 2014] M. Faruqui and C. Dyer. Improving vector space word representations using multilingual
correlation. In EACL, 2014.
[Galley et al., 2008] M Galley, P. Chang, D. Cer, J. Finkel,
and C. Manning. NIST open machine translation 2008
evaluation: Stanford university’s system description. In
NISTOpenMT, 2008.
[Gao et al., 2014] J. Gao, X. He, W.-T. Yih, and L. Deng.
Learning continuous phrase representations for translation
modeling. In ACL, 2014.
[Goto et al., 2011] I. Goto, B. Lu, K. P. Chow, E. Sumita, andB. K. Tsou. Overview of the patent machine translation

Table 4: Bilingual Phrase Generation (BPG) results.
The results in Table 4 indicate that the proposed BPG and
BGSM methods can work well together and enhance SMT
performance significantly up to +1.33 BLEU. They also outperform state-of-the-art method up to +0.79 BLEU.
Efficiency Comparison
We compare the efficiencies for model training and computing the probability scores of phrases pairs using CSTM and
BGSM. Two thousand phrase pairs are randomly selectedfrom the whole IWSLT-2014 FR-EN corpus. The CPU timeof training models (the whole corpus) and calculating theirprobability scores (2,000 sentences) is shown in Table 5.
The results in Table 5 demonstrate that BGSM is much
more efficient than CSTM, especially for training, the former
can be more than 50 times as fast as the later.

2955

[Shi et al., 2015] T. Shi, Z. Liu, Y. Liu, and M. Sun.
Learning cross-lingual word embeddings via matrix cofactorization. In ACL, 2015.
[Stolcke, 2002] A. Stolcke. SRILM-an extensible language
modeling toolkit. In ICSLP, 2002.
[Vulić and Moens, 2015] I. Vulić and M.-F. Moens. Bilingual word embeddings from non-parallel documentaligned data applied to bilingual lexicon induction. In
ACL, 2015.
[Wang et al., 2013] R. Wang, M. Utiyama, I. Goto,
E. Sumita, H. Zhao, and B.-L. Lu. Converting continuousspace language models into n-gram language models forstatistical machine translation. In EMNLP, 2013.
[Wang et al., 2014] R. Wang, H. Zhao, B.-L. Lu,
M. Utiyama, and E. Sumita. Neural network based
bilingual language model growing for statistical machinetranslation. In EMNLP, 2014.
[Wang et al., 2015] R. Wang, H. Zhao, B.-L. Lu,
M. Utiyama, and E. Sumita. Bilingual continuousspace language model growing for statistical machinetranslation. IEEE/ACM TASLP, 2015.
[Wang et al., 2016a] P.-L. Wang, Y. Qian, H. Zhao, and
F. Soong. Learning distributed word representations forbidirectional LSTM recurrent neural network. In NAACL,
2016.
[Wang et al., 2016b] R. Wang, M. Utiyama, I. Goto,
E. Sumita, H. Zhao, and B.-L. Lu. Converting continuousspace language models into n-gram language models withefficient bilingual pruning for statistical machine translation. ACM TALLIP, 2016.
[Ward Jr, 1963] J. H. Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 1963.
[Zhang and Zhao, 2013] J. Zhang and H. Zhao. Improving
function word alignment with frequency and syntactic information. In IJCAI, 2013.
[Zhang et al., 2014] J. Zhang, S. Liu, M. Li, M. Zhou, andC. Zong. Bilingually-constrained phrase embeddings formachine translation. In ACL, 2014.
[Zhao and Kit, 2011] H. Zhao and C. Kit. Integrating unsupervised and supervised word segmentation: The role ofgoodness measures. Information Sciences, 2011.
[Zhao et al., 2010] H. Zhao, C.-N. Huang, M. Li, and B.-L.
Lu. A unified character-based tagging framework for Chinese word segmentation. ACM TALLIP, 2010.
[Zhao et al., 2013] H. Zhao, X. Zhang, and C. Kit. Integrative semantic dependency parsing via efficient large-scalefeature selection. Journal of Artificial Intelligence Research, 2013.
[Zou et al., 2013] W. Y. Zou, R. Socher, D. Cer, and C. D.
Manning. Bilingual word embeddings for phrase-based
machine translation. In EMNLP, 2013.

task at the NTCIR-9 workshop. In NTCIR-9 Workshop,
2011.
[Ji and Ploux, 2003] H. Ji and S. Ploux. Lexical knowledge
representation with contexonyms. In MT Summit, 2003.
[Karp, 1972] R. M. Karp. Reducibility among combinatorial problems. In Complexity of Computer Computations.
Springer, 1972.
[Koehn et al., 2007] P. Koehn, H. Hoang, A. Birch,
C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan,
W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. Moses: Open source toolkit for
statistical machine translation. In ACL, June 2007.
[Lauly et al., 2014] S. Lauly, H. Larochelle, M. Khapra,
B. Ravindran, V. C. Raykar, and A. Saha. An autoencoder
approach to learning bilingual word representations. In
NIPS, 2014.
[Levy and Goldberg, 2014] O. Levy and Y. Goldberg. Linguistic regularities in sparse and explicit word representations. In CONLL, 2014.
[Lu et al., 2015] A. Lu, W. Wang, M. Bansal, K. Gimpel, and
K. Livescu. Deep multilingual correlation for improved
word embeddings. In NAACL, 2015.
[Luce and Perry, 1949] R. D. Luce and A. D. Perry. A
method of matrix analysis of group structure. Psychometrika, 1949.
[Mikolov et al., 2013a] T. Mikolov, Q. Le, and I. Sutskever.
Exploiting similarities among languages for machinetranslation. arXiv, 2013.
[Mikolov et al., 2013b] T. Mikolov, I. Sutskever, K. Chen,
G. S. Corrado, and J. Dean. Distributed representationsof words and phrases and their compositionality. In NIPS,
2013.
[Miller et al., 1990] G. A. Miller, R. Beckwith, C. Fellbaum,
D. Gross, and K. J. Miller. Introduction to wordnet: An
on-line lexical database. International journal of lexicography, 1990.
[Och and Ney, 2004] F. J. Och and H. Ney. The alignment
template approach to statistical machine translation. Computational linguistics, 2004.
[Och, 2003] F. J. Och. Minimum error rate training in statistical machine translation. In ACL, 2003.
[Pennington et al., 2014] J. Pennington, R. Socher, andC. Manning. GloVe: Global vectors for word representation. In EMNLP, 2014.
[Ploux and Ji, 2003] S. Ploux and H. Ji. A model for matching semantic maps between languages (french/english, english/french). Computational Linguistics, 2003.
[Saluja et al., 2014] A. Saluja, H. Hassan, K. Toutanova, andC. Quirk. Graph-based semi-supervised learning of translation models from monolingual data. In ACL, 2014.
[Schwenk, 2012] H. Schwenk. Continuous space translation
models for phrase-based statistical machine translation. In
COLING, 2012.

2956