言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Bilingual Word Embedding:
A Survey from Supervised to Unsupervised
Haipeng Sun1∗ , Rui Wang2 , Kehai Chen2 , Masao Utiyama2 , Eiichiro Sumita2 , and Tiejun Zhao1
1 Harbin Institute of Technology
2 National Institute of Information and Communications Technology
hpsun@hit-mtlab.net and tjzhao@hit.edu.cn{wangrui, khchen, mutiyama, eiichiro.sumita}@nict.go.jp

1 Introduction
Word embedding is the basis of neural network based
nature language processing (NLP) tasks. As monolingual word embedding has been widely used in many languages (Mikolov et al., 2013a), similarities among different languages were exploited by Mikolov et al. (2013b).
The similarity between words in two or more languages
can be represented by the correlation between word embedding. Bilingual word embedding (BWE) is to mappthe relationship between different monolingual word embeddings. Recently, several works show that BWE can belearn without bilingual supervision (Zhang et al., 2017a;
Artetxe et al., 2018b). In this paper, we survey the
BWE from supervised methods to unsupervised methods: supervised BWE methods that need a large training dictionary, semi-supervised BWE methods that needa small seed dictionary or only parallel corpus, unsupervised BWE methods.

2 Supervised BWE
The supervised BWE proposed by Mikolov et al.
(2013b) exploits similarities between the source language
and the target language by a linear transformation matrix. Supervised BWE methods usually need a large dictionary, and the linear matrix would be trained between
language pairs in this dictionary. A set of word pairs{X, Z} and their word vector representations {xi , zi }ni=1
are given, where n is the size of the word pairs, d is thedimension of word embedding. The training objective∗ This work was conducted while Haipeng was an intern in NICT.

function of BWE is as follows:
∑
2
2
min
∥W xi − zi ∥ = min ∥W X − Z∥ ,
W

W

i

(1)

where W is the transformation matrix to be learned suchthat W X approximates Z. X, Z are the word embedding matrices of size d × n, respectively. Dinu et al.
(2015) used a L2-regularized least-squares error objective instead of previous objective function:2

min ∥W X − Z∥ + λ ∥W ∥ .
W(2)They also used a globally corrected neighbour retrieval
method to mitigate the hubness problem. Xing et al.
(2015) improved the performance of bilingual word embedding by enforcing word embedding normalized andan orthogonality constraint on W to preserve the lengthnormalization. Then the bilingual word embedding problem has become the Orthogonal Procrustes problem andW = U V T in the equation (2) can be acquired by the singular value decompositionZX T = U ΣV T . After wordembedding could be normalized, the equation (2) that isequivalent to maximize the sum of cosine similarities canbe reformulated as follows:∑maxcos(W xi , zi ).
(3)
W

i

Artetxe et al. (2016) considered the monolingual invariance through orthogonality to prevent the degradation in monolingual tasks. Length normalization andmean centering could be taken into consideration, so theequation (2) is also equivalent to maximizing the sum of
dimension-wise covariance as∑max
cov(W xi , zi ).
(4)W― 221 ―i

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Smith et al. (2017) showed that the linear transformation should be orthogonal to preserve self-consistent.
The inverted softmax retrieval replaced nearest neighbors

data. The objective was optimized as∑ ∑minLl (w, h; θl ) + λΩ(θe , θf ),θ e ,θ fmethod to achieve better translation pairs. Artetxe et al.

l∈{e,f } w,h∈Dl(6)

a multi-step framework of linear transformations such aswhere Ω(·) is cross-lingual regularization term that wasused to constrain monolingual models over the context hnormalization, whitening, orthogonal transformation, re-and target word w training pairs in the dataset D during

weighting, and dimensionality reduction behavior.

jointly training.

(2018a) generalized and improved BWE mappings with

Compared with previous methods that learn transfor-

Wang et al. used bilingual maximum complete sub-mation matrix from source language to target language,graphs (cliques), which play the role of a minimal unit for

Faruqui and Dyer (2014) mapped the word embedding

bilingual sense representation (Wang et al., 2016, 2018).

to a shared space where their similarity was maximized

Cliques are dynamically extracted according to the con-through canonical correlation analysis that is a statisticaltextual information. Consequently, correspondence anal-

analysis method for measuring the linear correlation between two multidimensional variables as

ysis, principal component analyses, and neural networks
are used to summarize the clique-word matrix into lower
dimensions to build the embedding model.

W, G = CCA(X, Z) = arg max ρ(W X, GZ)
W,G= arg max ρ( √W,GE[(W X)(GZ)]
E[(W X)2 ]E[(GZ)2 ](5)

3.2 Small Lexicon as Supervision),

where ρ(·) means the correlation between the projected
vectors, and W, G are projection matrices which map
X, Z to the shared space in the equation (5). Lu et al.
(2015) extended previous work with deep canonical cor-

Artetxe et al. (2017) proposed self-learning framework
to learn BWE with almost 25 word dictionary. During
the self-learning, the squared Euclidean distance could
be minimized:

relation analysis to learn non-linear mapping.

min
W

∑∑
i

2

Dij ∥W xi − zj ∥ ,(7)jwhere D is a binary matrix, and Dij = 1 if the ith source

3 Semi-supervised BWE

word is aligned with the jth target word in the dictio-

As BWE was developing, some researchers found thatnary. The source and target embedding would be thethe supervision is not always necessary. Therefore, somelength normalized and mean centered, and W would beworks tried to reduce the supervision in BWE. Specifi-

constrained to be an orthogonal matrix. So the equation

cally, the aligned sentence pairs and small lexicon seeds
were used as supervision of BWE.

(7) is equivalent to maximizing the dot product as
max T r(W T ZDT X T ),W

where T r(·) means the sum of all elements in the main

3.1 Aligned Sentences as Supervision
BilBOWA(Bilingual

Bag-of-Words(8)

diagonal of matrix, W = U V T in the equation (8)without

Alignments) (Gouws et al., 2015) is a simple andcan be acquired by the singular value decompositionZDT X T = U ΣV T .

computationally-efficient model for learning bilingual
distributed representations of words which can scale
to large monolingual datasets and does not require
word-aligned parallel training data.

4

Instead it trains

directly on monolingual data and extracts a bilingualsignal from a smaller set of raw-text sentence-aligned

Unsupervised BWE
However, the lack of large word pair dictionary posesa major practical problem for many language pairs. The
unsupervised BWE has attracted much attention. Zhang― 222 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

et al. (2017a) matched the distributions of the trans-

the similarity between mapped source words and target

formed source language embedding x ∼ Px and targetwords, aslanguage embedding z ∼ Pz via generative adversarial

CSLS(W x, z) = 2cos(W x, z) − r(W x) − r(z).

network training. Two sets of monolingual word embedding {xi }ni=1 and {zi }mi=1 with dimensionality d arer(W x) =trained separately on two languages. The discriminator
D loss function is given by

1K

1r(z) =
K

LD = −Ez∼Pz logD(z) − Ex∼Px log(1 − D(W x)).
(9)
The generator loss function is as follows:

(15)

∑

cos(W x, z).

(16)cos(W x, z).

(17)z∈N (W x)∑

W x∈N (z)where z ∈ N (W x) means the K nearest neighborhoodof the mapped source embedding W x and W x ∈ N (z)means the K nearest neighborhood of the target embed-

LG = −Ex∼Px logD(W x)ding z. Dou et al. (2018) added variational autoencoder= −Ex∼Px logD1 (W x) − Ez∼Pz logD2 (W z)

into the generative adversarial network in order to learn= −Ex∼Px logD(W x) − λEx∼Px cos(x, W T W x),

latent variables that can capture semantic meaning of

T

(10)

words. Therefore the discriminator D loss function is
LD = Evz ∼qGz (v|z) [logD(vz )]where W ∈Rd×d is the transformation matrix to belearned such that W x approximates z, its transpose W T
which can transform the target language z back to the+ Evx ∼qGx (v|x) [log(1 − D(vx ))].

(18)

The objective function of generator Gx is to minimize

source language x. D1 , D2 are two separate discrimina-

LGx = Evx ∼qGx (v|x) [logPx′ (x|vx )]tors. λ is a hyperparameter. Zhang et al. (2017b) used the− Evx ∼qGx (v|x) [logD(vx )],Wasserstein distance instead of cross-entropy in the loss

where qGx (v|x) and qGz (v|z) is the posterior distribu-function. The Wasserstein distance istion of the latent variables, Px′ (x|vx ) is the reconstruc-

W(PW x , Pz )1
sup Ez∼Pz [f (z)] − Ex∼Px [f (W x)],=K ∥f ∥L ≤Ktion distribution. The generator Gz function is similar.
(11)

Sinkhorn distances and back-translation losses were usedin the (Xu et al., 2018). The objective function isL(W, G) = dsh (W ) + dsh (G) + βdbt (W, G),∑
dbt (W, G) =1 − cos(xi , G(W (xi )))+

where f that is a K-Lipschitz function can be approximated with a neural network. The objective function forthe discriminator D can be formulated as
LD = Ex∼Px [fD (W x)] − Ez∼Pz [fD (z)].

(19)

i

∑

(12)

1 − cos(zi , W (G(zi ))),

(20)

(21)

j

where G is the transformation matrix to be learned such
The generator loss function is given by
LD = −Ex∼Px [fD (W x)].

that Gz approximates x, dsh (W ) is the Sinkhorn distance(13)

between PW x , Pz , dsh (G) is the Sinkhorn distance between PGz , and Px , β is the hyperparameter.

Conneau et al. (2018) used the same discriminator loss

Artetxe et al. (2018b) proposed a self-learning method

function as the equation (9). For the the generator G, thelike Artetxe et al. (2017) without any word dictionary.

objective is to minimize

The objective function is similar as the equation (7),LG = −Ex∼Px logD(W x) − Ez∼Pz log(1 − D(z)).
(14)

which considered two directional transformation as∑∑max
Dij (W x · Gz)
(22)
W,G

They also built a synthetic parallel vocabulary to refine
the mapping W , and used cross-domain similarity local
scaling (CSLS) instead of nearest neighbors to measure

W

=

V

T

i

j

and G

=

U T in the equation (22)can be acquired by the singular value decompositionZDT X T = U ΣV T .

― 223 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

5 Conclusion and Future Work

Stephan Gouws, Yoshua Bengio, and Greg Corrado.
2015. Bilbowa: Fast bilingual distributed representa-

In this paper, we surveyed the developing of bilingual

tions without word alignments. In ICML, Lille, France.

word embedding methods, from supervised to unsupervised. We focused on the algorithms and methods instead of the corresponding performances. In the future,

Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, andwe will present the performances of different methods.

for improved word embeddings. In NAACL, Denver,In addition, we will survey the applications of bilingual

Colorado.

word embedding in NLP, especially unsupervised neuralmachine translation.

Karen Livescu. 2015. Deep multilingual correlation

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.

参考文献

Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b.
Exploiting similarities among languages for machine

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016.

translation. CoRR, abs/1309.4168.

Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In

Samuel L. Smith, David H.P. Turban, Steven Hamblin,

EMNLP, Austin, Texas.

and Nils Y. Hammerla. 2017. Offline bilingual word
vectors, orthogonal transformations and the inverted

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017.

softmax. In ICLR, Toulon, France.

Learning bilingual word embeddings with (almost) nobilingual data. In ACL, Vancouver, Canada.

Rui Wang, Hai Zhao, Sabine Ploux, Bao-Liang Lu, andMasao Utiyama. 2016. A bilingual graph-based se-

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a.

mantic model for statistical machine translation. In IJ-

Generalizing and improving bilingual word embedding
mappings with a multi-step framework of linear transformations. In AAAI, New Orleans, Louisiana.

CAI, New York, USA.
Rui Wang, Hai Zhao, Sabine Ploux, Bao-Liang Lu,
Masao Utiyama, and Eiichiro Sumita. 2018. Graph-

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b.

based bilingual word embedding for statistical ma-

A robust self-learning method for fully unsupervised
cross-lingual mappings of word embeddings. In ACL,
Melbourne, Australia.

chine translation. TALLIP, 17(4).
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015.
Normalized word embedding and orthogonal trans-

Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word

form for bilingual word translation. In NAACL, Denver, Colorado.

translation without parallel data. In ICLR, Vancouver,
Canada.

Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu.
2018. Unsupervised cross-lingual transfer of word em-

Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni.

bedding spaces. In EMNLP, Brussels, Belgium.

2015. Improving zero-shot learning by mitigating the
hubness problem. In ICLR, San Diego, California.

Meng Zhang, Yang Liu, Huanbo Luan, and Maosong
Sun. 2017a.

Zi-Yi Dou, Zhi-Hao Zhou, and Shujian Huang. 2018. Un-

bilingual lexicon induction.

supervised bilingual lexicon induction via latent vari-

In ACL, Vancouver,

Canada.

able models. In EMNLP, Brussels, Belgium.
Manaal Faruqui and Chris Dyer. 2014. Improving vector

Adversarial training for unsupervised

Meng Zhang, Yang Liu, Huanbo Luan, and Maosong

space word representations using multilingual correlation. In EACL, Gothenburg, Sweden.

Sun. 2017b. Earth mover’s distance minimization for
unsupervised bilingual lexicon induction. In EMNLP,
Copenhagen, Denmark.

― 224 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.