Accurate Cross-lingual Projection between Count-based Word Vectors
by Exploiting Translatable Context Pairs♣♢♡♢♡♢♢♠

Shonosuke Ishiwatari , Nobuhiro Kaji , Naoki Yoshinaga , Masashi Toyoda , Masaru Kitsuregawa{ishiwatari, kaji, ynaga, toyoda, kitsure} @tkl.iis.u-tokyo.ac.jp♣The University of Tokyo, ♢IIS, the University of Tokyo, ♡NICT, Japan, ♠NII, Japan

Overview
Problem: Word vectors in diﬀerent languages are notcomparable because they are learned from diﬀerent corpora!"#&'"()

Approach: Project a vector from a language into anotherlanguage space [Fung+, 98][Mikolov+, 13]&'"()

!"#*)!%+,"("-&'$"#%&'"()

$"#%

!"#2

*+,(%-'!#.+!%/,"("0&'1&'"()

Idea: Incorporate previous approaches to realize an accurate projection between word vectors!

Previous studies

Proposed approach

Directly map count-based word vectors foreach dimension by using a dictionary
amigo

friend( friend , amigo )( cat, gato )

gato

cat

Can use relationships between context words
Only one-to-one mapping is allowed

Learning-based approach [Mikolov+, 13]
Learn a linear transformation (i.e., a matrix)
between predict-based word vectors( friend , amigo )( cat, gato )learn

・Use count-based word vectors to utilize theknowledge about context words
・The existing knowledge are obtained from- training set and- surface similarity between context words

・Weight the corresponding elements in thematrix with two bonus terms( friend , amigo ) learn( cat, gato )minWni=12

 W xi − zi  −βtrain[Mikolov+, 13]

amigo(j,k)∈Dtrainbanco

wjk − βsim

wjk(j,k)∈Dsim

x: source vector, z: target vector, W: matrix

Can ﬁnd many-to-many correlation between
the elements automaticallyCan’t utilize the context words to learn a modelCan ﬁnd many-to-many correlation between
the elements automaticallyCan use relationships between context words

Experiments!J".EDD39EDH.)F.$J".$9E4#*EM)4!"#$#"$!"#$#!%!%#$#!"
12#$#3%
3%#$#12!%#$#12
12#$#!%%&'()*)+,-./01

%2345,-.671

%&'()*)+,-./01

8<=

8</

8<=

8</

8<=

&'()
&'.)
(',)
*'0)
,'0)
-'()

**'+)
./'0)
.4'/)
..'4)
.4'4)
.0'.)

,('&)
..'0)
0'4)
..'-)(',)**'4)

-.'.)
*-'.)
**'*)#*-'/)
.4'0)
4&',)

=>?@)
0/?0)#/=?=)#/0?/)#/6?0)#
0C?=)#

A@?A)
>6?A)#
0>?B)#
C@?6)#
0@?/)#=/?6)#%2345,-.671

89):)#";

.?..('4;

.'##E4'IE*

/?..DJ'D("4*'##DE**

*'##9EKK'$

C?..E4'IE*

4'#F9"G3"4$*H

4'##"E$

0?..9EKK'$

,'##$394

,'##DJ'D("4>?.."E$

('##IE("('##W#J

=?..L'*;

/,,'##DJ'D("489):)#";

8</!):.=.$9E4#*EM)4.DE4;';E$"#.)F.
NO.PDJ'D("4Q.PR:.S.T4Q.

!

Evaluation procedure:
1. Given a word vector in the source language
2. Translate the vector into the target language
3. Choose the top-n (n = 1, 5) similar vectors
in the target language
4. Check if the correct translation is includedin the n vectors

projectbank

Objective function:

project

Data:・Wikpedia dumps for learning word vectors
・Open Multilingual Wordnet for extracting
bilingual pairs to train and test the projection- Most frequent 11k words for train- The subsequent 1k words for test

friendD)99"D$.
E4#L"9

UI:ED$.)F.$J".#'V".)F.$9E'4'45.;E$E.PT#.S.T4Q
60

Proposed

50[Fung+, 98]40

Accuracy[%]

Dictionary-based approach [Fung+, 98]

30
20
100

[Mikolov+, 13]01000020000

30000

40000

size of the training set (# bilingual pairs)

50000