Introduction to Geometry ofCross-Lingual Embeddings
Yoshinari Fujinuma

1Agenda● Understand the importance of orthogonal constraints usedto train cross-lingual embeddings2

Basics of Cross-Lingual Embeddings
●
●

●

●

Shared embedding space acrossmultiple languages
Assumption: Geometric
relationship of the word vectors
are similar across languages
Popular methods learn a linear
projection matrix to map whole
embedding space into anotherPros: Leverage training data fromanother language
Image from [Mikolov+ 2013]3

Survey of Cross-lingual embeddings [Ruder+ 2017]
1.

2.

3.

Mapping-based approach [Mikolov+ 2013, etc.]
a. Understand why orthogonal constraints are important [Xing+ 2015]b. Unsupervised cross-lingual embeddings [Conneau+ 2017, etc.]
c. On the Limitations of Unsupervised Bilingual Dictionary Induction[Søgaard+ 2018]
Psuedo-parallel corpus approach (i.e., Code-switching approach)a. Replace words in a monolingual corpus and make a psuedo-code
switched corpus
Joint training approach4Understand why orthogonal constraint is importantfor cross-lingual embeddings
●

●
●
●

Geometric interpretation ofa. dot productb. skip-gram with negative sampling models [Mimno+ 2017]
Length normalization of word vectors
Orthogonal constraints for mapping two monolingual embeddings [Xing+2015]
Cross-lingual embeddings using mean squared error [Mikolov+ 2013] andorthogonal constraints

5

Geometric interpretation of Dot Product●When dot product (u・v = ||u|| ||v|| cos θ) is○ Negative: Vectors point opposite direction○ Positive: Vectors point the same direction

Image from https://www.quora.com/Can-a-scalar-product-be-negative

6

Geometric interpretation of Skip-gram with negativesampling models●●●●●

Word vector w_i
Context vector c_j
Negative context vector c_s“The king likes to eat cakes” -> (w_i, c_j) = (“king”, “eat”)
E.g., (w_i, c_s) = (“king”, “university”)

7

Geometric interpretation of Skip-gram with negative
sampling models
kingkingeatuniversity
Making w_i and c_j point thesame direction

Making w_i and negative context
vector c_s point the opposite
direction

8

Length normalization of vectors● Make the length of the vector being ||u|| = 1● Dot product becomes equivalent to cosine similarity○ u・v = ||u|| ||v|| cos θ = cos θqueen

kingking
LengthNormalization

Radius=1
square/spherequeen

9Before and after the length normalization
BeforeImage from [Xing+ 2015]

After

10

Intuition of orthogonal projection● Preserves the dot product of any two vectors after
mapped to the shared cross-lingual embedding space
EN embedding space
king
queen

Cos. sim = 0.7

Cross-lingual embedding space
queen
king

Cos. sim = 0.7

11Intuition of orthogonal projection● Preserves the dot product of any two vectors after
mapped to the shared cross-lingual embedding space12

Cross-lingual Embeddings at High-Level
EN-ES
Cross-lingual
Embeddings
ENMonolingual
Embeddings

EN-ES Bilinguallexicon

ES
Monolingual
Embeddings

13

Objective Function and Orthogonal Constraint●●●●Mean squared error [Mikolov+ 2013] with orthogonal constraints[Xing+ 2015, etc]
X: English word vectors in a bilingual lexicon
Z: Target language (e.g., Spanish) word vectors in a bilingual lexicon
W: Projection matrix from EN to target lang (or vice versa)

Minimize the mean squared error of thevectors we want to align:E.g.,● X = (u_“king”, u_“queen”)● Z = (v_“el rey”, v_“la reina”)
14

Objective Function and Orthogonal Constraint●●●●

Mean squared error [Mikolov+ 2013] with orthogonal constraints[Xing+ 2015, etc]
X: English word vectors in a bilingual lexicon
Z: Target language (e.g., Spanish) word vectors in a bilingual lexicon
W: Projection matrix from EN to target lang (or vice versa)
Pros of Orthogonal constraint
1. Preserves the dot product in the original
embedding space2. Avoids overfitting W to the translation
pairs in the bilingual lexicon
3. Has closed form solution using SVD
15(Proscurtes problem)

Unsupervised Cross-lingual Embedding [Conneau+2018, etc.]●●●Input: Two monolingual embeddings○ Does not use any form of bilingual resources (e.g., parallel corpus,bilingual lexicon)
Used in the following papers:○ Two “unsupervised machine translation” papers [Lample+ 2018a,Artexte+ 2018a]○ More recent version of those [Lample+ 2018b, Artexte+ 2018b]
Learns by iterative Proscurtes approach (“soft” orthogonal constraint)

16

What is not covered in this talk
1.
2.
3.
4.
5.

CCA-based approach [Faraqui+ 2014]
Non-linear approach [Lu+, 2015]
Unsupervised Machine Translation [Lample+ 2018a, Artexte+ 2018a, etc.]
Hubness problem [Dinu+, 2015] and its solution discussed in [Conneau+2018]
Few recent papers on cross-lingual embedding

17