Unsupervised Joint Training of Bilingual Word Embeddings
Benjamin Marie

Atsushi Fujita{bmarie, atsushi.fujita}@nict.go.jp
National Institute of Information and Communications Technology (NICT), Kyoto, Japan

Summary

Assumptions and research question

Key points

Assumptions• Joint training directly learns bilingual word embeddings(BWE) on parallel data (cf. mapping methods)

1. Mapping methods for BWE are limited by the dissimilarity between
the original word embedding spaces to be mapped (Søgaard+, 16)• Unsupervised machine translation generates synthetic parallel data by translating monolingual data

2. Supervised joint training algorithms (Upadhyay+, 16) are robust tonoisy data

• Unsupervised joint training uses synthetic parallel data forpseudo-supervision

3. Synthetic parallel data of a reasonable quality can be generated
through unsupervised machine translation (Artetxe+, 18a; Lample+,18)
Research question

Findings• Largely outperforms BWE obtained through unsupervised mapping in bilingual lexicon induction tasks

Do synthetic sentence pairs, generated without supervision, supply useful bilingual contextual information for jointly learning
better BWE?

• Robust to noisy synthetic parallel data and takes advantage ofmonolingual and bilingual contexts simultaneously

Unsupervised Joint Training

Evaluation in Bilingual Lexicon Induction

Unsupervised Mapping

Unsupervised Joint Training(Artetxe et al., 2018; Lample et al., 2018)(this work)

Unsupervised
training for initial BWE

Unsupervisedmachine translation
Iterative refinement

Synthetic
bilingual lexicon

Synthetic
parallel sentences

Are BWE unsupervisedly and jointly trained on noisy synthetic
data better than unsupervised mapped BWE?
Data• Monolingual training data:
News crawl for en-de and en-fr, Common Crawl for en-id• Test sets: “full” Muse Wikipedia bilingual lexicons
Baseline systems: unsupervised mapping• VECMAP: fastText word embeddings mapped without supervision(Artetxe+, 18b)

Iterative refinement
Pseudo-supervised
mapping for BWE

Evaluated systems
Pseudo-supervised
joint training for BWE• Training data: synthetic parallel data generated with unsupervised
statistical machine translation (Lample+, 18) by translating source
and/or target sentences• BIVEC: bilingual skipgram using pre-trained word alignments (Luong+,15)

Jointly trained BWE

Mapped BWE• SENTID: skipgram on a word/sentence-ID matrix (Levy+, 17)
Results (acc@1)

Evaluation in Monolingual word Analogy

MethodData usedsrc-tgt

Does unsupervised joint training improve or preserve monolingually the quality of the word embeddings (as observed for supervised bilingual skipgram)?

VECMAP

all-all

42.4

59.0

67.7

70.0

58.9

59.5

BIVEC
SENTID

10M-0
10M-0

45.8
45.8

59.2
60.1

73.9
74.4

71.3
71.8

70.4
69.8

69.7
69.2

BIVEC
SENTID

0-10M
0-10M

43.7
43.5

63.4
63.572.0
72.6

74.3
74.8

67.3
67.5

72.3
73.4

BIVEC
SENTID

10M-10M
10M-10M

44.9
45.4

54.9
62.1

73.9
74.2

73.8
74.0

69.5
69.4

72.1
73.0

Settings• Task: English word analogy (Mikolov+, 13)• Unsupervised systems: VECMAP (bilingual mapping), BIVEC (bilingualjoint training), and fastText (monolingual)
Results
MethodData used

Accuracy

VECMAP

239M(en)-38M(fr)

77.8

BIVEC

10M(en)-10M(synthetic fr)

65.7

fastText

239M(en)
10M(en)79.1
64.6• BIVEC outperforms fastText when they are trained on the same data
⇒ Joint algorithms take advantage of noisy but bilingual contexts
to monolingually improve word embeddings

en→dede→en

en→frfr→en

en→idid→en• Unsupervised joint training outperforms unsupervised mapping by a large margin– More than 10 points of improvement for en-id– Higher accuracy when synthetic parallel data do not containsynthetic English (e.g., “10M-0” for en→de, en→fr, and en→id)• BIVEC and SENTID performs similarly– pre-trained word alignments are unnecessary⇒ Joint algorithms are robust to noise and learn better BWE forbilingual lexicon induction