A Resource-Free Evaluation Metric for Cross-Lingual
Word Embeddings based on Graph Modularity
Yoshinari Fujinuma1 Jordan Boyd-Graber2 Michael J. Paul11ͪ汬(meal, rice)ͪ΅Ω(meal, rice) .71

eating.70.75 .75 ᷣΏᷣΏΡ(eat)

(eat)

.82
.71
.71 .83 .76
汯Ζ䠷Ρ
(drink)汯ΗΡdrink(take)(drinkable).71.72 .77 .75.71 eat .71 ᅗͥ
nutritious(cook).72.70ჅΗΡ(warm up)consume

University of Colorado 2 University of Marylandlollipop
ubuntu .82 .82
firefox .80 ios.81.81 .84
mozillachromeandroid䋠उᰀ(Miyagino)揷᝛૝(Kamo River).72 .73

ኸៗ.71 (Rumoi)ϹϐςЄϞЀύ(firefox).72
ช෭ᰀ(Kasugano).72
उઊ(Shiroyama).75
ઊṛ(Yamataka)

1/28

Motivation
IYou encounter a disaster in Ethiopia
I Want a document classifier, but no labeled data in Amharic
I One solution: Exploit labeled data in English
Cross-Lingual Word Embedding
slow: -0.21, 0.35, …
ቀርፋፋ VORZ : -0.32, 0.45, …
…

I

How do you evaluate when labeled data is not available in Amharic?
2/28

OutlineI

Motivation
I Limitations: Clustering by language
I Graph modularity
I Correlations of graph modularity and downstream tasks
I Comparing to other metrics
I Conclusion

3/28

Limitations of Cross-Lingual Word Embeddings
IWords within a language tend to be closer than words from another language
We call this “Clustering by language”
I Discourages the transfer of knowledge from one language to anotherIFRQJHVWLRQVORZODWHVORZO\ODWHIDVWHUVWURQJHUVORZHUZRUVHVOXJJLVKVORZVORZV

t-SNE projection of an EN-JA embedding

4/28

Limitations of Cross-Lingual Word Embeddings
IWords within a language tend to be closer than words from another language
We call this “Clustering by language”
I Discourages the transfer of knowledge from one language to anotherIt-SNE projection of an EN-JA embedding

5/28

Quantifying Clustering by Language Using a Lexical Graph
IMain Idea: Use “Clustering by Language” to evaluate embeddings
I Convert cross-lingual word embeddings into cross-lingual lexical graphs
I k-nearest neighbor graphII

Nodes: Words
Edges: Cosine similarity between wordsᷣΏΡ(eat).83.82 汯Ζ(drink)

.75

eat
6/28

Quantifying Clustering by Language Using a Lexical GraphI

Distinguish good vs. bad embeddings by looking at the structureͪ汬(meal, rice)ͪ΅Ω(meal, rice) .71.75

eatingᷣΏ.75 (eat).70 ᷣΏΡ(eat).82.71.71 .83 .76
汯Ζ䠷Ρ
(drink)汯ΗΡdrink(take)(drinkable).71.72 .77 .75.71 eat .71 ᅗͥnutritious(cook)
.72
.70
ჅΗΡ
(warm up)consumeFigure 1: Goodlollipop
ubuntu .82 .82
firefox .80 ios.81.81 .84
mozillachromeandroid䋠उᰀ(Miyagino)揷᝛૝(Kamo River).72 .73ኸៗ.71 (Rumoi)ϹϐςЄϞЀύ(firefox).72
ช෭ᰀ(Kasugano).72
उઊ(Shiroyama).75
ઊṛ(Yamataka)Figure 2: Bad
7/28

Outline

I

Motivation
I Limitations: Clustering by language
I Graph modularity
I Correlations of graph modularity and downstream tasks
I Comparing to other metrics
I Conclusion

8/28

Defining Graph Modularity (Newman, 2003)
IFocus on edges that are connected to the same language
I Modularity = “actual intra-lingual edges” - “expected intral-lingual edges”ᷣΏΡ(eat).83
eat2−( 0.832 ) × 2 ≈ −0.34ᷣΏΡ(eat)<.82

汯Ζ(drink)

.83
eat
0.82
2.47 22−( 0.834 ) + 4 − ( 4 ) ≈ −0.14
9/28

Defining Graph Modularityͪ汬(meal, rice)ͪ΅Ω(meal, rice) .71.75

eatinglollipop.82.82
ubuntufirefox .80 ios.81.81 .84
mozillachromeandroidᷣΏ.75 (eat).70 ᷣΏΡ(eat).82.71.71 .83 .76
汯Ζ䠷Ρ
(drink)汯ΗΡdrink(take)(drinkable).71.72 .77 .75.71 eat .71 ᅗͥnutritious(cook)
.72
.70
ჅΗΡ
(warm up)consumeModularity: 0.143<䋠उᰀ(Miyagino)揷᝛૝(Kamo River).72 .73ኸៗ.71 (Rumoi)ϹϐςЄϞЀύ(firefox).72
ช෭ᰀ(Kasugano).72
उઊ(Shiroyama).75ઊṛ(Yamataka)

Modularity: 0.672
10/28

Outline

I

Motivation
I Limitations: Clustering by language
I Graph modularity
I Correlations of graph modularity and downstream tasks
I Comparing to other metrics
I Conclusion

11/28

Experiment Setup

I

Cross-Lingual Embedding Methods
ISupervised
Mean Squared Error (Mikolov et al., 2013, MSE)
MSE+Orthogonal Constraint (Xing et al., 2015)
I Canonical Correlation (Faruqui and Dyer, 2014, CCA)II

I

Unsupervised
I
I

Vecmap (Artetxe et al., 2018)
MUSE (Conneau et al., 2018)12/28

Experiment Setup

I

Cross-Lingual Embedding Methods
I

Supervised
Mean Squared Error (Mikolov et al., 2013, MSE)
MSE+Orthogonal Constraint (Xing et al., 2015)
I Canonical Correlation (Faruqui and Dyer, 2014, CCA)
I
I

I

Unsupervised
I
I

Vecmap (Artetxe et al., 2018)
MUSE (Conneau et al., 2018)12/28

Experiment Setup
I

Source Language
IIEnglish

Target Languages
SpanishItalian
I DanishI Japanese
I Hungarian
I AmharicII

See results in the paper

13/28

Task 1: Cross-Lingual Document Classification

Englishdocuments

Train

Document
Classifier

Test

Targetlang.
documents

I

Classification task of four topics
I Dataset: Reuters RCV1, RCV2 corpora (Lewis et al., 2004)

14/28

Task 1: Cross-Lingual Document Classification

Modularity

0.7
Dim
100200LangDAES
ITJA

0.6
0.5
0.4
0.2

I

0.4
0.6
Classification Accuracy

Spearman’s Correlation = −0.665

0.8

15/28

Task 2: Bilingual Lexical Induction
Bilingual Lexicon

X ሞ FDWCatDog
Eat<latexit sha1_base64="X8gJojoLvC0x13Pz62RcWtph1rk=">AAACDXicbVDLSsNAFJ3UV62vqks3g0VwVRIRdFl047KCfWAbymRy0w6dmYSZiVBCv8GNW/0Ld+LWb/An/AYnbRa29cCFwzn3Xu49QcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8epotCiMY9VNyAaOJPQMsxw6CYKiAg4dILxbe53nkBpFssHM0nAF2QoWcQoMVZ67NMR0LEgajyo1ty6OwNeJV5BaqhAc1D96YcxTQVIQznRuue5ifEzogyjHKaVfqohIXRMhtCzVBIB2s9mF0/xmVVCHMXKljR4pv6dyIjQeiIC2ymIGellLxf/9YJAWJ3JMP9wcWcvNdG1nzGZpAYknR8RpRybGOfR4JApoIZPLCFUMfsHpiOiCDU2wIoNyFuOY5W0L+qeW/fuL2uNmyKqMjpBp+gceegKNdAdaqIWokiiF/SK3pxn5935cD7nrSWnmDlGC3C+fgGur5w1</latexit><latexit

RetrieveX ᇦ (dog)⇥ ℅(paper)<latexit sha1_base64="X8gJojoLvC0x13Pz62RcWtph1rk=">AAACDXicbVDLSsNAFJ3UV62vqks3g0VwVRIRdFl047KCfWAbymRy0w6dmYSZiVBCv8GNW/0Ld+LWb/An/AYnbRa29cCFwzn3Xu49QcKZNq777ZTW1jc2t8rblZ3dvf2D6uFRW8epotCiMY9VNyAaOJPQMsxw6CYKiAg4dILxbe53nkBpFssHM0nAF2QoWcQoMVZ67NMR0LEgajyo1ty6OwNeJV5BaqhAc1D96YcxTQVIQznRuue5ifEzogyjHKaVfqohIXRMhtCzVBIB2s9mF0/xmVVCHMXKljR4pv6dyIjQeiIC2ymIGellLxf/9YJAWJ3JMP9wcWcvNdG1nzGZpAYknR8RpRybGOfR4JApoIZPLCFUMfsHpiOiCDU2wIoNyFuOY5W0L+qeW/fuL2uNmyKqMjpBp+gceegKNdAdaqIWokiiF/SK3pxn5935cD7nrSWnmDlGC3C+fgGur5w1</latexit><latexit<latexit sha1_base64="dvHItGQ2SCgtNms/rOFtIYGbZFc=">AAACCnicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF2wptKJPJtB06mYSZG6GE/oEbt/oX7sStP+FP+A1O2ixs64GBwzn33rn3BIkUBl332ymtrW9sbpW3Kzu7e/sH1cOjtolTzXiLxTLWjwE1XArFWyhQ8sdEcxoFkneC8W3ud564NiJWDzhJuB/RoRIDwShaqdNDEXFT6Vdrbt2dgawSryA1KNDsV396YczSiCtkkhrT9dwE/YxqFEzyaaWXGp5QNqZD3rVUUfuLn83WnZIzq4RkEGv7FJKZ+rcjo5ExkyiwlRHFkVn2cvFfLwgiqwsV5uctzuymOLj2M6GSFLli8yUGqSQYkzwXEgrNGcqJJZRpYe8gbEQ1ZWjTywPyluNYJe2LuufWvfvLWuOmiKoMJ3AK5+DBFTTgDprQAgZjeIFXeHOenXfnw/mcl5acoucYFuB8/QLLLJqa</latexit>

Precision@1 = 0.67
I

Translate words from a source language to a target language
I Dataset: MUSE test set
16/28

Task 2: Bilingual Lexical Induction

Modularity

0.7Dim100200LangDAESITJA

0.60.50.4
5I

1015
Precision@120

Spearman’s correlation to graph modularity = −0.789

17/28

OutlineI

Motivation
I Limitations: Clustering by language
I Graph modularity
I Correlations of graph modularity and downstream tasks
I Comparing to other metrics
I Conclusion

18/28

Metric Used in MUSE (Conneau et al., 2018)
I

MUSE trains a cross-lingual mapping matrix W without any bilingual lexicon

Select W using
validation metric
Adversarial
Step
I

Select W using
validation metricRefinement
Step

W

Default validation metric is cross-lingual similarity local scaling(CSLS) (Conneau et al., 2018)
19/28

CSLS vs. Modularity for MUSEIAverage Prec@1

Replace CSLS with modularity and compare them
I Modularity makes MUSE stable on distant language pairs
I MUSE(+CSLS) is unstable on distant language pairs (Søgaard et al., 2018)

60MetricCSLS
Modularity

40200DA

DEJA
Target Language

ZH
20/28

Comparing to Other Metrics

I
I

A good metric captures information not captured by other metrics
Predict classification accuracy by linear regression using
I

Resource-free metrics
II

I

Resource-dependent metrics
III

Modularity
CSLS (Conneau et al., 2018)
QVEC-CCA (Ammar et al., 2016)
Average cosine similarity between translations

Ablation study by omitting each metric

21/28

Comparing to Other Metrics
Using all metrics: R2 = 0.8141Predicted AccuracyI

R 2 = 0.554

0.75

dim100200lang
DA
IT0.500.25
00Modularity
QVEC-CCA
Cosine similarity
CSLS
R2 :↓ 0.260

0.25 0.50 0.75
1Actual Classification Accuracy

22/28

Comparing to Other Metrics
I

Using all metrics: R2 = 0.814

Modularity
QVEC-CCA
Cosine similarity
CSLS
R2 :↓ 0.023

IModularity
QVEC-CCA
Cosine similarity
CSLS
R2 :↓ 0.044

Modularity
QVEC-CCA
Cosine similarity
CSLS
R2 :↓ 0.111

Modularity
QVEC-CCA
Cosine similarity
CSLS
R2 :↓ 0.260

Modularity is a good metric and captures information not captured by othermetrics
22/28

OutlineIMotivation
I Limitations: Clustering by language
I Graph modularity
I Correlations of graph modularity and downstream tasks
I Comparing to other metrics
I Conclusion

23/28

Conclusion & SummaryI

Graph modularity is a good & cheap evaluation measure for cross-lingual
embeddings
III

Correlated to downstream tasks
Successful as a validation metric (for MUSE)

But combine with other metrics if possible
I

Modularity looks at only the structure, not the meanings.
ͪ汬(meal, rice)ͪ΅Ω(meal, rice) .71.75

ᷣΏ.75 (eat).70 ᷣΏΡ(eat)eating.82.71.71 .83 .76
汯Ζ䠷Ρ
(drink)汯ΗΡdrink(take)(drinkable).71.72 .77 .75.71 eat .71 ᅗͥ
nutritious(cook).72.70
ჅΗΡ
(warm up)ͪ汬
(meal, rice)ͪ΅Ω(meal, rice) .71.75

consumeModularity: 0.143

eating=.70ᷣΏ.75 (eat)ᷣΏΡ(eat).82.71.71.76
汯Ζ.83 䠷Ρ(drink)paper汯ΗΡ(take).77 .75(drinkable) .71.72.71 eat .71 ᅗͥ
nutritious(cook).72.70
ჅΗΡ(warm up)

consumeModularity: 0.143

24/28

Q&AIQuestions?

25/28

References
Ammar, W., Mulcaire, G., Tsvetkov, Y., Lample, G., Dyer, C., and Smith, N. A. (2016). Massively multilingual
word embeddings. Computing Research Repository, arXiv:1602.01925. version 2.
Artetxe, M., Labaka, G., and Agirre, E. (2018). A robust self-learning method for fully unsupervised
cross-lingual mappings of word embeddings. In Proceedings of the Association for Computational Linguistics.
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and Jégou, H. (2018). Word translation without paralleldata. In Proceedings of the International Conference on Learning Representations.
Faruqui, M. and Dyer, C. (2014). Improving vector space word representations using multilingual correlation. In
Proceedings of the European Chapter of the Association for Computational Linguistics.
Lewis, D. D., Yang, Y., Rose, T. G., and Li, F. (2004). RCV1: A new benchmark collection for text
categorization research. Journal of Machine Learning Research, 5.
Mikolov, T., Le, Q. V., and Sutskever, I. (2013). Exploiting similarities among languages for machinetranslation. Computing Research Repository, arXiv:1309.4168. version 1.
Newman, M. E. J. (2003). Mixing patterns in networks. Physical Review E, 67(2).
Søgaard, A., Ruder, S., and Vulić, I. (2018). On the limitations of unsupervised bilingual dictionary induction.
In Proceedings of the Association for Computational Linguistics.
Xing, C., Wang, D., Liu, C., and Lin, Y. (2015). Normalized word embedding and orthogonal transform forbilingual word translation. In Proceedings of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies.

26/28