SlowGuess/NAACL_2021_OCR huggingface dataset:
Random restarts Previous work (Vulic et al., 2019; Glavaš et al., 2019) shows that using GANs to train the mapping matrix  $W$  is not stable. Hartmann et al. (2019) propose to solve this problem with the random restart technique. More specifically, before going to the step of refinement, they randomly train ten mapping matrices, choosing only the best model among them for the next step. The best model is selected with the unsupervised model selection criterion. Their experiments show that this model selection method has the best performance on bilingual lexicon induction. We follow (Vulic et al., 2019; Glavaš et al., 2019) and apply the same random restart technique to train the single linear mapping and use it to initialize each subspace-specific generator.
4 CLWE mapping refinement
As in previous work, after GANs have been used to find a mapping from source to target word embeddings, a refinement step can be used to improve this mapping. Refinement involves first inducing a seed dictionary of word translations, and then refining the mapping using this seed dictionary.
Bidirectional seed dictionary induction Using the mapping learned with adversarial training, the translations  $(w_{t_1},w_{t_2},\dots,w_{t_{10000}})$  for the top ten-thousand source words  $(w_{s_1},w_{s_2},\dots,w_{s_{10000}})$  are retrieved and then back-translated into the source
language  $(w_{s_1}^{\prime}, w_{s_2}^{\prime}, \ldots, w_{s_{10000}}^{\prime})$ . The mutual translation pairs  $(w_{s_i}, w_{t_i})$  such that  $w_{s_i} = w_{s_i}^{\prime}$  constitute the seed dictionary. This guarantees that the induced seed dictionary will be bidirectional.
Mapping refinement The refinement step is based on the Procrustes Analysis (Schönemann, 1966). With the seed dictionary, the mapping can be updated using the objective in equation (1), and forced to be orthogonal using singular value decomposition (SVD) (Xing et al., 2015). Later work combines the Procrustes Analysis with stochastic dictionary induction (Artetxe et al., 2018) and greatly improves the performance of the standard refinement (Hartmann et al., 2019). More specifically, in order to prevent local optima, after each iteration some elements of the similarity matrix are randomly dropped, so that the similarity distributions of words change randomly and the new seed dictionary for the next iteration varies.
Global and local refinement Refinement can be applied to our multi-linear mapping in two different ways. First, after the training of all the subspace alignments, we can refine a linear relationship between the transformed source embeddings and the target embeddings, like previous unsupervised methods. This we call global refinement. It is noteworthy that the combination of the multi-linear mapping trained by our multi-discriminator model and the refined single linear mapping is still multilinear. Second, we can also refine the mapping of each subspace separately. More concretely, for a given subspace  $(V_{s}^{i}, V_{t}^{i})$ , we build a local seed dictionary and use the local seed dictionary to update the mapping  $G^{i}(v_{s}^{i})$ . We call this local refinement. We evaluate both global and local refinement in the next section.
Bilingual lexicon induction (BLI) has become a standard task for evaluating CLWE models. However, according to Glavaš et al. (2019) and Zhang et al. (2020), BLI performance of a given CLWE model doesn't always correlate with performance in other cross-lingual downstream tasks. In this section, we evaluate our proposal on both the task of BLI and the task of cross-lingual document classification (CLDC).
We evaluate our system both with and without refinement. Since GAN-based methods of learning CLWE are often criticized for their instability at
inducing the seed dictionary, we report the average over 10 runs for the BLI without-refinement setting. We include the random restart technique for other tasks and report the result of the best model selected by the unsupervised model selection criterion. We evaluate our model both with global refinement (G-Ref) and local refinement (L-Ref).
BLI setting We use the dataset provided by Conneau et al. (2018) for the task of BLI. This dataset contains high quality dictionaries for more than 150 language pairs. For each language pair, it provides a training dictionary of 5000 words and a test dictionary of 1500 words. This dataset allows us to have a better understanding of the performance of our proposal on many different language pairs. For each language pair, we retrieve the best translations of source words in the test dictionary using CSLS, and we report the accuracy with precision at one  $(\mathrm{P}@\mathrm{1})$
CLDC setting We use the multilingual classification benchmark (MLDoc) provided by Schwenk and Li (2018) for the task of CLDC. MLDoc contains training and test documents with balanced class priors for eight languages: German (de), English (en), Spanish (es), French (fr), Italian (it), Japanese (ja), Russian (ru) and Chinese (zh). We follow previous works (Glavaš et al., 2019; Zhang et al., 2020) and train a CNN classifier on English using 10,000 documents and test the classifier on the other seven languages. Each language contains 4000 test documents. The input of the classifier comes from the CLWE models. We report the average accuracy over ten runs.
Language pairs In this paper, we focus on projecting foreign language embeddings into the English space. We choose the eight languages included in MLDoc for both the BLI and CLDC tasks. Within the seven non-English languages, Japanese, Russian and Chinese are languages distant from English and the others are languages similar to English. For the task of BLI, we also investigate Turkish, another language distant from English.
Monolingual word embeddings We use the pretrained FastText embedding models (Bojanowski et al., 2017) for our experiments. These embeddings of 300 dimensions are pretrained on Wikipedia dumps and publicly available. Following previous works, we use the first 200,000 most
BLI Task - with refinement
de	es	fr	it	ja	ru	tr	zh
PROC	73.1	83.6	82.2	77.5	37.9	64.3	63.1	40.0
RCSLS	73.1	83.1	83.1	78.9	39.3	64.6	63.1	43.0
MUSE	73.7	83.0	82.2	78.5	29.3	62.7	60.5	38.1
VecMap	73.6	83.7	82.9	78.5	34.7	63.1	61.3	36.4
Ours GRef	74.1	83.7	82.4	78.6	34.1	64.0	61.2	38.2
Ours LRef	66.6	79.3	77.8	70.3	23.7	46.5	39.7	29.7
Table 1: BLI task results with refinement. Bold shows the best score within unsupervised systems and underline shows the best score over all the systems.
BLI Task - without refinement
de	es	fr	it	ja	ru	tr	zh
Successful runs averages
MUSE	53.9	68.9	66.9	60.7	14.7	38.1	22.3	16.2
VecMap	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Ours	55.5	69.3	67.3	59.3	18.3	38.1	28.4	19.1
Failures
MUSE	3	1	0	1	5	5	4	9
VecMap	10	10	10	10	10	10	10	10
Ours	1	1	0	0	5	4	4	8
Table 2: BLI task results for unsupervised models without refinement. We consider accuracy below  $2\%$  as failure and report the average accuracy with  $\mathrm{P@1}$  over the successful runs. Bold represents the best score.
frequent words for each monolingual embedding model. $^{7}$  We apply iterative normalization (Zhang et al., 2019) on each embedding model before training.
Baselines The objective of our proposal is to improve the mapping ability of GANs by learning a multi-linear mapping instead of only a single-linear mapping. Therefore, we use the GAN-based system MUSE (Conneau et al., 2018)<sup>8</sup> as our main unsupervised baseline. Since the unsupervised method proposed by Artetxe et al. (2018)<sup>9</sup> is considered a robust CLWE system, we also use it as our second unsupervised baseline (VecMap in the tables). In the setting with refinement, we use the iterative refinement with stochastic dictionary induction for all the unsupervised systems.<sup>10</sup> We also include two supervised systems, Procrustes (PROC) (Conneau et al., 2018) and Relaxed CSLS (RCSLS) (Joulin et al., 2018), to better understand
The original pretrained Latvian fastText model only consists of 171,000 words.
$^{9}$ https://github.com/artetxem/vecmap
<sup>10</sup>We disabled the re-weighting technique since it's not applicable for L-Ref. However, adding re-weighting to VecMap, MUSE and G-Ref doesn't change the gaps between them.
<sup>5</sup>https://github.com/zhangmoszhi/retrofit_clwe
<sup>6</sup>https://fasttext.cc/docs/en/pretrained-vectors.html
our method. Both PROC and RCSLS are robust supervised systems for learning CLWE and have been widely used previously (Glavaš et al., 2019; Zhang et al., 2020). We also wanted to include the supervised system proposed by Nakashole (2018), which learns multiple local mappings between embedding spaces, but their code is not publicly available.
5.1 Performance on BLI
We report the results of BLI both with and without refinement in Tables 1 and 2, respectively.
The results in Table 2 show a clear improvement from our multi-linear mapping, compared with single-linear GANs. We perform better than MUSE for both average accuracy and number of failures in almost every language pair. The advantages are more striking on distant language pairs.
VecMap is considered the most robust unsupervised model for learning CLWE. However, according to Hartmann et al. (2019), the advantage of VecMap mostly comes from its refinement technique. From the results in Table 1, we can see that when using the same refinement technique, our best model selected from ten random restarts using the unsupervised metric performs as well as VecMap or even better. Our model achieves higher scores on four language pairs and comes close for the other language pairs.
The results shown in Table 1 demonstrate that our model is comparable with supervised systems when using iterative refinement with stochastic dictionary induction and random restarts. We even perform better than PROC and RCSLS on similar language paris such as German and Spanish to English (de-en and es-en).
From the results in Table 1, we can easily see that the global refinement outperforms local refinement. Using local refinement we even perform much worse than our GAN-based baseline. This phenomenon does not surprise us since local refinement can easily lead to overfitting on a given subspace, and we leave the investigation of alternative refinement methods to future work.
5.2 Performance on CLDC
We report the results on the task of CLDC without and with refinement in Tables 3 and 4, respectively.
From the results shown in Table 3, we can see that our multi-linear model continues to maintain its advantage over the single-linear GAN, MUSE, in the setting without refinement. When refinement is added, MUSE becomes a little better than
CLDC Task - without refinement
de	es	fr	it	ja	ru	zh
MUSE	79.2	69.7	71.5	56.4	27.6	56.3	60.8
VecMap	21.4	24.6	23.1	23.7	21.2	23.6	22.3
Ours	80.1	67.4	73.2	62.3	30.3	61.5	69.5
Table 3: CLDC results on MLDoc dataset (Schwenk and Li, 2018) without refinement. Bold represents the best score.
CLDC Task - with refinement
de	es	fr	it	ja	ru	zh
PROC	81.4	69.6	70.7	62.9	30.0	64.9	32.0
RCSLS	81.6	70.5	71.1	62.4	29.7	64.3	31.8
MUSE	80.9	69.4	70.6	59.9	28.9	61.1	45.6
VecMap	81.8	69.8	71.0	64.0	28.4	63.2	35.4
Ours G-Ref	80.7	69.2	70.9	62.6	30.2	61.3	55.9
Ours L-Ref	79.5	69.1	69.9	62.6	30.1	62.9	59.9
Table 4: CLDC results on MLDoc dataset (Schwenk and Li, 2018) with refinement. Bold shows the best score within unsupervised systems and underline shows the best score over all the systems.
our model on German and Spanish. However, we still perform better on all those languages that are distant from English.
Differently from the task of BLI, there is no obvious advantage in the supervised baselines over our multi-linear model both with and without refinement. Conversely, as the results in Table 3 indicate, our model without refinement performs comparably or better than either our supervised baselines or VecMap in the setting with refinement. For example, our model achieves 69.5 of accuracy on Chinese test data, while the best supervised model, PROC, only has 32.0 accuracy.
While CLWE refinement is a necessary step for the BLI task, for the CLDC task our model does not seem to need refinement. As the performance gap illustrated in Figure 3 shows, our model performs worse when adding refinement for languages such as French, Japanese, Russian and Chinese, which includes all the languages distant from English. Furthermore, even for languages where we benefit from refinement, the improvement is limited.
In this paper, we propose a multi-adversarial learning method for cross-lingual word embeddings.
Figure 3: Performance gap on the CLDC task. The left panel represents the gap between our multi-linear model with refinement and without refinement. The right panel represents the performance gap between our model without refinement and the best baseline (best results selected from our supervised and unsupervised baselines). Blue bars indicate the cases where the model without refinement performs better than its competitors. Yellow bars represent the opposite cases.
Our system learns different linear mappings for different source subspaces instead of just learning a single one for the whole source space. The results of our experiments on bilingual lexicon induction and cross-lingual document classification on both close languages and distant languages prove that learning cross-lingual word embeddings with a multi-linear mapping improves performance over a single-linear mapping. Future work will focus on learning multi-linear mappings for contextualized embeddings.
David Alvarez-Melis and Tommi S. Jaakkola. 2018. Gromov-Wasserstein Alignment of Word Embedding Spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, volume 1881-1890, Brussels, Belgium.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 451-462, Vancouver, Canada.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume Long Papers, pages 789-798, Melbourne, Australia.
Antonio Valerio Miceli Barone. 2016. Towards Cross-Lingual Distributed Representations without Parallel Text Trained with Adversarial Autoencoders. In
Proceedings of the 1st Workshop on Representation Learning for NLP, pages 121-126, Berlin, Germany.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135-146.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. 2017. Parseval Networks: Improving Robustness to Adversarial Examples. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 854-863, Sydney, Australia.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word Translation Without Parallel Data. In Proceedings of the 6th International Conference on Learning Representations, pages 1-14, Vancouver, Canada.
Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving Zero-Shot Learning by Mitigating the Hubness Problem. In Proceedings of the 3rd International Conference on Learning Representations, volume Workshop Track, pages 1-10, Toulouse, France.
Haim Dubossarsky, Ivan Vulic, Roi Reichart, and Anna Korhonen. 2020. Lost in Embedding Space: Explaining Cross-Linguual Task Performance with Eigenvalue Divergence. arXiv, pages 1-10.
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710-721, Florence, Italy.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, page 2672–2680, Montréal, Canada.
Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual Dependency Parsing Based on Distributed Representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1234–1244, Beijing, China.
Mareike Hartmann, Yova Kementchedjhieva, and Anders Søgaard. 2019. Comparing Unsupervised Word Translation Methods Step by Step. In Proceedings of the 33rd Conference on Neural Information Processing Systems, pages 6033-6043, Vancouver, Canada.
Yedid Hoshen and Lior Wolf. 2018. An Iterative Closest Point Method for Unsupervised Word Translation. ArXiv.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herve Jegou, and Edouard Grave. 2018. Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979-2984, Brussels, Belgium.
Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing Crosslingual Distributed Representations of Words. In Proceedings of the 24th International Conference on Computational Linguistics, pages 1459-1473, Mumbai, India.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In Proceedings of the 1st International Conference on Learning Representations, pages 1-12, Arizona, USA.
Ndapa Nakashole. 2018. NORMA: Neighborhood Sensitive Maps for Multilingual Word Embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 512-522, Brussels, Belgium.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistic, pages 184–193, Florence, Italy.
Milos Radovanovic, Alexandros Nanopoulos, and Mirjana Ivanovic. 2010. Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data. Journal of Machine Learning Research, 11:2487-2531.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A Survey Of Cross-lingual Word Embedding Models. Journal of Artificial Intelligence Research, 65:569-631.
M. Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. 2019. Efficient Parameter-free Clustering Using First Neighbor Relations. In Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8934-8943, Long Beach, California, USA.
Holger Schwenk and Xian Li. 2018. A Corpus for Multilingual Document Classification in Eight Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, Miyazaki, Japan.
Peter H. Schonemann. 1966. A generalized solution of the Orthogonal Procrustes problem. Psychometrika, 31:1-10.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline Bilingualword Vectors, Orthogonal Transformations and the Inverted Softmax. In Proceedings of the 15th International Conference on Learning Representations, pages 1-10, Toulouse, France.
Anders Søgaard, Sebastian Ruder, and Ivan Vulic. 2018. On the Limitations of Unsupervised Bililingual Dictionary Induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume Long Papers, pages 778-788, Melbourne, Australia.
Ivan Vulic, Goran Glavaš, Roi Reichart, and Anna Korhonen. 2019. Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4407-4418, Hong Kong, China.
Ivan Vulic, Sebastian Ruder, and Anders Søgaard. 2020. Are All Good Word Vector Spaces Isomorphic? arXiv, pages 1-11.
Haozhou Wang, James Henderson, and Paola Merlo. 2019. Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4419-4430, Hong Kong, China.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1006-1011, Denver, Colorado.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial Training for Unsupervised Bilingual Lexicon Induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1959-1970, Vancouver, Canada.
Mozhi Zhang, Yoshinari Fujinuma, Michael J Paul, and Jordan Boyd-Graber. 2020. Why Overfitting Isn't Always Bad: Retrofitting Cross-Linguual Word Embeddings to Dictionaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2214–2220, Online.
Mozhi Zhang, Keyulu Xu, Ken-ichi Kawarabayashi, Stefanie Jegelka, and Jordan Boyd-Graber. 2019. Are Girls Neko or Shōjo? Cross-Linguual Alignment of Non-Isomorphic Embeddings with Iterative Normalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3180-3189.
Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual Word Embeddings for Phrase-Based Machine Translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393-1398, Seattle, Washington, USA.