SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1: Precision@1 for the MUSE BLI task. All baselines leverage CSLS to be the retrieve metric during inference except for Xu et al. (2018) which uses cosine similarity. The bold numbers indicate the best results of supervised and unsupervised methods. "GM" means applying the group mapping technique described in §3.4.
Method	en-fi	en-pl	en-tr
→	←	→	←	→	←
Supervised
5k+Pro.+Ref.	47.3	59.5	58.2	66.9	46.3	59.2
Unsupervised
(Conneau et al., 2017)	0.1	59.8	53.9	0.0	45.4	0.0
(Søgaard et al., 2018)	45.0	59.1	57.3	66.7	45.4	61.4
Ours (without GM)	47.1	59.2	59.7	68.4	50.2	59.7
Ours (with GM)	48.1	60.4	60.8	69.0	51.4	60.9
Table 2: Precision@1 for the MUSE BLI task of morphologically rich languages. The bold numbers indicate the best results of all methods. Pro.: Procrustes; Ref.: Refinement.
ent in morphological traits from commonly benchmarked languages which are morphological poor isolating or exclusively concatenating languages. For these languages, Søgaard et al. (2018) leverage identical tokens in both languages as the seeds (Artetxe et al., 2017), followed by the Procrustes solution plus the refinement process, which generates relatively good results. We compare our results with the supervised method, i.e., use 5k dictionary to start up followed by Procrustes + refinement, MUSE (Conneau et al., 2017) and Søgaard et al. (2018) on these languages.
From the table, we see that the GAN-based method (MUSE) fails to give good results of some directions, maybe due to its unstable training. Using identical tokens as the seed gives good results (Søgaard et al., 2018) and compares with the supervised method. Our method performs well on these morphologically rich languages, and even outperforms the supervised method. We also conduct experiments on other morphologically rich
languages such as Estonian, Greek, and Hungarian, but fail to converge.
4.3.4 Effect of Group Mapping
From Table 1 and Table 2, we also find that leveraging the group mapping (GM, §3.4) contributes to bilingual lexicon induction, especially for some distant languages such as en-ru, en-zh, and morphologically rich languages, with the improvement from 0.7 to 1.2 points. This result indicates the assumption that the embedding spaces of different languages are isomorphic may only hold locally. With the help of the cliques we extracted, we can find those locality features via clustering.
4.4 Sensitivity to Hyper-parameters
Notice that our method depends on three major hyper-parameters: (1) the number of words  $N$  we use to build word graphs; (2) the threshold  $\theta$  to prune the edges in the graphs; (3) the number of iterations  $I$  we do. In this subsection, we discuss the impact of these hyper-parameters on the BLI results, taking en2fr as an example. We depict the precision@1 on different hyper-parameter settings in Figure 2.
Figure 2: Influence of the hyper-parameters.
From the figure, we find that the performance of our method is sensitive to the choice of  $N$  and  $\theta$ . If  $N$  is too small, the cliques extracted cannot reach agreement semantically across different languages because of the sparsity of semantic units. If  $N$  is too large, the improperly trained low-frequency word vectors will impair the performance too. As for  $\theta$ , if the threshold is too small, then much noise will be introduced into the word graphs, not only reducing the quality of extracted cliques but increasing the execution time of the BK algorithm. For  $I$ , we find that the performance improves fast when  $I$  is increased from 0 to 2, but reaches convergence at 5. Too many iterations hurt the performance because, at this time, the seed dictionary inferred from the mapped cliques is redundant.
4.5 Influence to Unsupervised MT
It has been shown that BLI can benefit unsupervised machine translation (MT) (Lample et al., 2018; Marie and Fujita, 2018; Ren et al., 2019) by building Statistical Machine Translation (SMT) with the induced bilingual lexicons and language models as SMT features, followed by an iterative back-translation process. In this part, we will discuss the influence of different bilingual lexicon induction methods (Conneau et al., 2017; Artetxe et al., 2018b) to the performance of the initial SMT model, and report the BLEU scores on newstest2014 en-fr and en-de tasks in Table 3. Note that we do not do the subsequent iterative back-translation process. From the table, we see that the performance of unsupervised SMT is restricted to the quality of BLI results. As our method provides better word translations, the initial SMT models benefit from ours accordingly.
BLI Method	en2fr	fr2en	en2de	de2en
MUSE	11.74	15.34	8.14	11.03
VecMap	13.04	16.40	9.12	11.98
Ours	13.91	17.21	10.24	12.41
Table 3: BLEU of initial unsupervised SMT. The SMT features are word translation tables inferred from different BLI methods and pre-trained language models.
5.1 Extracted Cliques
In this part, we give some examples of the English cliques extracted with our method, as listed in Table 5. From the table, we see that our method can extract reasonable cliques containing words that share similar meanings. Each clique can be regarded as a semantic unit, which is more explicit than the PCA-based initialization method (Hoshen and Wolf, 2018) where they represent the semantic units with a fixed number of principal components. An interesting phenomenon is that "May" is not in the fifth clique which groups all the words of months. This is because, in this dataset, all the words are lower-cased so that "may" is also a modal verb. Besides, we observe the extracted cliques of other languages and find they are also reasonable, which are not listed here due to space limitation.
3Tested by multi-bleu.pl.
3482
en	fr	zh
MUSE	VecMap	Ours	MUSE	VecMap	Ours
and	partshare)	étabir(establish)	et(and)	也(too)	/	和(and)
his	n	matin(morning)	lui(him)	此(now)	第六(sixth)	他(he)
south	un (a)	avait(had)	ouest(west)	台北(Taipei)	(prize)	北(north)
august	flotte(fleet)	mars(march)	mars (march)	电影(film)	第五(fifth)	三月(march)
build	paris(Paris)	seule(alone)	faire(make)	用作(used as)	了解(understand)	形成(form)
Table 4: Examples of seeds produced with different methods. Inside the brackets is the interpretation of the words.
id	words
1	, . -) (
2	and also both well addition additionally besides
3	his himself him he her
4	northeastern west south southeastern southeast east southwest northeast northwest southwestern north
5	january march august july september october june april decaber november february
6	science scientists scientific biology mathematics physics chemistry sciences
…	…
Table 5: Examples of English cliques extracted from the word graph in the first iteration. The bold words are the central words in their respective cliques.
5.2 Seed Dictionary
To demonstrate that our method can produce good initial solutions for learning cross-lingual embeddings, in this part, we give an example of the seed dictionary inferred during the first iteration with our method, compared with that inferred by MUSE (Conneau et al., 2017) and VecMap (Artetxe et al., 2018b). The language pairs we choose are en-fr and en-zh, as listed in Table 4. From the table, we find that our method produces initial solutions with higher quality. This is because our coarse-to-fine process can effectively filter out the noise from the start. Notice that the initial solution produced by MUSE in the first iteration is not good, which may be because the GAN based method is not stable enough at the beginning of the training.
Bilingual lexicon induction (BLI) is an important task of machine translation. Recent methods for bilingual lexicon induction are mostly based on unsupervised cross-lingual word embeddings (Zhang et al., 2017; Artetxe et al., 2017; Conneau et al., 2017; Artetxe et al., 2018b; Xu et al., 2018; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018). They follow the same procedure that is first building initial solutions (a seed dictionary) and then learning a mapping function be
tween the two word embedding spaces. During inference, for a given source word, they find the target word via the nearest neighbors search by calculating the distance of the mapped source embedding and all target word embeddings. The main focus of the previous methods is how to find the initial solution, which is the most important part.
Their methods can be divided into three categories according to the way of finding the initial solution. The first category is using heuristic rules such as treating identical words as the seed (Artetxe et al., 2017), but this kind of method is restricted to languages sharing the vocabulary or at least the notation of numbers. The second category is adversarial methods (Zhang et al., 2017; Conneau et al., 2017; Xu et al., 2018; Alvarez-Melis and Jaakkola, 2018). They train a generator to finish mapping between the two word embedding spaces, and a discriminator to distinguish the mapped embeddings from the target embeddings. However, they suffer from the drawbacks of generative adversarial models, i.e., the sensitivity of hyper-parameters, long training time and lack of interpretability (Hoshen and Wolf, 2018). The third category is structure-based methods, which achieve the state-of-the-art performance on BLI. They either leverage the intra-linguistic word similarity information (Artetxe et al., 2018b) or principal components of monolingual word embeddings (Hoshen and Wolf, 2018), but their methods infer initial solutions just based on word-level information which is limited and prone to contain some noise due to the insufficient training of pre-trained embeddings. Different from their methods, ours leverages clique-level information which is richer and more accurate, and uses a coarse-to-fine procedure to reduce the adverse effect of the noise mentioned above.
In this paper, we propose a novel graph-based coarse-to-fine paradigm for unsupervised bilingual
lexicon induction. Our method uses clique-level information and reduces the bad effect of noise in the pre-trained embeddings. The experiments show that our method can significantly improve the bilingual word induction performance after several iterations compared with strong baselines, even for distant language pairs. In the future, we will consider combining our method with Graph Neural Networks to update the word graphs we build.
This work is supported in part by National Key R&D Program of China AAA0102301, and NSFC 61925203 & U1636210 & 61421003.
Eralp Abdurrahim Akkoyunlu. 1973. The enumeration of maximal cliques of large graphs. SIAM Journal on Computing, 2(1):1-6.
David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-wasserstein alignment of word embedding spaces. In Proceedings of the 2018 Conference on EMNLP, pages 1881-1890.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of ACL (Volume 1: Long Papers), pages 451-462.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations. In Thirty-Second AAAI Conference on Artificial Intelligence.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), pages 789-798.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018c. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on EMNLP, Brussels, Belgium. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In Proceedings of the 57th Annual Meeting of ACL.
Mikel Artetxe and Holger Schwenk. 2018. Margin-based parallel corpus mining with multilingual sentence embeddings. arXiv preprint arXiv:1811.01136.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087.
Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2014. Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568.
David Eppstein and Darren Strash. 2011. Listing all maximal cliques in large sparse real-world graphs. In International Symposium on Experimental Algorithms, pages 364-375. Springer.
Yedid Hoshen and Lior Wolf. 2018. Non-adversarial unsupervised word translation. In Proceedings of the 2018 Conference on EMNLP, pages 469-478.
Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukuttan, and Bamdev Mishra. 2019. Learning multilingual word embeddings in latent metric space: a geometric approach. Transactions of the Association for Computational Linguistics, 7:107-120.
HC Johnston. 1976. Cliques of a graph-variations on the bron-kerbosch algorithm. International Journal of Computer & Information Sciences, 5(3):209-238.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on EMNLP, pages 2979-2984.
Richard M Karp. 1972. Reducibility among combinatorial problems. In Complexity of computer computations, pages 85-103. Springer.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on EMNLP, pages 5039-5049.
Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151-159.
Benjamin Marie and Atsushi Fujita. 2018. Unsupervised neural machine translation initialized by unsupervised statistical machine translation. arXiv preprint arXiv:1810.12703.
Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168.
Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. 2019. Analyzing the limitations of cross-lingual word embedding mappings. In Proceedings of the 57th Annual Meeting of ACL.
Shuo Ren, Zhirui Zhang, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Unsupervised neural machine translation with smt as posterior regularization. In Thirty-Three AAAI Conference on Artificial Intelligence.
Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859.
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), pages 778-788.
Rui Wang, Hai Zhao, Sabine Ploux, Bao-Liang Lu, and Masao Utiyama. 2016. A bilingual graph-based semantic model for statistical machine translation. In *IJCAI*, pages 2950–2956.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of NAACL: Human Language Technologies, pages 1006-1011.
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. 2018. Unsupervised cross-lingual transfer of word embedding spaces. In Proceedings of the 2018 Conference on EMNLP, pages 2465–2474.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of ACL (Volume 1: Long Papers), pages 1959-1970.