SlowGuess/ACL_2020_OCR huggingface dataset:
Shuo Ren†‡, Shujie Liu§, Ming Zhou§, Shuai Ma†‡
$^{\dagger}$ SKLSDE Lab, Beihang University, Beijing, China
$^{\ddagger}$ Beijing Advanced Innovation Center for Big Data and Brain Computing, China
Microsoft Research Asia, Beijing, China
† {shuoren,mashuai} @ buaa.edu.cn § {shujliu,mingzhou} @ microsoft.com
Unsupervised bilingual lexicon induction is the task of inducing word translations from monolingual corpora of two languages. Recent methods are mostly based on unsupervised cross-lingual word embeddings, the key to which is to find initial solutions of word translations, followed by the learning and refinement of mappings between the embedding spaces of two languages. However, previous methods find initial solutions just based on word-level information, which may be (1) limited and inaccurate, and (2) prone to contain some noise introduced by the insufficiently pre-trained embeddings of some words. To deal with those issues, in this paper, we propose a novel graph-based paradigm to induce bilingual lexicons in a coarse-to-fine way. We first build a graph for each language with its vertices representing different words. Then we extract word cliques from the graphs and map the cliques of two languages. Based on that, we induce the initial word translation solution with the central words of the aligned cliques. This coarse-to-fine approach not only leverages clique-level information, which is richer and more accurate, but also effectively reduces the bad effect of the noise in the pre-trained embeddings. Finally, we take the initial solution as the seed to learn cross-lingual embeddings, from which we induce bilingual lexicons. Experiments show that our approach improves the performance of bilingual lexicon induction compared with previous methods.
Bilingual lexicon induction (BLI) is an important task of machine translation and becomes an essential part of recent unsupervised machine translation approaches (Lample et al., 2018; Artetxe et al., 2018c; Marie and Fujita, 2018; Ren et al., 2019; Artetxe et al., 2019). Previous methods for BLI are
mostly based on unsupervised cross-lingual word embeddings (Zhang et al., 2017; Artetxe et al., 2017; Conneau et al., 2017; Artetxe et al., 2018b; Xu et al., 2018; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018), the goal of which is to find a mapping function, typically a linear transformation (Mikolov et al., 2013), to map the source embeddings into the target embedding spaces. To do this, they first build a seed dictionary (known as the initial solution) with different methods and then learn the optimal mapping function that fits the seed dictionary. Based on the mapping function, a new dictionary of higher quality is inferred from the cross-lingual word embeddings by finding nearest neighbors in the target embedding space. With the new dictionary, the mapping function is further refined to fit it. The inference of the dictionary and the refinement of the mapping function are iteratively done until the final convergence. During the whole procedure, the initialization stage is important and heavily focused in previous work.
Previous methods for finding the initial solution fall into three categories. The first one is heuristic rules such as treating identical words as the seed (Artetxe et al., 2017), but this kind of method is restricted to languages sharing the alphabet. The second category is adversarial methods (Zhang et al., 2017; Conneau et al., 2017; Xu et al., 2018; Alvarez-Melis and Jaakkola, 2018), but suffering from the drawbacks of generative adversarial models, i.e., the sensitivity of hyper-parameters, long training time, etc. The third category is structure-based methods (Artetxe et al., 2018b; Hoshen and Wolf, 2018), which is more flexible and robust than other categories, and achieve the state-of-the-art BLI performance. In Artetxe et al. (2018b), they first compute a similarity matrix of all words in the vocabulary, and then represent each word with the distribution of the similarity values, while in Hoshen and Wolf (2018), they project the word
* Contribution during internship at MSRA.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3476-3485 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
vectors to the top 50 principal components of the embedding spaces. After that, both of them directly use the word representation of two languages to retrieve the initial bilingual lexicons by computing the cosine distances of source and target word representations. However, directly finding word alignments from scratch has some demerits. (1) The information that a word can provide is limited and independent of each other. (2) According to our observation, there is some noise in the pre-trained embeddings even for high-frequency words so that the initial word alignments derived from them are not accurate. Those mistakes in the initial word-level alignments can hurt the performance in the following iteration steps.
To solve those issues, we propose a novel graph-based coarse-to-fine paradigm to generate initial solutions for learning cross-lingual word embeddings, from which we induce bilingual lexicons. Specifically, given source and target languages, our method first uses pre-trained monolingual embeddings to construct a graph for each language, with the vertices representing different words, so that the mutual relationship between words is preserved. Next, we use the Bron-Kerbosch algorithm (Akkoyunlu, 1973) to extract cliques (a subset of vertices in which every two distinct vertices are adjacent) in the source and target graphs. After that, we calculate the clique embeddings and map the cliques from two graphs. We then treat the central words of the aligned cliques as the seeds to learn the mapping of the two word embedding spaces.
Our contributions are threefold. (1) By building word graphs, we leverage the clique-level information extracted from them. The cliques cluster similar words and assemble their mutual relationship of them, providing richer and more accurate information. (2) We propose the coarse(clique extraction)-to-fine(seed induction) procedure for the BLI task, which effectively reduces the bad effect of the noise in the pre-trained embeddings; (3) We improve the BLI performance on the MUSE dataset with our method, even compared with strong baselines.
Unsupervised bilingual lexicon induction (BLI) is the task of inducing word translations from monolingual corpora of two languages. Recently proposed methods follow the same procedure, i.e., first learning cross-lingual embeddings in an unsupervised way (§2.1) and then inducing bilingual
lexicons from the embedding spaces (§2.2).
2.1 Unsupervised Cross-lingual Embeddings
Previous methods for learning cross-lingual embeddings can be roughly divided into two categories (Ormazabal et al., 2019), i.e., mapping methods and joint learning methods. As the second category, the skip-gram (Luong et al., 2015) for example, requires bilingual corpus during training, current methods for unsupervised cross-lingual embeddings mainly fall into the first category. Given pretrained monolingual embeddings of two languages, the mapping methods try to map the source and target embedding spaces through a linear transformation (Mikolov et al., 2013)  $\mathbf{W} \in \mathbf{M}_{d \times d}(\mathbb{R})$ , where  $\mathbf{M}_{d \times d}(\mathbb{R})$  is the space of  $d \times d$  matrices of real numbers and  $d$  is the dimension of the embeddings. Based on that, Xing et al. (2015) propose to constrain  $\mathbf{W}$  to be orthogonal, i.e.,  $\mathbf{W}^{\top} \mathbf{W} = \mathbf{I}$ , and Conneau et al. (2017) find this is a Procrustes problem which advantageously offers a closed-form solution obtained from singular value decomposition (SVD) of  $\mathbf{Y} \mathbf{X}^{\top}$  as follows:
$$
\mathbf {W} ^ {*} = \underset {\mathbf {W}} {\arg \min } | | \mathbf {W} \mathbf {X} - \mathbf {Y} | | _ {F} = \mathbf {U V} ^ {\top}, \tag {1}
$$
$$
\text {w i t h} \mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {\top} = \operatorname {S V D} \left(\mathbf {Y} \mathbf {X} ^ {\top}\right)
$$
where  $\mathbf{X}$  and  $\mathbf{Y} \in \mathbf{M}_{d \times n}(\mathbb{R})$  consist of the embeddings of the bilingual lexicons  $\{x_i, y_i\}_{i=1}^n$  in the seed dictionary.
Therefore, there are two steps to learn unsupervised cross-lingual embeddings. The first step is to find an initial solution (also known as the seed dictionary), and the second one is to obtain the desired  $\mathbf{W}$  according to Eq. (1). The above two steps can be iteratively done, by inducing new seed dictionary from the learned cross-lingual embeddings with the method introduced next, and using the new dictionary to refine the matrix  $\mathbf{W}$  (known as the "refinement" process in some literature).
The first step, i.e., finding the initial solution, is crucial because it decides the direction of the following iteration. Loads of previous work are devoted to finding good initial solutions with different methods, as is described in §1. But their methods only exploit word-level information, which is limited and may be inaccurate due to the noise in pretrained monolingual embeddings, leading to mistakes in the initial word-level alignments. Therefore, we propose a novel graph-based coarse-to-fine paradigm to find the initial solution of higher qual
ity, leveraging clique-level information which we think is richer and more accurate.
2.2 Bilingual Lexicon Induction
Based on the learned cross-lingual embeddings, bilingual lexicons can be induced from the mapped spaces via the nearest neighbor (NN) method by calculating the cosine distance of the mapped source embeddings and the target embeddings. However, this method suffers from the "hubness" problem (Dinu et al., 2014) such that some target words appear as the nearest neighbors of many source words. To mitigate this problem, alternatives of the distance function have been proposed, such as invsoftmax (Smith et al., 2017), CSLS (Conneau et al., 2017) and margin-based scores (Artetxe and Schwenk, 2018). Among them, CSLS, as a special case of margin-based scores, is widely used in the SOTA embedding-based BLI methods. Formally, CSLS calculates the distance between the mapped and the target embeddings as follows:
$$
\operatorname {C S L S} (\mathbf {W x}, \mathbf {y}) = 2 \cos (\mathbf {W x}, \mathbf {y}) - r _ {\mathrm {T}} (\mathbf {W x}) - r _ {\mathrm {S}} (\mathbf {y}) \tag {2}
$$
$$
r _ {\mathrm {T}} (\mathbf {W} \mathbf {x}) = \frac {1}{K} \sum_ {y \in \mathcal {N} _ {\mathrm {T}} (\mathbf {W} \mathbf {x})} \cos (\mathbf {W} \mathbf {x}, \mathbf {y}) \tag {3}
$$
is the mean similarity of a source embedding  $\mathbf{x}$  to its  $K$  target neighborhoods  $(\mathcal{N}_{\mathrm{T}}(\mathbf{W}\mathbf{x}))$ . Similarly,  $r_{\mathrm{S}}(\mathbf{y})$  is the mean similarity of a target embedding  $\mathbf{y}$  to its neighborhoods.
As is mentioned before, recent work on bilingual lexicon induction (BLI) is mostly based on unsupervised cross-lingual embeddings, whose key point is to find initial solutions to learn the mapping function. However, previous methods find initial solutions just based on word-level information, which may be limited and inaccurate due to the noise in pre-trained monolingual embeddings. Therefore, we exploit the information provided by word cliques and figure out a coarse-to-fine procedure to denoise and find the initial solution of higher quality. Based on that, we learn the cross-lingual embeddings and induce word translations.
As shown in Figure 1, our method for BLI can be roughly divided into several steps. Given the source and target languages, we first build a graph for each language. The graph vertex represents
the word. Next, we extract word cliques from the graphs and map the cliques of two languages in an unsupervised way. Then, we induce the seed dictionary from the bilingual cliques by choosing the respective central words of the aligned cliques. After that, we learn cross-lingual embeddings with the help of the induced seed dictionary. The above steps can be iteratively done until the final convergence. By building word graphs, we can use the clique-level information which is richer and more accurate than what a single word provides. Besides, the whole coarse-to-fine procedure also reduces the bad effect of the noise in the pre-trained embeddings, because the clique-level alignment (coarse) is more accurate at the beginning and the word alignments inferred from it (fine) are more reasonable. We will next introduce each step.
3.1 Word Graph Construction
Given the pre-trained monolingual embeddings, we can derive an edge-weighted graph from them by regarding words as the vertices and their similarities as edges. Formally, the graph is
$$
G = <   V, E > \tag {4}
$$
where  $V$  is the vertex set (vocabulary of each language) and  $\mathbf{E}$  is the edge set. The edges are built with monolingual embedding similarities. For example, for language  $x$ , to define the edges, we first get the word-to-word similarity matrix  $\mathbf{M}$  with
$$
\mathbf {M} _ {i, j} = \left\{ \begin{array}{l l} \operatorname {C S L S} (\mathbf {x} _ {i}, \mathbf {x} _ {j}), & i \neq j \\ 0, & i = j \end{array} \right. \tag {5}
$$
where  $\mathbf{x}_i$  and  $\mathbf{x}_j$  are the normalized embeddings of two words respectively. We set the main diagonal elements to zero to avoid self-loop. Theoretically, there is one edge between any two arbitrary words with the edge weight to be  $\mathbf{M}_{i,j}$ , but if the weight of an edge is too small, it will provide little information and introduce a lot of noise. Therefore, we prune these non-informative edges with  $\mathbf{M}_{i,j}$  less than a threshold of  $\theta$ . Meanwhile, the pruning greatly reduces the computation time of the next step. We build two graphs  $G_{x}$  and  $G_{y}$  for two languages  $x$  and  $y$  in this way respectively.
3.2 Clique Extraction and Mapping
Different from previous methods, we infer the initial solution not using word-level information but from word cliques, which we think is richer and more accurate. Following Wang et al. (2016), the
Figure 1: Overview of our method. In each iteration, based on the word graphs, we first map the cliques of two languages in an unsupervised way, and then infer the seed dictionary to learn cross-lingual word embeddings.
"clique" here means a maximum complete subgraph where every two distinct vertices in the clique are adjacent. Extracting cliques from a given graph is a nontrivial problem and is shown to be NP-complete (Karp, 1972). In this paper, we adopt Bron-Kerbosch (BK) algorithm (Akkoyunlu, 1973) with pivoting (Johnston, 1976) to extract the cliques from a given graph. Having extracted the word cliques of two languages, we calculate clique embeddings by averaging the embedding vectors of all words in each clique. We choose the word whose embedding is closest to its clique embedding as the central word of each clique. After that, we follow Artetxe et al. (2018b) to map the cliques of two languages in a fully unsupervised way, i.e. to learn cross-lingual clique embeddings.
We use the clique extraction rather than clustering methods because (1) a word may fall into different categories because of polysemy, which can be well modeled by the cliques, and (2) the BK algorithm is much more efficient than clustering.
3.3 Seed Dictionary Induction
§3.2 maps the clique embeddings of two languages into the same space so that we can retrieve aligned cliques. For each source clique, we choose the nearest target clique according to the CSLS similarity score calculated by Eq. (2). Remember that we have chosen the central word for each clique after the clique extraction in §3.2, so the seed dictionary inferring process is simply picking the central words of the aligned cliques just as shown in Figure 1. Note that we remove the duplication of seed word pairs in this process.
3.4 Cross-lingual Embedding Learning
Based on the initial solution (known as the seed dictionary), we then learn cross-lingual word embeddings following the Procrustes and refinement
process introduced in §2.1. After obtaining the learned cross-lingual word embeddings, we rebuild the word graphs with the help of them and iterate the whole process again until the final convergence as shown in Figure 1.
Previously methods used a single matrix  $\mathbf{W}$  as transformation function between the embedding spaces of two languages, based on the assumption that the embedding spaces of different languages are isomorphic (Mikolov et al., 2013). However, this is doubtful because the isomorphic assumption may not hold all the time (Søgaard et al., 2018). Fortunately, the cliques we extracted naturally provide good local features for us, because they are usually much different from each other in meanings, which enables us to investigate alternatives to a single mapping matrix  $\mathbf{W}$ . Therefore, after the final iteration, we divide all the cliques into  $K$  groups via clustering, i.e.,  $\{L_i\}_{i=1}^K$ , and train an individual matrix  $\mathbf{W}_i$  for each of them. We denote this process as "group mapping". Each  $\mathbf{W}_i$  is initialized with the learned  $\mathbf{W}$  and fine-tuned as
$$
\mathbf {W} _ {i} = \underset {\mathbf {W} _ {i}} {\arg \min } \left| \left| \mathbf {W} _ {i} \mathbf {X} _ {i} - \mathbf {Y} _ {i} \right| \right| _ {\mathrm {F}}, \text {s . t .} \mathbf {W} _ {i} ^ {\top} \mathbf {W} _ {i} = \mathbf {I} \tag {6}
$$
where  $\mathbf{X}_i$  and  $\mathbf{Y}_i$  are the embedding matrices of words belonging to  $L_i$ . We divide each word into the group closest to its word embedding. The whole training procedure is shown in Algorithm 1.
After the training, we can obtain the renewed word graphs of both languages as well as their cliques, and get a set of group mapping matrices  $\{\mathbf{W}_i\}_{i=1}^k$ . During the inference, for each source word  $x$ , we first find its closest clique  $C_s$  by calculating the similarities of  $x$ 's embeddings to all clique embeddings. Next, we retrieve the group  $L_s$  that  $C_s$  belongs to, and choose the corresponding  $\mathbf{W}_s$ . Then,
Algorithm 1: Training procedure of the proposed graph-based coarse-to-fine method.
we retrieve the translation of  $x$  by calculating the CSLS score of  $\mathbf{W}_s\mathbf{x}$  and each target embedding  $\mathbf{y}$ , similar to Eq. (2) introduced in §2.2.
Bilingual lexicon induction (BLI) measures the word translation accuracy in comparison to a gold standard. We report results on the widely used MUSE dataset (Conneau et al., 2017). This dataset consists of monolingual fastText (Bojanowski et al., 2017) embeddings of many languages and dictionaries for many language pairs divided into training and test sets. The evaluation follows the setups of Conneau et al. (2017).
4.2.1 Pre-processing
We choose the top 10,000 word embeddings to build word graph because the monolingual embeddings of low-frequency words may be trained insufficiently. The embeddings are normalized following Artetxe et al. (2018b). Specifically, we first apply length normalization to the embeddings, and then mean center each dimension. After that, we do length normalization again to ensure the word embeddings have a unit length.
4.2.2 Clique Extraction
An efficient algorithm for clique extraction is the Bron-Kerbosch (BK) algorithm, which is a recursive backtracking algorithm that searches for all maximal cliques in a given graph  $G$ . The pruning operation described in §3.1 makes the word graph a sparse graph, for which the BK algorithm can be made to run in time  $O(dn3^{d/3})$  (Eppstein and Strash, 2011), where  $n$  is the number of vertexes in  $G$ , and  $d$  is the degeneracy<sup>1</sup> of the graph. We choose a public efficient C implementation of BK algorithm<sup>2</sup>, and only extract the cliques that contain no less than three words. According to our observation, the cliques can be extracted within several seconds with this code.
4.2.3 Clique and Word Embedding Mapping
In our experiment, the clique embeddings of two languages are mapped with the method proposed by Artetxe et al. (2018b). We use their public code to finish this step. We initialized  $W$  with a random orthogonal matrix. After building the seed dictionary, we first solve the Procrustes problem (Eq. (1)), followed by the refinement process.
4.3.1 Baselines
We choose several supervised and unsupervised methods to be our baselines. The supervised baselines include: (1) The iterative Procrustes method proposed by Smith et al. (2017); (2) The multi-step framework proposed by Artetxe et al. (2018a); (3) a geometric method proposed by Jawanpuria et al. (2019). The unsupervised baselines include (1) MUSE proposed by Conneau et al. (2017), which is a GAN based method followed by a refinement process; (2) a Wasserstein GAN based method combined with distribution matching and back translation, proposed by Xu et al. (2018); (3) a method proposed by Alvarez-Melis and Jaakkola (2018) that views the mapping problem as optimal transportation and optimize the Gromov-Wasserstein distance between embedding spaces; (4) A robust self-learning method proposed by Artetxe et al. (2018b), which leverages the intra-linguistic word similarity information to infer initial solutions, followed by a self-learning iteration; (5) A nonadversarial method proposed by Hoshen and Wolf
$^{1}$ In graph theory, a k-degenerate graph is an undirected graph in which every subgraph has a vertex of degree  $\leq k$ .  $^{2}$ https://github.com/aaronmcaid/Maximal Cliques
Method	en-fr	en-de	en-es	en-it	en-ru	en-zh
→	←	→	←	→	←	→	←	→	←	→	←
Supervised
(Smith et al., 2017)	81.1	82.4	73.5	72.4	81.4	82.9	43.1	38.0	51.7	63.7	42.7	36.7
(Artetxe et al., 2018a)	80.5	83.1	73.5	73.5	80.5	83.8	61.3	39.6	50.5	67.3	32.3	43.4
(Joulin et al., 2018)	83.3	84.1	79.1	76.3	84.1	86.3	-	-	57.9	67.2	45.9	46.4
(Jawanpuria et al., 2019)	82.1	84.2	74.9	76.7	81.9	85.5	-	-	52.8	67.6	49.1	45.3
Unsupervised
(Conneau et al., 2017)	82.3	81.1	74.0	72.2	81.7	83.3	77.4	76.1	44.0	59.1	32.5	31.4
(Xu et al., 2018)	77.9	75.5	69.3	67.0	79.5	77.8	72.6	73.4	-	-	-	-
(Alvarez-Melis and Jaakkola, 2018)	81.3	78.9	71.9	72.8	81.7	80.4	78.9	75.2	45.1	43.7	-	-
(Artetxe et al., 2018b)	82.3	83.6	75.1	74.3	82.3	84.7	78.8	79.5	49.2	65.6	-	-
(Hoshen and Wolf, 2018)	82.3	84.1	74.7	73.0	82.1	84.1	77.9	77.5	47.5	61.8	-	-
Ours (without GM)	82.7	83.4	75.5	75.7	82.6	84.8	78.6	79.5	48.9	63.9	38.1	35.2
Ours (with GM)	82.9	83.9	75.3	76.1	82.9	85.3	79.1	79.9	49.7	64.7	38.9	35.9
(2018), which uses PCA-based alignment to initialize and iteratively refine the alignment.
4.3.2 Results of Common Languages
We report the result of the BLI task on the MUSE dataset (Conneau et al., 2017). The language pairs we choose are French (fr), German (de), Spanish (es), Italian (it), Russian (ru), Chinese (zh) from and to English(en), as shown in Table 1.
From Table 1, we find that our proposed method significantly outperforms previous methods on nearly all directions, especially on en-de and en-zh pairs, with the improvements of 2 to 6 points compared with previous state-of-the-art unsupervised approaches. The results on some language pairs such as en-fr, en-de and en-es are remarkably competitive with strong supervised methods.
We also see that for distant languages, i.e., en-ru and en-zh, our method achieves good results, on which some unsupervised baselines fail to converge. However, the results are still far lagging behind the supervised methods, indicating that the seed dictionaries built with our method may not be perfect for these distant languages. This may root in the original diversified training data of the monolingual embeddings on those pairs. Even so, we still significantly outperforms the MUSE (Conneau et al., 2017) for the en-ru and en-zh pairs.
4.3.3 Results of Morphologically Rich Languages
We also list results of some morphologically rich languages, i.e., Finnish (fi), Polish (pl) and Turkish (tr) in Table 2, which are selected by Søgaard et al. (2018). They find that these languages are differ