SlowGuess/ACL_2020_OCR huggingface dataset:
Vanja Mladen Karan\* Ivan Vulic\* Anna Korhonen\* Goran Glavaš
$\spadesuit$ TakeLab, Faculty of Electrical Engineering and Computing, University of Zagreb
$\diamond$  Language Technology Lab, TAL, University of Cambridge
$\clubsuit$  Data and Web Science Group, University of Mannheim mladen.karan@fer.hr {iv250,alk23}@cam.ac.uk goran@informatik.uni-mannheim.de
Effective projection-based cross-lingual word embedding (CLWE) induction critically relies on the iterative self-learning procedure. It gradually expands the initial small seed dictionary to learn improved cross-lingual mappings. In this work, we present CLASSYMAP, a classification-based approach to self-learning, yielding a more robust and a more effective induction of projection-based CLWEs. Unlike prior self-learning methods, our approach allows for integration of diverse features into the iterative process. We show the benefits of CLASSYMAP for bilingual lexicon induction: we report consistent improvements in a weakly supervised setup (500 seed translation pairs) on a benchmark with 28 language pairs.
1 Introduction and Motivation
Cross-lingual word embeddings (CLWEs), that is, representations of words in a shared cross-lingual vector space, enable multilingual modeling of meaning and facilitate cross-lingual transfer for downstream NLP tasks (Ruder et al., 2019). One of their primary use cases is bilingual lexicon induction (BLI), that is, learning translation correspondences across languages which benefit the development of core language technology also for resource-poor languages and domains (Adams et al., 2017; Smith et al., 2017; Heyman et al., 2018; Hangya et al., 2018; Vulic et al., 2019).
Earlier work focused on joint CLWE induction from bilingual corpora, relying on word-(Klementiev et al., 2012; Gouws and Søgaard, 2015), sentence-(Zou et al., 2013; Hermann and Blunsom, 2014; Coulmance et al., 2015; Levy et al., 2017), or document-level supervision (Søgaard et al., 2015; Vulic and Moens, 2016). However, recent focus is predominantly on post-hoc alignment of independently trained monolingual word embeddings: the
so-called projection-based or mapping approaches (Mikolov et al., 2013; Conneau et al., 2018; Joulin et al., 2018; Artetxe et al., 2018b; Patra et al., 2019). Such methods are particularly suitable for weakly supervised learning setups: they support CLWE induction with only as much as few thousand word translation pairs as the bilingual supervision. $^{1}$
One critical component of weakly supervised projection-based CLWEs is a self-learning procedure that iteratively refines the initial seed dictionary to learn projections of increasingly higher quality. This process leads to substantial improvements of the initially mapped space, especially with smaller seed dictionaries (Artetxe et al., 2017; Vulic et al., 2019). However, current self-learning procedures are still rather basic, typically relying only on direct extraction of (mutual) nearest neighbors from the current shared space (Conneau et al., 2018; Artetxe et al., 2018b; Glavaš et al., 2019). In this work, we propose a more sophisticated self-learning procedure for weakly supervised projection-based CLWE methods, and show its benefits for a wide range of language pairs.
We frame self-learning as iterative classification-based process, which yields several benefits over the previously used self-learning mechanisms. 1) It enables integration of a variety of heterogeneous features at different levels of granularity (e.g., word-level vs. orthographic features); some trans
In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only (Conneau et al., 2018; Artetxe et al., 2018b; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018; Chen and Cardie, 2018; Mohiuddin and Joty, 2019, inter alia). However, as shown in recent comparative empirical analyses (Glavaš et al., 2019; Vulić et al., 2019), using seed sets of only 500-1,000 translation pairs, with all other components equal, always outperforms fully unsupervised methods. Therefore, we focus on a more natural weakly supervised setup (Artetxe et al., 2020) instead, i.e., we assume the existence of at least 500 seed translations for each language pair in consideration.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6915-6922 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
lation cues (e.g., subword-level overlap) have been ignored by previous self-learning approaches. 2) It allows us to control for the reliability of translation pairs considered as candidates for the dictionary updates in the current iteration. Effectively, this helps reduce noise in the process as the training dictionary grows. 3) As suggested by prior work on classification-based BLI (Irvine and Callison-Burch, 2017; Heyman et al., 2017), framing the actual BLI task as a classification problem results in further gains in the final BLI performance.
We extensively evaluate our classification-based self-learning procedure, termed CLASSYMAP, on the standard BLI data set (Glavaš et al., 2019) spanning 28 pairs of diverse languages. The integration of the proposed self-learning method into VECMAP (Artetxe et al., 2018b), a state-of-the-art projection-based CLWE framework, yields substantial gains over previous self-learning procedures. We demonstrate that the improvements are indeed achieved through the synergy of diverse features used by the classifier. We also demonstrate further BLI improvements when we treat BLI as a supervised classification-based task.
2 Classification-Based Self-Learning
Projection-Based CLWE Methods (linearly) align independently trained monolingual word embeddings  $X_{1}$  of the source language  $L_{1}$  and  $X_{2}$  (target language  $L_{2}$ ), using a seed word translation dictionary  $D$  (Mikolov et al., 2013; Artetxe et al., 2018a). Working in weakly supervised setups, we assume the existence of some translation pairs ( $\approx$  500 pairs) in  $D$ . Let  $X_{1,D} \subset X_{1}$  and  $X_{2,D} \subset X_{2}$  refer to the row-aligned subsets of monolingual embedding spaces containing vectors of translation pairs from  $D$ . Those are used to learn orthogonal transformations  $T_{1}$  and  $T_{2}$  that define the final shared cross-lingual space  $W_{cl} = W_{1} \cup W_{2}$ , where  $W_{1} = X_{1}T_{1}$  and  $W_{2} = X_{2}T_{2}$ .
Our departure point is a standard self-learning setup from related work (Artetxe et al., 2018b; Conneau et al., 2018), outlined in the following. At each iteration  $k$ , the dictionary  $D^{(k)}$  is first used to learn the joint space  $\mathbf{W}_{cl}^{(k)} = \mathbf{W}_1^{(k)} \cup \mathbf{W}_2^{(k)}$ .
The nearest neighbours in  $W_{cl}^{(k)}$  are then used to extract the new dictionary  $D^{(k + 1)}$ . Previous work typically relies on a variant of mutual nearest neighbours in the aligned embedding space of the current iteration to select likely translation candidates for the next. However, as hinted by Lubin et al. (2019), that procedure still results in many noisy candidates inserted in the extended seed sets, and the error may get amplified over subsequent iterations.
New Self-Learning Procedure. Therefore, we propose a more versatile self-learning process. We train a supervised classifier in each iteration: given a word pair, it produces a probability score denoting to which extent the pair is a correct translation pair. The classifier can be fed a wide range of features on the character, subword, and word level.
We apply the classifier in two ways. First, at iteration  $k$  the classification scores are used to select likely translation candidates which are added to the dictionary  $D^{(k + 1)}$  for iteration  $k + 1$ . Second, similar to Heyman et al. (2017), at test time we use the classifier scores to rerank translation candidates produced by 1) finding nearest neighbours in the final aligned embedding space and 2) considering orthographically similar candidates. A high-level overview of the proposed classification-based self-learning procedure is outlined in Algorithm 1.
Self-Learning: Components. For implementing the AlignEmbeddings operation (see Algorithm 1) we rely on the VECMAP<sup>4</sup> system (Artetxe et al., 2018b) in its supervised variant. The nn
2We use VECMAP due to its very competitive and robust BLI performance according to the recent comparative studies (Glavaš et al., 2019; Vulić et al., 2019; Doval et al., 2019). We note that our methodology is equally applicable to other projection-based methods that employ self-learning e.g., (Conneau et al., 2018; Mohiuddin and Joty, 2019), and our preliminary results with other methods suggest the similar benefits stemming from the classification-based approach.
3We later show in §3 that both usages are beneficial for BLI. The former yields improved CLWEs directly. We plan to probe the usefulness of the CLWEs in other tasks beyond BLI in future work. The latter (reranking) step, on the other hand, is tied to the BLI task in particular. For this reason we later report all BLI results both with and without reranking.
4https://github.com/artetxem/vecmap
6916
function returns word pairs that are nearest neighbours in a given aligned embedding space. The TrainClassifier functionality can be instantiated using any standard classification framework. In this work, we opt for a simple a multi-layer perceptron with a single hidden layer.
A very important design choice concerns generating negative training examples for the classifier. All word pairs in the dictionary at current iteration  $D^{(k)}$  are used as positive examples. For each positive pair  $(s, t)$ , we generate two negative examples: 1)  $(s, x)$ , where  $x$  is sampled uniformly from  $N_{o}$  target words which are orthographically (measured by edit distance) most similar to  $s$ ; 2)  $(s, y)$ , where  $y$  is sampled uniformly from  $N_{c}$  target words closest (by cosine) to  $s$  in the current space  $W_{cl}^{(k)}$ .
This strategy performed considerably better than randomly generating negative examples. The intuition is as follows: at test time the classifier must operate on word pairs that are generated using nearest neighbour search. Such word pairs are not random, but are rather very close in the aligned embedding space and are often orthographically similar. Thus, this strategy for generating negative samples makes the train conditions for the classifier better reflect the test conditions.
Features. The classification-based approach allows for the integration of a wide spectrum of diverse features that capture different word translation evidence. We outline the sets of features used in this work, computed for each word pair  $(s,t)$ .
F1. Edit distance - Levenshtein and Jaro-Winkler distance between  $s$  and  $t$  (Cohen et al., 2003). Following Heyman et al. (2017) we also include normalized edit distance, log of the rank of  $t$  in a list sorted by edit distance with respect to  $s$ , as well as a product of these two values.
F2. Cosine similarity of  $s$  and  $t$  in  $\mathbf{W}_{cl}^{(k)}$  (at iter  $k$ ).
F3. Aligned embeddings of  $s$  and  $t$ , PCA-reduced to 10 dimensions (20 features in total).
F4. Normalized n-gram overlap (Sarić et al., 2012);
F5. Character n-grams - we extract all character n-grams and use  $\chi^2$  feature selection to select the 10 most indicative ones. The intuition is to allow the model to recognize indicative prefixes or suffixes.
F6. Subword-level similarity - we use multilingual subword embeddings (SWEs) based on BPEs (Heinzerling and Strube, 2018). We add the following features: i) we average the BPEs of  $s$  and  $t$  and calculate cosine similarity of the resulting vectors,
ii) the pairwise maximum cosine similarity of all pairs of SWEs (one from  $s$  and the other from  $t$ ), and iii) the Earth Mover's distance between the two sets of SWEs (Kusner et al., 2015).
F7. Frequencies - we provide the rank of the word in a list of all words sorted by frequency. The ranks are normalized by the number of words.
At test time, if we use the classifier to perform the final reranking, we take for each source word  $s$  a set of candidate target word translations as the union of 1) the top  $N_{ro}$  target word neighbours of  $s$  by edit distance, and 2) the top  $N_{rc}$  target word neighbours of  $s$  by cosine in the final aligned  $W_{cl}$ . We then score the  $N_{ro} + N_{rc}$  candidates using the classifier from the last self-learning iteration.
Monolingual Vectors and BLI Data. Following prior work (Artetxe et al., 2018b; Glavaš et al., 2019), we start from monolingual fastText vectors trained on full Wikipedias for each language (Bojanowski et al., 2017); vocabularies are trimmed to the 200K most frequent words. We evaluate on the standard BLI dataset from Glavaš et al. (2019): it comprises 28 language pairs with a good balance of typologically similar and distant languages: English (EN), German (DE), Italian (IT), French (FR), Russian (RU), Croatian (HR), Turkish (TR), and Finnish (FI). As our focus is on weakly supervised setups, we use only 500 translation pairs as our initial seed dictionary. We report BLI performance using the standard Precision@1 (P@1) measure.
Classifier Details. We use the Adam optimizer (Kingma and Ba, 2015) and regularize the model via  $\ell_2$ -penalty on the weights and early stopping on  $10\%$  of held-out data. Early stopping is performed for each language pair separately, while other hyperparameter values are found by grid search maximizing a three-fold cross-validation score on the training data for a randomly selected language pair (EN-HR), and reused in all other experiments.
Hyperparameters. We find values for other hyperparameters on held-out data for a randomly chosen language pair: EN-HR. Unless otherwise stated, we fix them to the following values for all other experiments and language pairs. In Algo-
5 Hidden layer sizes explored are 3, 5, 10, 20, 25 and regularization strengths are 0.0001, 0.01, and 1. The values selected by grid search were 25 and 1, respectively.
6917
rithm 1,  $P = 1000$ ,  $K = 500$ ,  $n = 30$ . Further, we sample 2 negative examples per each positive example from the sets of size  $N_{o} = N_{c} = 5$ ,  $N_{ro} = N_{rc} = 3$  when doing the final reranking. We note that more careful tuning of these values could lead to further improvements in results.
Baselines. We compare to the VECMAP system (Artetxe et al., 2018b) in its semi-supervised variant as a robust and highly competitive self-learning framework (Glavaš et al., 2019; Vulić et al., 2019).
3.2 Results and Discussion
The main results over a representative selection of language pairs and setups are provided in Table 1. Full results over all 28 pairs are provided in Appendix A. The results indicate several important findings. First, classification-based self-learning is more powerful than the standard VECMAP self-learning: we observe gains on 22/28 pairs using CLASSYMAP without the final reranking step, even without language pair-dependent fine-tuning. Second, framing BLI as a classification task leads to further gains: we report improvements on 25/28 pairs using CLASSYMAP with the final reranking step over both supervised and semi-supervised VECMAP variants. Using reranking with CLASSYMAP seems useful across the board.[6]
As a side finding, our results also revalidate the evident usefulness of the self-learning procedure for weakly supervised setups in general (Vulic et al., 2019): the average  $\mathrm{P}@\mathrm{l}$  score across All languages of a supervised VECMAP method based on the same initial dictionary, but without any self-learning, is only 0.111, while we report the average of 0.365 (with final reranking) in Table 1.
Importantly, the gains seem more pronounced for more "difficult", typologically dissimilar, and morphologically rich language pairs such as TR- RU or DE-TR, than for similar languages such as IT-FR, with more isomorphic monolingual spaces (Søgaard et al., 2018). To analyze this further, we have run additional experiments on the BLI evaluation sets of Vulic et al. (2019) comprising more typologically distant language pairs<sup>7</sup>, with similar conclusions. For instance,
VECMAP (sup)	VECMAP	CLASSYMAP
TR-HR	.030	.160/.171	.200/.227
DE-TR	.050	.207/.203	.221/.268
TR-FI	.034	.200/.176	.217/.235
TR-RU	.028	.123/.152	.162/203
FI-HR	.049	.249/.195	.252/.278
DE-HR	.058	.229/.206	.246/.268
DE-RU	.111	.193/.208	.212/.239
EN-X	.177	.357/.325	.375/.401
No EN	.089	.310/.286	.322/.353
All	.111	.321/.296	.334/.365
Table 1: P@1 BLI scores for a selection of language pairs. We also perform the average scores over pairs that include English (EN-X) and those that do not (No EN), as well as the averages for all pairs (All). The  $a/b$  score format denotes a score without  $(a)$ , and with the final reranking step  $(b)$ . All improvements of CLASSYMAP with reranking over the strongest baseline (i.e., VECMAP with self-learning) are significant  $(p < 0.05)$  according to the non-parametric shuffling test (Yeh, 2000) with the Bonferroni correction.
with 500 seed pairs CLASSYMAP with reranking scores  $24.6\mathrm{P}@\mathrm{l}$  for Estonian-Esperanto and 16.6 for Hungarian-Basque. The strongest baselines achieve  $\mathrm{P@1}$  of 20.0 and 13.8, respectively. In sum, our classification-based approach holds promise to guide future work especially on distant pairs.
Step Size and the Number of Iterations. We now analyze how two vital components of self-learning impact the final BLI scores: 1) the number of added dictionary entries per iteration (i.e., step size, see Table 3), and 2) the number of iterations (Figure 1). For brevity, we run the analyses on several "difficult" language pairs: DE-RU, TR-FI, HR-FR, and EN-FI. The results suggest that the step size has only moderate impact on the final scores, and is language pair-dependent. However, all three options improve over the baseline self-learning method, and final reranking is again useful across the board. According to Figure 1, the optimal number of iterations is also pair-dependent: TR-FI performance steadily increases over time, while DE-RU hits the peak after only 5 iterations and steadily declines afterwards. This finding calls for a more careful tuning of this parameter in future work.
Feature Ablation Analysis. We also perform an ablation analysis, reported in Table 4. Overall, the results suggest that different features contribute to the final performance. This corroborates our hypothesis that one of the main advantages of the classification-based approach is its ability to fuse different translation evidence. However, there are cases (e.g., using BPE for DE-RU or TR-FI) where
<sup>6</sup>We have also probed a variant where we learn a classifier for the final reranking step on top of VECMAP's output after its self-learning procedure. However, as suggested by the results in Table 1, this leads to drops in performance compared to standard semi-supervised VECMAP. We speculate that this is due to higher levels of noise in the final VECMAP dictionary.
$^{7}$ github.com/cambridgeletl/panlex-bli
6918
500	1k	3k	5k
DE-RU	.111 / .193 / .212 / .239	.232 / .191 / .224 / .249	.301 / .194 / .244 / .277	.303 / .192 / .262 / .290
EN-FI	.081 / .238 / .299 / .350	.219 / .238 / .313 / .363	.320 / .238 / .318 / .362	.352 / .240 / .330 / .370
HR-FR	.053 / .352 / .363 / .411	.178 / .351 / .368 / .406	.325 / .352 / .376 / .420	.353 / .359 / .372 / .417
TR-FI	.034 / .200 / .217 / .235	.111 / .197 / .234 / .249	.213 / .197 / .246 / .266	.242 / .198 / .258 / .274
(a) Without reranking
(b) With reranking
Figure 1: BLI performance  $(\mathrm{P}@\mathrm{1})$  of CLASSYMAP for varying numbers of self learning iterations.
Table 2: Performance for varying initial dictionary sizes (500, 1k, 3k, 5k seed translation pairs). The numbers in each entry delimited with '/' are P@1 scores of 1) supervised VECMAP, 2) VECMAP with self-learning, 3) CLASSYMAP without reranking, and 4) CLASSYMAP with reranking, respectively.
Entries added	DE-RU	TR-FI	HR-FR	EN-FI
18x500	.219/.242	.215/.228	.362/.391	.298/.313
36x250	.220/.244	.217/.227	.363/.416	.295/.312
60x150	.220/.242	.220/.225	.352/.403	.314/.318
Table 3: P@1 BLI scores when varying the number of new dictionary entries added per iteration (i.e., iterations  $\times$  entries). The  $a / b$  score format denotes a score without  $(a)$ , and with the final reranking step  $(b)$ .
Feature Sets	DE-RU	TR-FI	HR-FR	EN-FI
F1 + F7	.232/.182	.202/.147	.335/.223	.268/.172
+ F2	.231/.231	.194/.195	.350/.366	.280/.281
+ F3	.247/.260	.199/.204	.333/.365	.284/.292
+ F6	.244/.249	.191/.186	.348/.377	.280/.292
+ F4 + F5	.258/.255	.205/.211	.344/.376	.306/.301
Table 4: Feature ablation.  $n = 10$ . We experiment with Edit dist. (F1), frequencies (F7), cosine (F2), PCA (F3), BPE (F6), and n-grams (F4 + F5).
a feature set can negatively affect performance. In sum, this small ablation study warrants finer-grained and language pair-dependent feature selection in future work.
Seed Dictionary Size. We also provide additional results when varying the size of the initial seed dictionary in Table 2. The main finding is that, while the absolute BLI scores are naturally higher with larger seed dictionaries, CLASSYMAP remains useful even with much larger dictionary sizes (check the results with 3k and 5k seed pairs). CLASSYMAP with reranking remains the strongest BLI method, corroborating our previous findings.
We introduced CLASSYMAP, a novel classification-based approach to self-learning, which is a crucial component of projection-based cross-lingual word embedding induction models in low-data regimes. We reported its usefulness and robustness across a wide spectrum of diverse language pairs in the BLI task, confirming the usefulness of learning classifiers both as part of the self-learning procedure as well as for the final word retrieval in the BLI task.
This proof-of-concept work opens up a wide spectrum of interesting avenues for future research, including the use of more powerful classifiers, more sophisticated features (e.g., character-level transformers), and fine-grained linguistic analyses on the importance of disparate features over different language pairs. One particularly exciting direction is the application of our classification-based self-learning framework on top of the most recent methods that induce bilingual spaces via non-linear alignments (Glavaš and Vulić, 2020; Mohiuddin and Joty, 2020). The code is available online at: https://github.com/mladenk42/ClassyMap.
IV and AK are supported by the ERC Consolidator Grant LEXICAL (no 648909) awarded to AK. GG is supported by the Eliteprogramm of the BadenWürttemberg Stiftung (AGREE grant). We thank the reviewers for their insightful suggestions.