SlowGuess/NAACL_2021_OCR huggingface dataset:
Haozhou Wang $^{1}$ , James Henderson $^{2}$ , Paola Merlo $^{1}$
<sup>1</sup> University of Geneva
$^{2}$  Idiap Research Institute
{haozhou.wang, paola.merlo} @unige.ch, james.henderson@idiap.ch
Generative adversarial networks (GANs) have succeeded in inducing cross-lingual word embeddings —maps of matching words across languages— without supervision. Despite these successes, GANs' performance for the difficult case of distant languages is still not satisfactory. These limitations have been explained by GANs' incorrect assumption that source and target embedding spaces are related by a single linear mapping and are approximately isomorphic. We assume instead that, especially across distant languages, the mapping is only piece-wise linear, and propose a multi-adversarial learning method. This novel method induces the seed cross-lingual dictionary through multiple mappings, each induced to fit the mapping for one subspace. Our experiments on unsupervised bilingual lexicon induction and cross-lingual document classification show that this method improves performance over previous single-mapping methods, especially for distant languages.
1 Introduction and background
Word embeddings, continuous vectorial representations of words, have become a fundamental initial step in many natural language processing (NLP) tasks for many languages. In recent years, their cross-lingual counterpart, cross-lingual word embeddings (CLWE)—maps of matching words across languages—have been shown to be useful in many important cross-lingual transfer and modeling tasks such as machine translation, cross-lingual document classification and zero-shot dependency parsing (Klementiev et al., 2012; Zou et al., 2013; Guo et al., 2015; Conneau et al., 2018; Glavaš et al., 2019; Zhang et al., 2020).
In these representations, matching words across different languages are represented by similar vectors. Following the observation of Mikolov et al. (2013) that the geometric positions of similar words in two embedding spaces of different languages ap
pear to be related by a linear relation, the most common method aims to map between two pretrained monolingual embedding spaces by learning a single linear transformation matrix. Due to its simple structure design and competitive performance, this approach has become the mainstream of learning CLWE (Glavaš et al., 2019; Vulić et al., 2019; Ruder et al., 2019).
Initially, the linear mapping was learned by minimizing the distances between the source and target words in a seed dictionary. Early work from Mikolov et al. (2013) uses a seed dictionary of five-thousand word pairs. Since then, the size of the seed dictionary has been gradually reduced, from several-thousand to fifty word pairs (Smith et al., 2017), reaching a minimal version of only sharing numerals (Artetxe et al., 2017).
More recent works on unsupervised learning have shown that mappings across embedding spaces can also be learned without any bilingual evidence (Barone, 2016; Zhang et al., 2017; Conneau et al., 2018; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018; Artetxe et al., 2018). More concretely, these fully unsupervised methods usually consist of two main steps (Hartmann et al., 2019): an unsupervised step which aims to induce the seed dictionary by matching the source and target distributions, and then a pseudo-supervised refinement step based on this seed dictionary.
The system proposed by Conneau et al. (2018) can be considered the first successful unsupervised system for learning CLWE. They first use generative adversarial networks (GANs) to learn a single linear mapping to induce the seed dictionary, followed by the Procrustes Analysis (Schönemann, 1966) to refine the linear mapping based on the induced seed dictionary. While this GAN-based model has competitive or even better performance compared to supervised methods on typologically-similar language pairs, it often exhibits poor performance on typologically-distant language pairs,
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 463-472 June 6-11, 2021. ©2021 Association for Computational Linguistics
pairs of languages that differ drastically in word forms, morphology, word order and other properties that determine how similar the lexicon of a language is. More specifically, their initial linear mapping often fails to induce the seed dictionary for distant language pairs (Vulić et al., 2019). Later work from Artetxe et al. (2018) has proposed an unsupervised self-learning framework to make the unsupervised CLWE learning more robust. Their system uses similarity distribution matching to induce the seed dictionary and stochastic dictionary induction to refine the mapping iteratively. The final CLWE learned by their system performs better than the GAN-based system. However, their advantage appears to come from the iterative refinement with stochastic dictionary induction, according to Hartmann et al. (2019). If we only consider the performance of a model induced only with distribution matching, GAN-based models perform much better. This brings us to our first conclusions, that a GAN-based model is preferable for seed dictionary induction.
Fully unsupervised mapping-based methods to learn CLWE rely on the strong assumption that monolingual word embedding spaces are isomorphic or near-isomorphic, but this assumption is not fulfilled in practice, especially for distant language pairs (Søgaard et al., 2018). Experiments by Vulic et al. (2020) also demonstrate that the lack of isomorphism does not arise only because of the typological distance among languages, but it also depends on the quality of the monolingual embedding space. If we replace the seed dictionary learned by an unsupervised distribution matching method with a pretrained dictionary, keeping constant the refinement technique, the final system becomes more robust (Vulic et al., 2019).
All these previous results indicate that learning a better seed dictionary is a crucial step to improve unsupervised cross-lingual word embedding induction and reduce the gap between unsupervised methods and supervised methods, and that GAN-based methods hold the most promise to achieve this goal. The results also indicate that a solution that can handle the full complexity of induction of cross-lingual word embeddings will show improvements in both close and distant languages.
In this paper, we focus on improving the initial step of distribution matching, using GANs (Hartmann et al., 2019). Because the isomorphism assumption is not observed in reality, we argue that a
Figure 1: Translation accuracy from English to Chinese and to French for different English subspaces. We only include the top fifty-thousand most frequent English words in the pretrained fastText embeddings. The gold translations come from Google Translate.
successful GAN-based model must not learn only one single linear mapping for the entire distribution, but must be able to identify mapping subspaces and learn multiple mappings. We propose a multi-adversarial learning method which learns different linear maps for different subspaces of word embeddings.
2 Limitations of single-linear mappings
If the assumption by Mikolov et al. (2013) that similar words across source and target languages are related by a single linear relation holds exactly or even approximately, the distance between source and target embedding spaces should be (nearly) evenly minimized during the training of the initial mapping. More specifically, each source subspace should be mapped (nearly) equally well to its corresponding target space, so that the translation ability of the single linear mapping should be similar across different source subspaces.
To verify this expectation, we use the GAN-based system MUSE to train two linear mappings (without refinement) (Conneau et al., 2018). One mapping relates two typologically distant languages, English and Chinese, and the other maps the English space to the space of French - a typologically similar language. We use pretrained FastText embeddings. We split the English space into ten subspaces by running  $k$ -means clustering. We evaluate the trained linear mappings by calculating the translation accuracy with precision at one  $(\mathbb{P}@\mathbb{1})$  how often the highest ranked translation is the correct one for each subspace, using the translations from Google Translate as the gold dataset.
$^{1}$ https://github.com/facebookresearch/MUSE
$^{2}$ https://fasttext.cc/docs/en/pretrained-vectors.html
To reduce the influence of infrequent words, we only consider the first fifty-thousand most frequent source words.
As we can see in Figure 1, the distribution of accuracies of different subspaces is not uniform or even nearly so. This is true for both language pairs, but particularly for the distant languages, where the general mapping does not work at all in some subspaces. Similar phenomena were also discovered by Nakashole (2018) where source words are grouped into different categories. This lack of uniformity in results corroborates the appropriateness of designing a model that learns different linear mappings for different subspaces instead of only learning a single linear mapping for the entire source space.
3 Multi-adversarial CLWE learning
To learn different mappings for different source subspaces, we propose a method for training one GAN for each source subspace. These multi-discriminator GANs encourage the distribution of mapped word embeddings from a specific source subspace to match the distribution of word embeddings from the corresponding target subspace.
The first step of our proposed method is to train a single linear mapping, as in previous approaches. This is used to find aligned subspaces. Our proposed multi-discriminator GAN model then learns the multi-linear mapping. This section starts with the two GAN models, followed by the subspace alignment method, and then describes methods used to improve the GAN training.
3.1 Unsupervised CLWE learning
We first define the task of learning CLWEs and the role of GANs in the previous work of Conneau et al. (2018). Let two monolingual word embeddings  $V_{s}^{j} = \{v_{s_{1}},\dots,v_{s_{j}}\}$  and  $V_{t}^{k} = \{v_{t_{1}},\dots,v_{t_{k}}\}$  be given. In previous work, mapping  $V_{s}^{j}$  to  $V_{t}^{k}$  means seeking a linear transformation matrix  $W$ , so that the projected vector  $Wv_{i}$  of a source word is close to the vector of its translation in the target language. The basic idea underlying supervised methods is using a seed dictionary of  $n$  word pairs  $\{(w_{s_1},w_{t_1}),\dots,(w_{s_n},w_{t_n})\}$  to learn the matrix  $W$  by minimizing the distance in (1), where  $v_{s_i}$  and  $v_{t_i}$  represent the embeddings of  $w_{s_i}$  and  $w_{t_i}$ . The trained matrix  $W$  can then be used to map
the source word embeddings to the target space.
$$
\min  _ {W} \sum_ {i = 1} ^ {n} \| W v _ {s _ {i}} - v _ {t _ {i}} \| ^ {2} \tag {1}
$$
In an unsupervised setting, the seed dictionary is not provided. Conneau et al. (2018) propose a two-step system where the seed dictionary is learned in an unsupervised fashion. In a first step, they use GANs to learn an initial linear transformation matrix  $W$  and use this to induce a seed dictionary by finding the translations of the first ten-thousand most frequent source words. In a second step, the seed dictionary is used to refine the initial matrix  $W$ . In this work we focus on the GAN component of this model.
3.2 GAN learning of a single linear mapping
Previous GAN-based systems learn a single linear mapping from the source embedding space to the target embedding space. In such models, a source word is trained against a target word sampled from the whole target distribution, and the resulting single linear mapping is applied to all the source words. We first introduce the basic GAN architecture for CLWE of Conneau et al. (2018). We use this model as our comparative baseline and as the initial stage of our proposed method.
A standard GAN model plays a min-max game between a generator  $G$  and a discriminator  $D$  (Goodfellow et al., 2014). The generator learns from the distribution of source data and tries to fool the discriminator by generating new samples which are similar to the target data.
When we adapt the basic GAN model to learning CLWE, the goal of the generator is to learn the linear mapping matrix  $W$ . The discriminator  $D$  detects whether the input is from the distribution of target embeddings  $p_{v_t}$ . Conneau et al. (2018) use the loss functions in (2) and (3) to update the discriminator and the generator, respectively.  $G(v_s) = W v_s$ , and  $D(v)$  denotes the probability that the input vector  $v$  came from the target distribution  $p_{v_t}$  rather than the generator applied to samples from the source distribution  $p_{v_s}$ .
$$
l _ {D} = - \log D (v _ {t}) - \log (1 - D (G (v _ {s}))) \tag {2}
$$
$$
l _ {G} = - \log D (G (v _ {s})) - \log (1 - D (v _ {t})) \tag {3}
$$
The parameters of both generator and discriminator are updated alternatively by using stochastic gradient descent. However, a number of additional methods are needed for robust reliable training of such GANs, which are discussed in Section 3.5.
Figure 2: Architecture of our multi-discriminator model. The generator  $G^{i}$  for each source subspace  $V_{s}^{i}$  is trained against the discriminator  $D_{s}^{i}$  for the aligned subspace  $V_{t}^{i}$  and a whole-language discriminator  $D_{l}^{i}$ .
3.3 GAN learning of a multi-linear mapping
Unlike previous work, we propose learning different linear mappings for different source subspaces. We propose a multi-discriminator GAN where a source word from one subspace is trained against a target word sampled from the aligned target subspace.
For each subspace of source embeddings, we propose a multi-discriminator adversarial model to train the specific mapping for vectors that belong to this subspace. As the architecture in Figure 2 illustrates, the generator of the given source subspace  $i$  takes the vector sampled from the sub-distribution as input and maps it to the target language. Differently from standard GANs, the mapped vector  $G^{i}(v_{s}^{i}) = W^{i}v_{s}^{i}$  will be fed into two discriminators. First, a subspace-specific discriminator  $D_{s}^{i}$  judges whether the vector has come from the correspondent target subspace  $i$ . Thus, we use the vectors sampled from both source and target subspaces to train  $D_{s}^{i}$ . Second, a normal language discriminator  $D_{l}^{i}$  judges whether the vector has come from the whole target distribution. This language discriminator helps avoid local optima for the specific subspace.
Both discriminators are two-layer perceptron classifiers. Except for the different sampling ranges, their loss function is similar to equations (2) and (3):
$$
l _ {D _ {l} ^ {i}} = - \log D \left(v _ {t} ^ {\prime}\right) - \log \left(1 - D \left(G \left(v _ {s} ^ {\prime}\right)\right)\right) \tag {4}
$$
$$
l _ {D _ {s} ^ {i}} = - \log D \left(v _ {t} ^ {i}\right) - \log \left(1 - D \left(G \left(v _ {s} ^ {i}\right)\right)\right) \tag {5}
$$
where  $v_{s}^{\prime}$  and  $v_{t}^{\prime}$  are sampled from the 75-thousand most frequent source and target words, and  $v_{s}^{i}$  and  $v_{t}^{i}$  are sampled from the specific source subspace  $V_{s}^{i}$  and its corresponding target subspace  $V_{t}^{i}$ .
Since the outputs of both discriminators are used for training the generator, the loss function of the subspace-specific generator  $G^{i}$  can be written as:
$$
\begin{array}{l} l _ {G ^ {i}} = - \lambda \left(\log D _ {l} ^ {i} \left(G ^ {i} \left(v _ {s} ^ {i}\right)\right) + \log \left(1 - D _ {l} ^ {i} \left(v _ {t} ^ {\prime}\right)\right)\right) \\ - (1 - \lambda) \left(\log D _ {s} ^ {i} \left(G ^ {i} \left(v _ {s} ^ {i}\right)\right) + \log \left(1 - D _ {s} ^ {i} \left(v _ {t} ^ {i}\right)\right)\right) \tag {6} \\ \end{array}
$$
where  $\lambda$  is a coefficient that we call global confidence, which balances the contributions of the two discriminators in updating the generator. In practice, we find that setting  $\lambda$  to 0.5 for each subspace works well for the final result.
Additionally, as the similarities between the entire distribution and the distribution of different subspaces are different, it is justified to use different lambdas for different subspaces instead of using a single one. We therefore propose a metric to set  $\lambda$  dynamically, based on the proportion of the eigenvalue divergence between the two subspaces and the eigenvalue divergence between the whole source and target distributions, as shown in (7). In this paper, we only report results with dynamic  $\lambda$ .
$$
\lambda = \frac {E V D \left(V _ {s} ^ {i} , V _ {t} ^ {i}\right)}{E V D \left(V _ {s} , V _ {t}\right)} \tag {7}
$$
The eigenvalue divergence between two embedding distributions  $V_{1}$  and  $V_{2}$  can be computed as shown in (8), where  $e_k^{V_1}$  and  $e_k^{V_2}$  represent the eigenvalues of  $V_{1}$  and  $V_{2}$ .
$$
E V D \left(V _ {1}, V _ {2}\right) = \sum_ {k = 1} ^ {d} \left(\log e _ {k} ^ {V _ {1}} - \log e _ {k} ^ {V _ {2}}\right) ^ {2} \tag {8}
$$
All subspace-specific generators are initialized with the single linear mapping discussed in Section 3.2.
3.4 Subspace alignment
The above multi-discriminator GAN assumes that we have an alignment between source subspaces and target subspaces. We first present the method we use to produce aligned subspaces in both source and target distributions, and then the clustering method we use to find coherent subspaces, which are both important for the model's improved performance.
If we want to encourage words from a specific source subspace to be trained against words from a matching target subspace, we need to align the two cross-language subspaces. The second problem we need to solve for our multi-adversarial method
3We use different language discriminator models  $D_l^i$  for each subspace  $i$ , even though their training samples all come from the same distributions. This leads to more stable training, presumably because initially these language discriminators are randomly different.
to work is how to discover this alignment. Although metrics such as Gromov-Hausdorff distance (GH) (Patra et al., 2019) and Eigenvalue Divergence (EVD) (Dubossarsky et al., 2020) can be used to measure the similarity between two distributions and find the most similar target subspace for a given source subspace, matching between two sub-distributions may amplify any bias generated during the clustering.
To avoid this problem, we only run the clustering on the source side. For a given source embedding space  $V_{s}$ , we denote its subspaces after clustering as  $\{V_s^1, V_s^2, \dots, V_s^i, \dots, V_s^n\}$ , where  $n$  represents the number of subspaces. To align target words to their matching source subspace, we propose to first learn a single linear mapping from source to target space using the GAN-based method (without refinement) described previously, and then use the transpose of this linear mapping to retrieve the translation of each target word in the source language (using cross-domain similarity local scaling, defined below in Section 3.5). The subspace index of the target word is then set to the subspace index of this translation. In this way, the target embedding space  $V_{t}$  is partitioned into as many subspaces as the source embedding space, denoted as  $\{V_t^1, V_t^2, \dots, V_t^i, \dots, V_t^n\}$ . This gives us aligned subspace pairs  $(V_s^i, V_t^i)$ .
Although the single linear mapping from source language to target language is not good enough to get accurate translations, our experiments indicate that it is a good method to produce a subspace alignment. A possible reason for this result is that the clustering on the source language has already grouped similar words. Therefore, even if a translation turns out to be incorrect, it usually has the same subspace index as the best translation.
Parameter-free hierarchical clustering A major issue in clustering an embedding space is how to find a clustering that adapts to the space, without fixed parameters. To avoid having to identify the number of subspaces in advance, we use hierarchical clustering. Recent work proposes a parameter-free method called First Integer Neighbor Clustering Hierarchy (FINCH) (Sarfraz et al., 2019), which we use in this paper. Traditionally, clustering methods split a given space of vectors into different clusters by calculating the distances between the centroid and the other vectors. FINCH is developed based on the observation that the first neighbour of each vector is a sufficient statistic to
find links in the space, so that computing the distance matrix between all the vectors is not needed (Sarfraz et al., 2019). For a given vector space, one first computes an adjacency link matrix using the equation in (9).
$$
A (i, j) = \left\{ \begin{array}{l l} 1 & i f j = \kappa_ {i} ^ {1} o r \kappa_ {j} ^ {1} = i o r \kappa_ {i} ^ {1} = \kappa_ {j} ^ {1} \\ 0 & o t h e r w i s e \end{array} \right. \tag {9}
$$
where  $i, j$  denote the indices of vectors and  $\kappa_i^1$  represents the index of the first neighbour of the vector with index  $i$ . The connected components can then be detected from the adjacency matrix  $A$  by building a directed or undirected graph on  $A$ . No parameter needs to be set. When the clustering on the original first level (original data) is completed, the centroid of each cluster can then be considered as a data vector for the next level and a new level of clustering is computed using the same procedure. In theory, all the vectors will eventually be gathered into a single cluster. In practice, we find that using the clusters of the last level or the second-to-last level works well for our system.
3.5 Training the GANs
Training the GANs described in Sections 3.2 and 3.3 can be challenging. Based on previous work and our experience, we employ the following techniques during training.
Orthogonalization Previous work shows that enforcing the mapping matrix  $W$  to be orthogonal during the training can improve the performance (Smith et al., 2017). In the system of Conneau et al. (2018), they follow the work of Cisse et al. (2017) and approximate setting  $W$  to an orthogonal matrix with  $W \gets (1 + \beta)W - \beta (WW^{\top})W$ . This orthogonalization usually performs well when setting  $\beta$  to 0.001 (Conneau et al., 2018; Wang et al., 2019).
Cross-Domain Similarity Local Scaling The trained mapping matrix  $W$  can be used for retrieving the translation for a given source word  $w_{s}$  by searching a target word  $w_{t}$  whose embedding vector  $v_{t}$  is close to  $Wv_{s}$ . But Conneau et al. (2018) showed that using cross-domain similarity local scaling (CSLS) to retrieve translations is more accurate than standard nearest neighbor techniques and can reduce the impact of the hubs problem
4In the code of Sarfraz et al. (2019), the last level means the level before grouping all the data vectors into a single cluster.
(Radovanović et al., 2010; Dinu et al., 2015). Instead of just considering the distance between  $Wv_{s}$  and  $v_{t}$ , CSLS also takes into account the neighbours of  $v_{t}$  in the source language by minimising  $(2\cos (Wv_{s},v_{t}) - r_{t}(Wv_{s}) - r_{s}(v_{t}))$ , where  $r_t(Wv_s)$  denotes the mean similarity between a  $Wv_{s}$  and its neighbours in the target language, while  $r_s(v_t)$  represents the mean similarity between  $v_{t}$  and its neighbours in the source language.
Model selection criterion The cosine-based model selection criterion is another important component of adversarial training for selecting the best mapping matrix  $W$ . More specifically, at the end of each training epoch, the current mapping is used to translate the ten-thousand most frequent source words into target words and calculate the average cosine similarity between the source vectors and the target vectors. This cosine-based criterion has been shown to correlate well with the quality of  $W$  (Conneau et al., 2018; Hartmann et al., 2019).