SlowGuess/NAACL_2021_OCR huggingface dataset:
Cross-Lingual Word Embedding Refinement by  $\ell_1$  Norm Optimisation
Xutan Peng Chenghua Lin* Mark Stevenson  
Department of Computer Science, The University of Sheffield, UK  
{x.peng, c.lin, mark.stevenson}@shef.ac.uk
Cross-Linguual Word Embeddings (CLWEs) encode words from two or more languages in a shared high-dimensional space in which vectors representing words with similar meaning (regardless of language) are closely located. Existing methods for building high-quality CLWEs learn mappings that minimise the  $\ell_2$  norm loss function. However, this optimisation objective has been demonstrated to be sensitive to outliers. Based on the more robust Manhattan norm (aka.  $\ell_1$  norm) goodness-of-fit criterion, this paper proposes a simple post-processing step to improve CLWEs. An advantage of this approach is that it is fully agnostic to the training process of the original CLWEs and can therefore be applied widely. Extensive experiments are performed involving ten diverse languages and embeddings trained on different corpora. Evaluation results based on bilingual lexicon induction and crosslingual transfer for natural language inference tasks show that the  $\ell_1$  refinement substantially outperforms four state-of-the-art baselines in both supervised and unsupervised settings. It is therefore recommended that this strategy be adopted as a standard for CLWE methods.
Cross-Linguual Word Embedding (CLWE) techniques have recently received significant attention as an effective means to support Natural Language Processing applications for low-resource languages, e.g., machine translation (Artetxe et al., 2018b) and transfer learning (Peng et al., 2021).
The most successful CLWE models are the so-called projection-based methods, which learn mappings between monolingual word vectors with very little, or even zero, cross-lingual supervision (Lample et al., 2018; Artetxe et al., 2018a; Glavaš et al., 2019). Mainstream projection-based CLWE models typically identify orthogonal mappings by
minimising the topological dissimilarity between source and target embeddings based on  $\ell_2$  loss (aka. Frobenius loss or squared error) (Glavaš et al., 2019; Ruder et al., 2019). This learning strategy has two advantages. First, adding the orthogonality constraint to the mapping function has been demonstrated to significantly enhance the quality of CLWEs (Xing et al., 2015). Second, the existence of a closed-form solution to the  $\ell_2$  optima (Schönemann, 1966) greatly simplifies the computation required (Artetxe et al., 2016; Smith et al., 2017).
Despite its popularity, work in various application domains has noted that  $\ell_2$  loss is not robust to noise and outliers. It is widely known in computer vision that  $\ell_2$ -loss-based solutions can severely exaggerate noise, leading to inaccurate estimates (Aanæs et al., 2002; De La Torre and Black, 2003). In data mining, Principal Component Analysis (PCA) using  $\ell_2$  loss has been shown to be sensitive to the presence of outliers in the input data, degrading the quality of the feature space produced (Kwak, 2008). Previous studies have demonstrated that the processes used to construct monolingual and cross-lingual embeddings may introduce noise (e.g. via reconstruction error (Allen and Hospedales, 2019) and structural variance (Ruder et al., 2019)), making the presence of outliers more likely. Empirical analysis of CLWEs also demonstrates that more distant word pairs (which are more likely to be outliers) have more influence on the behaviour of  $\ell_2$  loss than closer pairs. This raises the question of the appropriateness of  $\ell_2$  loss functions for CLWEs.
Compared to the conventional  $\ell_2$  loss,  $\ell_1$  loss (aka. Manhattan distance) has been mathematically demonstrated to be less affected by outliers (Rousseeuw and Leroy, 1987) and empirically proven useful in computer vision and data mining (Aanas et al., 2002; De La Torre and Black, 2003; Kwak, 2008). Motivated by this insight, our paper proposes a simple yet effective post
*Chenghua Lin is the corresponding author.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2690-2701 June 6-11, 2021. ©2021 Association for Computational Linguistics
processing technique to improve the quality of CLWEs: adjust the alignment of any cross-lingual vector space to minimise the  $\ell_1$  loss without violating the orthogonality constraint. Specifically, given existing CLWEs, we bidirectionally retrieve bilingual vectors and optimise their Manhattan distance using a numerical solver. The approach can be applied to any CLWEs, making the post-hoc refinement technique generic and applicable to a wide range of scenarios. We believe this to be the first application of  $\ell_1$  loss to the CLWE problem.
To demonstrate the effectiveness of our method, we select four state-of-the-art baselines and conduct comprehensive evaluations in both supervised and unsupervised settings. Our experiments involve ten languages from diverse branches/families and embeddings trained on corpora of different domains. In addition to the standard Bilingual Lexicon Induction (BLI) benchmark, we also investigate a downstream task, namely cross-lingual transfer for Natural Language Inference (NLI). In all setups tested, our algorithm significantly improves the performance of strong baselines. Finally, we provide an intuitive visualisation illustrating why  $\ell_1$  loss is more robust than it  $\ell_2$  counterpart when refining CLWEs (see Fig. 1). Our code is available at https://github.com/Pzoom522/L1-Refinement.
Our contribution is three-fold: (1) we propose a robust refinement technique based on the  $\ell_1$  norm training objective, which can effectively enhance CLWEs; (2) our approach is generic and can be directly coupled with both supervised and unsupervised CLWE models; (3) our  $\ell_1$  refinement algorithm achieves state-of-the-art performance for both BLI and cross-lingual transfer for NLI tasks.
CLWE methods. One approach to generating CLWEs is to train shared semantic representations using multilingual texts aligned at sentence or document level (Vulic and Korhonen, 2016; Upadhyay et al., 2016). Although this research direction has been well studied, the parallel setup requirement for model training is expensive, and hence impractical for low-resource languages.
Recent years have seen an increase in interest in projection-based methods, which train CLWEs by finding mappings between pretrained word vectors of different languages (Mikolov et al., 2013; Lample et al., 2018; Peng et al., 2020). Since the input
embeddings can be generated independently using monolingual corpora only, projection-based methods reduce the supervision required for training and offer a viable solution for low-resource scenarios.
Xing et al. (2015) showed that the precision of the learned CLWEs can be improved by constraining the mapping function to be orthogonal, which is formalised as the so-called  $\ell_2$  Orthogonal Procrustes Analysis (OPA):
$$
\underset {\mathbf {M} \in \mathcal {O}} {\operatorname {a r g m i n}} \| \mathbf {A M} - \mathbf {B} \| _ {2}, \tag {1}
$$
where  $\mathbf{M}$  is the CLWE mapping,  $\mathcal{O}$  denotes the orthogonal manifold (aka. the Stiefel manifold (Chu and Trendafilov, 2001)), and  $\mathbf{A}$  and  $\mathbf{B}$  are matrices composed using vectors from source and target embedding spaces.
While Xing et al. (2015) exploited an approximate and relatively slow gradient-based solver, more recent approaches such as Artetxe et al. (2016) and Smith et al. (2017) introduced an exact closed-form solution for Eq. (1). Originally proposed by Schonemann (1966), it utilises Singular Value Decomposition (SVD):
$$
\mathbf {M} ^ {\star} = \mathbf {U V} ^ {\top}, \text {w i t h} \mathbf {U} \Sigma \mathbf {V} ^ {\top} = \operatorname {S V D} (\mathbf {A} ^ {\top} \mathbf {B}), \tag {2}
$$
where  $\mathbf{M}^{\star}$  denotes the  $\ell_2$ -optimal mapping matrix. The efficiency and effectiveness of Eq. (2) have led to its application within many other approaches, e.g., Ruder et al. (2018), Joulin et al. (2018) and Glavaš et al. (2019). In particular, PROC-B (Glavaš et al., 2019), a supervised CLWE framework that simply applies multiple iterations of  $\ell_2$  OPA, has been demonstrated to produce very competitive performance on various benchmark tasks including BLI as well as cross-lingual transfer for NLI and information retrieval.
While the aforementioned approaches still require some weak supervision (i.e., seed dictionaries), there have also been some successful attempts to train CLWEs in a completely unsupervised fashion. For instance, Lample et al. (2018) proposed a system called MUSE, which bootstraps CLWEs without any bilingual signal through adversarial learning. VECMAP (Artetxe et al., 2018a) applied a self-learning strategy to iteratively compute the optimal mapping and then retrieve bilingual dictionary. Comparing MUSE and VECMAP, the latter tends to be more robust as its similarity-matrix-based heuristic initialisation is more stable in most cases (Glavaš et al., 2019; Ruder et al., 2019). Very
recently, some studies bootstrapped unsupervised CLWEs by jointly training word embeddings on concatenated corpora of different languages and achieved good performance (Wang et al., 2020).
The  $\ell_2$  refinement algorithm. CLWE models often apply  $\ell_2$  refinement, a post-processing step shown to improve the quality of the initial alignment (see Ruder et al. (2019) for survey). Given existing CLWEs  $\{\mathbf{X}_{\mathrm{LA}},\mathbf{X}_{\mathrm{LB}}\}$  for languages  $\mathrm{L_A}$  and  $\mathrm{L_B}$ , bidirectionally one can use approaches such as the classic nearest-neighbour algorithm, the inverted softmax (Smith et al., 2017) and the cross-domain similarity local scaling (CSLS) (Lample et al., 2018) to retrieve two bilingual dictionaries  $D_{\mathrm{LA} \mapsto \mathrm{LB}}$  and  $D_{\mathrm{LB} \mapsto \mathrm{LA}}$ . Note that word pairs in  $D_{\mathrm{LA} \mapsto \mathrm{LB}} \cap D_{\mathrm{LB} \mapsto \mathrm{LA}}$  are highly reliable, as they form "mutual translations". Next, one can compose bilingual embedding matrices  $\mathbf{A}$  and  $\mathbf{B}$  by aligning word vectors (rows) using the above word pairs. Finally, a new orthogonal mapping is learned to fit  $\mathbf{A}$  and  $\mathbf{B}$  based on least-square regressions, i.e., perform  $\ell_2$  OPA described in Eq. (1).
Early applications of  $\ell_2$  refinement applied a single iteration, e.g. (Vulic and Korhonen, 2016). Due to the wide adoption of the closed-form  $\ell_2$  OPA solution (cf. Eq. (2)), recent methods perform multiple iterations. The iterative  $\ell_2$  refinement strategy is an important component of approaches that bootstrap from small or null training lexicons (Artetxe et al., 2018a). However, a single step of refinement is often sufficient to create suitable CLWEs (Lample et al., 2018; Glavaš et al., 2019).
A common characteristic of CLWE methods that apply the orthogonality constraint is that they optimise using  $\ell_2$  loss (see § 2). However, outliers have disproportionate influence in  $\ell_2$  since the penalty increases quadratically and this can be particularly problematic with noisy data since the solution can "shift" towards them (Rousseeuw and Leroy, 1987). The noise and outliers present in real-world word embeddings may affect the performance of  $\ell_2$ -loss-based CLWEs.
The  $\ell_1$  norm cost function is more robust than  $\ell_2$  loss as it is less affected by outliers (Rousseeuw and Leroy, 1987). Therefore, we propose a refinement algorithm for improving the quality of CLWEs based on  $\ell_1$  loss. This novel method, which we refer to as  $\ell_1$  refinement, is generic and can be applied post-hoc to improve the output of existing
CLWE models. To our knowledge, the use of alternatives to  $\ell_2$ -loss-based optimisation has never been explored by the CLWE community.
To begin with, analogous to  $\ell_2$  OPA (cf. Eq. (1)),  $\ell_1$  OPA can be formally defined and rewritten as
$$
\begin{array}{l} \operatorname *{argmin}_{\mathbf{M}\in \mathcal{O}}\| \mathbf{AM} - \mathbf{B}\|_{1} \\ = \underset {\mathbf {M} \in \mathcal {O}} {\operatorname {a r g m i n}} \operatorname {t r} \left[ \left(\mathbf {A M} - \mathbf {B}\right) ^ {\top} \operatorname {s g n} (\mathbf {A M} - \mathbf {B}) \right], \tag {3} \\ \end{array}
$$
where  $\operatorname{tr}(\cdot)$  returns the matrix trace,  $\operatorname{sgn}(\cdot)$  is the signum function, and  $\in \mathcal{O}$  denotes that  $M$  is subject to the orthogonal constraint. Compared to  $\ell_2$  OPA which has a closed-form solution, solving Eq. (3) is much more challenging due to the discontinuity of  $\operatorname{sgn}(\cdot)$ . This issue can be addressed by replacing  $\operatorname{sgn}(\cdot)$  with  $\tanh (\alpha (\cdot))$ , a smoothing function parameterised by  $\alpha$ , such that
$$
\underset {\mathbf {M} \in \mathcal {O}} {\operatorname {a r g m i n}} \operatorname {t r} \left[ (\mathbf {A M} - \mathbf {B}) ^ {\intercal} \tanh  \left(\alpha (\mathbf {A M} - \mathbf {B})\right) \right]. \tag {4}
$$
Larger values for  $\alpha$  lead to closer approximations to  $\operatorname{sgn}(\cdot)$  but reduce the smoothing effect. This approach has been used in many applications, such as the activation function of long short-term memory networks (Hochreiter and Schmidhuber, 1997).
However, in practice, we find that Eq. (4) remains unsolvable in our case with standard gradient-based frameworks for two reasons. First,  $\alpha$  has to be sufficiently large in order to achieve a good approximation of  $\mathrm{sgn}(\cdot)$ . Otherwise, relatively small residuals will be down-weighted during fitting and the objective will become biased towards outliers, just similar to  $\ell_2$  loss. However, satisfying this requirement (i.e., large  $\alpha$ ) will lead to the activation function  $\tanh (\alpha (\cdot))$  becoming easily saturated, resulting in an optimisation process that becomes trapped during the early stages. In other words, the optimisation can only reach an unsatisfactory local optimum. Second, the orthogonality constraint (i.e.,  $\mathbf{M}\in \mathcal{O}$ ) also makes the optimisation more problematic for these methods.
We address these challenges by adopting the approaches proposed by Trendafilov (2003). This method explicitly encourages the solver to only explore the desired manifold  $\mathcal{O}$  thereby reducing the  $\ell_1$  solver's search space and difficulty of the optimisation problem. We begin by calculating the gradient  $\nabla$  w.r.t. the objective in Eq. (4) through matrix differentiation:
$$
\nabla = \mathbf {A} ^ {\intercal} (\tanh  (\mathbf {Z}) + \mathbf {Z} \odot \cosh^ {- 2} (\mathbf {Z})), \tag {5}
$$
where  $\mathbf{Z} = \alpha (\mathbf{AM} - \mathbf{B})$  and  $\odot$  is the Hadamard product. Next, to find the steepest descent direction while ensuring that any  $\mathbf{M}$  produced is orthogonal, we project  $\nabla$  onto  $\mathcal{O}$ , yielding<sup>1</sup>
$$
\pi_ {\mathcal {O}} (\nabla) := \frac {1}{2} \mathbf {M} \left(\mathbf {M} ^ {\intercal} \nabla - \nabla^ {\intercal} \mathbf {M}\right) + \left(\mathbf {I} - \mathbf {M} \mathbf {M} ^ {\intercal}\right) \nabla . \tag {6}
$$
Here  $\mathbf{I}$  is an identity matrix with the shape of  $\mathbf{M}$ . With Eq. (6) defining the optimisation flow, our  $\ell_1$  loss minimisation problem reduces to an integration problem, as
$$
\mathbf {M} ^ {\star} = \mathbf {M} _ {\mathbf {0}} + \int - \pi_ {\mathcal {O}} (\nabla) \mathrm {d} t, \tag {7}
$$
where  $\mathbf{M_0}$  is a proper initial solution of Eq. (3) (e.g.,  $\ell_{2}$ -optimal mapping obtained via Eq. (2)).
Empirically, unlike the aforementioned standard gradient-based methods, by following the established policy of Eq. (6), the optimisation process of Eq. (7) will not violate the orthogonality restriction or get trapped during early stages. However, this  $\ell_{1}$  OPA solver requires extremely small step size to generate reliable solutions (Trendafilov, 2003), making it computationally expensive $^{2}$ . Therefore, it is impractical to perform  $\ell_{1}$  refinement in an iterative fashion like  $\ell_{2}$  refinement without significant computational resources.
Previous work has demonstrated that applying the  $\ell_1$ -loss-based algorithms from a good initial state can speed up the optimisation. For instance, Kwak (2008) found that feature spaces created by  $\ell_2$  PCA were severely affected by noise. Replacing the cost function with  $\ell_1$  loss significantly reduced this problem, but required expensive linear programming. To reduce the convergence time, Brooks and Jot (2013) exploited the first principal component from the  $\ell_2$  solution as an initial guess. Similarly, when reconstructing corrupted pixel matrices,  $\ell_2$ -loss-based results are far from satisfactory; using  $\ell_1$  norm estimators can improve the quality, but are too slow to handle large-scale datasets (Aanæs et al., 2002). However, taking the  $\ell_2$  optima as the starting point allowed less biased reconstructions to be learned in an acceptable time (De La Torre and Black, 2003).
Inspired by these works, we make use of  $\ell_1$  refinement to carry out post-hoc enhancement of existing CLWEs. Our full pipeline is described in
Algorithm 1  $\ell_1$  refinement
Input: CLWEs  $\{\mathbf{X}_{\mathrm{L}_A},\mathbf{X}_{\mathrm{L_B}}\}$
Output: updated CLWEs  $\{\mathbf{X}_{\mathrm{L_A}}\mathbf{M}^\star ,\mathbf{X}_{\mathrm{L_B}}\}$
$D_{\mathrm{LA}\mapsto \mathrm{LB}}\gets$  build dict via  $\mathbf{X}_{\mathrm{LA}}$  and  $\mathbf{X}_{\mathrm{LB}}$
2:  $D_{\mathrm{L_B} \leftrightarrow \mathrm{L_A}} \gets$  build dict via  $\mathbf{X}_{\mathrm{L_B}}$  and  $\mathbf{X}_{\mathrm{L_A}}$
3:  $D\gets D_{\mathrm{LA}\mapsto \mathrm{LB}}\cap D_{\mathrm{LB}\mapsto \mathrm{LA}}$
4: A, B  $\leftarrow$  looks up for  $D$  in  $\mathbf{X}_{\mathrm{L}_A}$ ,  $\mathbf{X}_{\mathrm{L}_B}$
5: perform integration to solve Eq. (7) for  $\mathbf{M}^{\star}$ , with initial value  $\mathbf{M}_0\gets \mathbf{I}$ , until stopping criteria are met
Algorithm 1 (see § 4.3 for implemented configurations). In common with  $\ell_2$  refinement (cf. § 2), steps 1-4 bootstrap a synthetic dictionary  $D$  and compose bilingual word vector matrices  $\mathbf{A}$  and  $\mathbf{B}$  which have reliable row-wise correspondence. Taking them as the starting state, in step 5 an identity matrix naturally serves as our initial solution  $\mathbf{M}_0$ .
During the execution of Eq. (7), we record  $\ell_1$  loss per iteration and see if either of the following two stopping criteria have been satisfied: (1) the updated  $\ell_1$  loss exceeds that of the previous iteration; (2) on-the-fly  $\mathbf{M}$  has non-negligibly departed from the orthogonal manifold, which can be indicated by the maximum value of the disparity matrix as
$$
\max  \left(\left| \mathbf {M} ^ {\mathrm {T}} \mathbf {M} - \mathbf {I} \right|\right) > \epsilon , \tag {8}
$$
where  $\epsilon$  is a sufficiently small threshold. The resulting  $\mathbf{M}^{\star}$  can be used to adjust the word vectors of  $\mathrm{L}_{\mathrm{A}}$  and output refined CLWEs.
A significant advantage of our algorithm is its generality: it is fully independent of the method used for creating the original CLWEs and can therefore be used to enhance a wide range of models, both in supervised and unsupervised settings.
In order to demonstrate the generality of our proposed method, we conduct experiments using two groups of monolingual word embeddings trained on very different corpora:
Wiki-Embs (Grave et al., 2018): embeddings developed using Wikipedia dumps for a range of ten diverse languages: two Germanic (English|EN, German|DE), two Slavic (Croatian|HR, Russian|RU), three Romance (French|FR, Italian|IT, Spanish|ES) and three non-Indo-European (Finnish|FI from the Uralic family, Turkish|TR from the Turkic family and Chinese|ZH from the Sino-Tibetan family).
News-Embs (Artetxe et al., 2018a): embeddings trained on a multilingual News text collection, i.e.,
<sup>1</sup>See Chu and Trendafilov (2001) for derivation details.
It takes averagely 3 hours and up to 12 hours to perform
Eq. (7) on an Intel Core i9-9900K CPU. In comparison, the time required to solve Eq. (2) in each training loop is less than 1 second and the iterative  $\ell_2$ -norm-based training takes 1 to 5 hours in total.
the WaCKy Crawl of  $\{\mathrm{EN},\mathrm{DE},\mathrm{IT}\}$ , the Common Crawl of FI, and the WMT News Crawl of ES.
News-Embs are considered to be more challenging for building good quality CLWEs due to the heterogeneous nature of the data, while a considerable portion of the multilingual training corpora for Wiki-Embs are roughly parallel. Following previous studies (Lample et al., 2018; Artetxe et al., 2018a; Zhou et al., 2019; Glavaš et al., 2019), only the first 200K vocabulary entries are preserved.
Glavaš et al. (2019) provided a systematic evaluation for projection-based CLWE models, demonstrating that three methods (i.e., MUSE, VECMAP, and PROC-B) achieve the most competitive performance. A recent algorithm (JA) by Wang et al. (2020) also reported state-of-the-art results. For comprehensive comparison, we therefore use all these four methods as the main baselines for both supervised and unsupervised settings:
MUSE (Lample et al., 2018): an unsupervised CLWE model based on adversarial learning and iterative  $\ell_2$  refinement;
VEcMAP (Artetxe et al., 2018a): a robust unsupervised framework using a self-learning strategy; Proc-B (Glavaš et al., 2019): a simple but effective supervised approach to creating CLWEs;
JA-MUSE and JA-RCSLS (Wang et al., 2020): a recently proposed Joint-Align (JA) Framework, which first initialises CLWEs using joint embedding training, followed by vocabularies reallocation. It then utilises off-the-shelf CLWE methods to improve the alignment in both unsupervised (JA-MUSE) and supervised (JA-RCSLS) settings.
In the original implementations, MUSE, PROC-B and JA were only trained on Wiki-Embs while VECMAP additionally used News-Embs. Although all baselines reported performance for BLI, they used various versions of evaluation sets, hence previous results are not directly comparable with the ones reposted here. More concretely, the testsets for MUSE/JA and VECMAP are two different batches of EN-centric dictionaries, while the testset for PROC-B also supports non-EN translations.
4.3 Implementation Details of Algorithm 1
The CSLS scheme with a neighbourhood size of 10 (CSLS-10) is adopted to build synthetic dictionaries via the input CLWEs. A variable-coefficient
ordinary differential equation (VODE) solver<sup>3</sup> was implemented for the system described in Eq. (7). Suggested by Trendafilov (2003), we set the maximum order at 15, the smoothness coefficient  $\alpha$  in Eq. (5) at 1e8, the threshold  $\epsilon$  in Eq. (8) at 1e-5, and performed the integration with a fixed time interval of 1e-6. An early-stopping design was adopted to ensure computation completed in a reasonable time: in addition to the two default stopping criteria in § 3, integration is terminated if  $\int \mathrm{d}t$  reaches 5e-3 ( $\mathrm{d}t$  is the differentiation term in Eq. (7)).
In terms of the tolerance of the VODE solver, we set the absolute tolerance at 1e-7 and the relative tolerance at 1e-5, following the established approach of Kulikov (2013). These tolerance settings show good generality empirically and were used for all tested language pairs, datasets, and models in our experiments.
We evaluate the effectiveness of the proposed  $\ell_1$  refinement technique on two benchmarks: Bililingual Lexicon Induction (BLI), the de facto standard for measuring the quality of CLWEs, and a downstream natural language inference task based on cross-lingual transfer. In addition to comparison against state-of-the-art CLWE models, we also report the performance of the single-iteration  $\ell_2$  refinement method which follows steps 1-4 of Algorithm 1 then minimises  $\ell_2$  loss in the final step.