Year: 2022

Abstract:
Learning a good transfer function to map the word vectors from two languages into a shared cross-lingual word vector space plays a crucial role in cross-lingual NLP. It is useful in translation tasks and important in allowing complex models built on a high-resource language like English to be directly applied on an aligned low resource language. While Procrustes and other techniques can align language models with some success, it has recently been identified that structural differences (for instance, due to differing word frequency) create different profiles for various monolingual embedding. When these profiles differ across languages, it correlates with how well languages can align and their performance on cross-lingual downstream tasks. In this work, we develop a very general language embedding normalization procedure, building and subsuming various previous approaches, which removes these structural profiles across languages without destroying their intrinsic meaning. We demonstrate that meaning is retained and alignment is improved on similarity, translation, and cross-language classification tasks.

heading: 1 INTRODUCTION
text: The standard multilingual NLP approaches typically do not jointly learn a single embedding, since words tend to cluster by language, and thus are not useful for translation and cross-lingual learning tasks. Rather, after learning individual embeddings, the usual approach is to map word vectors from multiple languages into a shared cross-lingual word vector space (Glavaš et al., 2019). This shared space creates a cross-lingual word embedding (CLWE) (Karan et al., 2020; Wang et al., 2019). These serve as a valuable tool for transferring data across different languages, understanding crosslinguistic differences, and cross-lingual transfer for downstream tasks, such as direct translation (Gouws et al., 2015; Heyman et al., 2017; Lample et al., 2018), cross-lingual information retrieval (Vulic & Moens, 2015), cross-lingual document classification (Klementiev et al., 2012), and crosslingual dependency parsing (Guo et al., 2015; Ahmad et al., 2019).
A common element of almost all CLWE methods is the use of a rigid, orthogonal transformation mapping one embedding onto another so they inhabit a shared linguistic space. An orthogonal transformation is a special class of transformations that can be interpreted as the space of (in our case, high-dimensional) rotations around the origin, and allowing a mirror flip. This family of transformations preserves (a) linear and (b) angular properties. By linear properties, we mean that the straight-line Euclidean distance between elements is preserved, as are more powerful properties like analogies (e.g., Paris - France + Italy ≈ Rome). Angular properties refer to measuring angles between pairs of points (from the origin), and as a result, cosine distance is preserved. Given a correspondence between pairs of objects across two embeddings, the classic Procrustes method, provides a closed-form solution which minimizes the sum of Euclidean distances. If the vectors are all first normalized, then this also maximizes the sum of cosine similarities (Dev et al., 2021).
Under this framework, there has been a flurry of work significantly improving CLWE model performance along with two directions. Semi-supervised and unsupervised models make these approaches require less input, and more amenable to lower-resource languages. For example Bootstrap Procrustes (PROC-B) (Glavaš et al., 2019; Vulić et al., 2019) is semi-supervised in that it starts with a small pairwise correspondence (of 500-1000 words), aligns those to infer a larger correspondence,
1
Published as a conference paper at ICLR 2022
and repeats applying Procrustes alignment. Methods like MUSE (Conneau et al., 2018a) are unsupervised, and use a GAN to estimate a correspondence before applying a Procrustes procedure.
The second direction is preprocessing the embeddings before applying the Procrustes alignment. These involve methods like removing the mean, removing principal components, and vector length normalization, discussed later. In principle, these methods aim to remove the geometry of data intrinsic to particular languages (but not shared across languages) while preserving similarity properties as assured by orthogonal alignment. The space of transformations allowed under orthogonal alignments is quite large, and we make the point that unless this data geometry is “normalized” it inhibits the alignment from optimizing over the entirety of this large space.
Finally, we note that methods like Canonical Correlation Analysis (CCA) (Faruqui & Dyer, 2014), Discriminative Latent Variable (DLV) (Ruder et al., 2018), and Ranking-based optimization (RCSLS) (Joulin et al., 2018) have also been applied towards finding an orthogonal alignment (or pair of alignments) which minimizes a different optimization function – since the objective function may not align with sums of squared Euclidean or cosine distance (Conneau et al., 2018a; Smith et al., 2017). Unlike the others, the RCSLS method notably does not require a rigid transformation.
This paper is on embedding preprocessing, and is agnostic to the alignment used afterward.
Our contribution. This work proposes a new and general approach to preprocessing word embeddings, subsuming many previous approaches. The key is Spectral Normalization which regularizes the spectral properties of monolingual embeddings by setting all of the top singular vectors to have the same singular value. It leaves alone the smaller singular values; these capture important information and cannot be zeroed out, but making them the same value as the top singular vectors introduces too much noise. Spectral normalization already performs as well as the best previous approaches on alignment and translation tasks, and since it applies a fairly uniform stretching to the embeddings it does not distort monolingual similarity performance. Moreover, we show layering Spectral Normalization within an iterative sequence with also centering and vector length normalization improves results further. We demonstrate this improvement on the standard translation tasks, as well as on downstream cross-lingual use cases of document classification and natural language inference. We provide code at https://github.com/poaboagye/SpecNorm.
Beyond CLWE. This paper focuses on normalization for the alignment of word vector embeddings. This is for a couple of reasons. First, most relevant prior work does the same in the context of language translation, and so it allows for direct comparison. Second, these contexts have generally the largest embeddings, are trained on the most data, and are easy to interpret. However, as foundation models (Bommasani et al., 2021) become ubiquitous, many other settings induce similar challenges. For instance, alignment of various models from different data sources allows for improved ensemble embeddings (Dev et al., 2021), and this could extend to image (Frome et al., 2013; Kiela & Bottou, 2014) or graph embeddings (Grover & Leskovec, 2016; Perozzi et al., 2014). Or in aligning embeddings of spatial data (Jenkins et al., 2019) or of merchants (Wang et al., 2021), one may want to align data from different geographic regions for tasks in transfer learning and domain adaptation. As an example of generality, we demonstrate an improvement via our normalization approaches in aligning genomic data embeddings (Demetci et al., 2020) in Appendix G.

heading: 2 EXISTING METHODS FOR ORTHOGONAL VECTOR SPACES ALIGNMENT
text: In an embedded representation of a set of n words, each ith word is associated with a vector ai ∈ Rd, so A = {a1, . . . , an} is the set of n word vector representation. These vector representations (derived by methods like word2vec (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), or FastText (Bojanowski et al., 2017)) are chosen so words with similar pairwise cosine similarity are found in the similar local context in large text corpora on which they are trained. Higher-level linear structure is shown to emerge, such as concept subspaces and analogies (Mikolov et al., 2013).
The focus of this paper is on aligning embeddings of two languages L1 and L2. Each embedding AL1 and AL2 , is only designed to ensure pairwise relationships between its word vectors, but the actual coordinates of those vectors do not have any explicit meaning. Yet, previous work has clearly demonstrated that there exists significant overall structural similarity, and alignment seeks to make correspondences between those structures for translation and joint understanding.
2
Published as a conference paper at ICLR 2022

heading: 2.1 PRE-PROCESSING EMBEDDINGS BEFORE ORTHOGONAL ALIGNMENT
text: It turns out directly aligning embeddings from two languages (even using the “optimal” Procrustes solution) does not provide the best possible joint embedding for translation tasks. While word meaning appears to hold a similar structure, languages have other properties such as differing word frequency, and this for instance leads to more frequent words having longer vectors in embeddings. This extra language-specific structure tends to interfere with alignment. As a result, a number of techniques have been developed to “normalize” the embeddings before Procrustes (or other) alignment. This, in some sense, allows the word meaning to dominate the optimization tasks without other confounding factors. We review the most common normalization approaches.
Mean Centering (C) subtracts the mean of all vectors in an embedding from each vector in that embedding. The result is that the mean of all vectors is 0. This is a rigid transformation, and so does not change the Euclidean distance between any pair of points in an embedding, and also preserves any linear property like analogies (e.g., Paris - France + Italy ≈ Rome). Dev et al. (2021) points out that this is the first step (followed by the Procrustes orthogonal transformation) to minimize the sum of squared Euclidean distances among paired words, under any rigid transformation. However, this does change the cosine distance between pairs of points.
Length Normalization (L) makes each vector have a 2-norm equal to 1, but retains its direction from the origin (Artetxe et al., 2016; Xing et al., 2015). This preprocessing step does not change the cosine distance between any pair of points in an embedding. But, it does change the Euclidean distance between pairs of points.
Despite these contrasting goals, these two normalizations each turn out to be individually effective in regularizing the geometry of the embeddings, and allow for better CLWE. Zhang et al. (2019), realized doing both was even more effective, and showed that iterating these two steps achieves the state-of-the-art way to preprocess, we denote as I-C+L. Iterative Normalization eventually transforms monolingual word embeddings to have unit-length and zero-mean simultaneously (in practice they terminate this iterative process after a few steps before it achieves these two goals exactly).
PCA Removal (PR) computes the principal component analysis (PCA) of an embedding, and then projects away from the direction of the top principal component, removing it. Mu & Viswanath (2018) observed these directions typically do not encode essential semantic relationships between words but rather align strongly with word frequency; and PCA removal before alignment led to improved performance on several tasks. Sachidananda et al. (2021) also showed this preprocessing improved the BLI task.

heading: 2.2 SPECTRAL STATISTICS OF EMBEDDINGS
text: Dubossarsky et al. (2020) recently documented how cross-lingual alignment is strongly affected by the spectral statistics of monolingual embeddings. We stack the embedded vectors ai ∈ Rd as rows in a n× d matrix A ∈ Rn×d. The SVD decomposes A into UΣV ⊤ where U and V contain the left and right singular vectors, and the singular values σ1 ≥ σ2 ≥ . . . ≥ σd ≥ 0 are on the diagonal of Σ. The effective rank of A is a smoother analog to rank (when there is noise in low rank components), defined er(A) = eH(Σ) where H(Σ) = − ∑d i=1 σ̄i log σ̄i with σ̄i = σi/ ∑d i=1 σi. The effective condition number κeff(A) = σ1/σer(A), which replaces the numerator (of condition number, σd) with the more robust singular value at the effective rank. This is desired to be small in stable data sets. The joint effective condition number measures the harmonic mean of the effective condition number across two matrices A,A′ as ECOND-HM(A,A′) = 2κeff(A)κeff(A
′) κeff(A)+κeff(A′)
. The singular value gap measures how similar the singular value sequences are between two matrices as SVG(A,A′) =∑d
i=1(log σi − log σ′i)2. These should both be smaller, for more comparable data sets. Dubossarsky et al. (2020) applied these to monolingual embeddings and demonstrated that the performances of several CLWE methods were closely tied to these spectral properties. Basically embeddings align better if they are better jointly conditioned, especially measured via joint effective condition number and the singular value gap. Contextual embeddings have also been shown to suffer from similar challenges (Ethayarajh, 2019; Xu & Koehn, 2021). Motivated by this idea, we propose methods that spectrally normalize embeddings improving these statistics while retaining intra-embedding meaning.
3
Published as a conference paper at ICLR 2022

heading: 3 NEW METHOD: SPECTRAL NORMALIZATION
text: Unit-length and zero-mean normalization makes embedding vectors from a language lie on a hypersphere with the center of the hypersphere centered at the origin. However, this does not take into account how the word embeddings vectors are clustered on the hypersphere. See example in Figure 1 where despite centering and length normalization, words are grouped differently across languages, and this prevents a close alignment. Approaches like PCA removal and mean centering have the effect of reducing the top principal component or top singular vector. As a result, if the spectral properties are extreme, it can help regularize them. However, this approach can be blunt. PCA removal makes the top singular value exactly 0, so the condition number becomes infinite.
To this effect, we propose a new algorithm Spectral Normalization that more gently regularizes the spectral properties of word embeddings; see Algorithm 1. We will then combine it with other approaches to again ensure the embedding vectors lie on the unit sphere. In Figure 1, this approach spreads out words on the unit sphere without reordering them, and allows for a closer alignment.
Algorithm 1 Spectral Normalization (SpecNorm(A, β)) 1: Compute svd(A) = UΣV ⊤; Let D ∈ Rd×d be a diagonal matrix. 2: Compute η = √ ∥A∥2F /d, where d is the dimension of the word embedding
3: for i = 1, . . . , d do 4: if (Σii > βη) then Dii ← Σii/(βη) 5: else Dii = 1. 6: return AVD−1
This updates part of the spectral properties of embedding A as a whole, using a parameter β ≥ 0 (fixed later as β = 2 via cross-validation). Based on an average of singular values η = √ ∥A∥2F /d, if a value is above β times that average, it adjusts it to βη. Hence, all of the top directions are given the same singular value. Otherwise, if it is below βη, it is considered a minor effect (some are quite small, and fairly noisy), and it is left alone. If these small ones are completely zeroed out, the critical information within is destroyed. However, if these small ones are given the same value (i.e., βη) then components which do not contribute to the most prevalent aspects of a vectors similarity are given more importance, and we observe that the usefulness of the embedding decreases.
Iterative Spectral Normalization. Spectral normalization makes the most sense (see Appendix J) in a setting where the vectors are already centered, and also unit length. While SpecNorm does not change the center of the data, it does not maintain the length of individual vectors. As such, we advocate combining these methods into a single iterative algorithm: I-C+SN+L as in Algorithm 2.
Algorithm 2 Iterative Spectral Normalization with C+L normalization (I-C+SN+L(A, #Iter)) 1: for #Iter steps do 2: A← Center A 3: A← SpecNorm(A) 4: A← Unit length normalization of A 5: return A
4
Published as a conference paper at ICLR 2022
We observe in Figure 2 that this process significantly improves the spectral properties compared to any other approach. Without preprocessing (None), the languages (EN: English, DE: German, HI: Hindi, JA: Japanese shown) have large effective condition numbers – indicating that there is a large disparity between meaningful singular values. Note the y-axis is in the log scale. Hence, aligning these languages without normalization would likely restrict alignment among top singular vectors, not allowing enough degree of freedom to align corresponding words.
In contrast, after preprocessing when these values are more uniform, rotations among the dimensions containing the top principal components will not have an influence on the data distribution, and can fully optimize the alignment between words. Moreover, Figure 2 shows that I-C+SN+L most decreases the effective condition number, joint effective condition number, and singular value gap. Further, these values are fairly uniform across languages, despite great variation beforehand (as shown with None). In fact, I-C+SN+L is much more effective than other methods.
Is Iteration Necessary? While alternating optimization is a common paradigm, is it necessary or useful to achieve these normalization goals? There may be many solutions which achieve length, center, and spectral normalization simultaneously. In fact, we observe that “centering” with the geometric median (Chatelon et al., 1978; Eyster et al., 1973; Overton, 1983) and then length normalizing vectors (we call this GeoMediaN, and detail it in Appendix A) achieves length and center normalization without iteration. However, as we observe, it under-performs I-C+L.

heading: 4 EXPERIMENTAL ANALYSIS
text: We provide an evaluation of our proposed preprocessing methods using eight (8) language embeddings pre-trained on Wikipedia (Bojanowski et al., 2017) of each language: Croatian (HR), English (EN), Finnish (FI), French (FR), German (DE), Italian (IT), Russian (RU), and Turkish (TR). We use the 300-dimensional fastText (Bojanowski et al., 2017) embeddings, and all vocabularies are trimmed to the 200K most frequent words.
Alignment evaluation tasks: BLI We evaluate and compare our proposed preprocessing methods mostly on the Bilingual Lexicon Induction (BLI) task, a word translation task. BLI has become the de facto evaluation task for CLWE models. For words in the source language, this task retrieves the nearest neighbors in the target language after alignment to check if it contains the translation. It reports the mean average precision (MAP) (Glavaš et al., 2019), which is equivalent to the mean reciprocal rank (MRR), of the translation. Unless stated otherwise, reported values on baseline methods are taken from (Glavaš et al., 2019), and use the Google Translate (GTrans) dictionary from (Glavaš et al., 2019). We trained (aligned) using 1k, 3k and 5k source words and evaluated (tested) on separate 2k source test queries, unless noted otherwise.
Alignment Algorithms. We evaluated and compared the result of several supervised rigidtransformation CLWE models on the evaluation benchmarks using our proposed methods. All have publicly available codes, links are found in the reference citation. These include Canonical Correlation Analysis (CCA) (Faruqui & Dyer, 2014), Procrustes (PROC) (Artetxe et al., 2016; Smith et al., 2017; Xing et al., 2015; Dev et al., 2021), Bootstrapping Procrustes (PROC-B) (Glavaš et al., 2019), and Discriminative Latent-Variable (DLV) (Ruder et al., 2018), as discussed in Section 1.
5
Published as a conference paper at ICLR 2022
We also consider Ranking-Based Optimization (RCSLS) (Joulin et al., 2018) which is not a rigid alignment. In a few places, we also compare with VECMAP (Artetxe et al., 2018) as an example of an unsupervised alignment process. This should only use the geometry of the global embedding structure, e.g., derived from the natural ontology, and our normalization method still helps.

heading: 4.1 HYPERPARAMETER TUNING
text: Our main proposed algorithm I-C+SN+L has a few simple parameters. To avoid overfitting, we choose these through cross-validation on English (EN) and a held-out set of five (5) languages Hindi (HI), Russian (RU), Chinese (ZH), Japanese (JA), Turkish (TR). Ten (10) Language pairs of the form EN-X and X-EN were considered. The hyperparameters β ∈ {1, 2, 3, 4, 5} and #Iter (number of iterations) ∈ {1, 2, 3, 4, 5} were fine-tuned for I-C+SN+L. We used the publicly available MUSE translation dictionary (Conneau et al., 2018a) for hyperparameter tuning. The Procrustes alignment algorithm was trained on 5k source words and evaluated on 1.5k source test queries. We reported the mean average precision (MAP) in Table 1 for β ∈ {1, 2, 3, 4, 5} and with #Iter ∈ {1, 2, 3, 4, 5}. We observe the value of β = 2 was consistently the best threshold (although any β ≥ 2 performed similarly). However, the result did not change much with respect to the number of iterations.
The tie between the #Iter hyperparameter was broken using their performance on 13 English word similarity benchmarks; details in Appendix D. In Table 2 reports the average Spearman rank coefficient score on the tasks (None means no normalization). (β, #Iter) = (2, 5) achieved the highest score. So hereafter, we applying I-C+SN+L with the hyperparameter (β, #Iter) = (2, 5).
Note that the proposed approach (I-C+SN+L) only increased this score for these similarity tasks, so showed no signs of distorting inherent information. Although Spectral Normalization does not exactly preserve the linear properties or angular properties (as centering and length normalization do, one each, respectively), it does not suffer ill effects. We hypothesize this is because it is somewhat uniformly stretching words along with the major modes of variation, and is effectively removing information not relevant to meaning, like frequency. This benign effect is in contrast to other spectral adjustments (removing small singular values, or setting all to the same value); see Appendix D.

heading: 4.2 BLI PERFORMANCE ACROSS NORMALIZATION AND ALIGNMENT ALGORITHMS
text: We compare and evaluate the BLI performance (MAP) of various normalization algorithms from previous works to our proposed algorithms. Using the MUSE translation dictionary, we trained CCA, PROC, PROC-B and RCSLS on 5k source words and evaluated on 1.5k source test queries. The following normalization algorithms were used in the comparison analysis: PR (PCA Removal) (Mu & Viswanath, 2018), GeoMediaN (Geometric Median Normalization), C+L (Mean centering and Length normalization, 1 round), I-C+L (Iterative Mean centering and Length normalization, 5 rounds) (Zhang et al., 2019), SN (Spectral Normalization, 1 round), C+SN+L (Mean centering, Spectral Normalization and Length normalization, 1 round), and I-C+SN+L (Iterative Mean
6
Published as a conference paper at ICLR 2022
centering, Spectral Normalization and Length normalization, 5 rounds). Specifically, we evaluated 18 language pairs, i.e., English (EN) from/to Bulgarian (BG), Catalan (CA), Czech (CS), German (DE), Spanish (ES), Korean (KO), Thai (TH) and Chinese (ZH), separate from hyperparameter tuning. The average is reported in Table 3, all results are in Appendix J. For almost all algorithms I-C+SN+L achieves the best scores (and especially on XL−EN, often considerably better). The only exceptions are on non-rigid RCSLS when C+SN+L (with no iteration) or just SN (without C+L) performs slightly better. So, Spectral Normalization, and in particular I-C+SN+L, is shown as the best way to normalize languages before alignment.
We also compute the average BLI MAP score across all 28 language pairs for more direct comparison to prior work (Glavaš et al., 2019), summarized in Table 4 and Appendix E. All results are in Appendix L. We compare I-C+SN+L (denoted with SN) against no normalization on various dictionary sizes: 1k, 3k and 5k source words and evaluated on 2k source test queries. In all cases, IC+SN+L consistently improves over the baseline; see Appendix E. This includes improvement over RCSLS which is non-rigid, so in principle could “learn” adjustments similar to our normalization in the process of alignment. We also tested on VECMAP, an unsupervised approach; I-C+SN+L preprocessing also improves this result from 0.375 to 0.410.
Published as a conference paper at ICLR 2022
Normalizing contextual type-level embeddings. Table 5 compares the impact of Iterative Normalization and Spectral Normalization on the BLI performance on aligning Cross-lingual Contextual Embedding. We follow the implementation details from Xu & Koehn (2021) for learning representatives within BERT (see Appendix C for more details). Then the learned normalization can be viewed as a composition to the functional embedding, so is compatible with downstream uses. Our proposed normalization algorithm, Iterative Spectral Normalization (I-C+SN+L), clearly outperforms Iterative Normalization (I-C+L) on the BLI task for aligning Contextual Embeddings.
This buttresses the claim made by Xu & Koehn (2021) that improving the degree of isomorphism between contextual embeddings spaces enhances the quality of learned alignment and subsequently a better performance on the BLI task.

heading: 4.3 DOWNSTREAM TASKS
text: We conclude by demonstrating that Spectral Normalization not only improves in direct translation tasks, but also captures an important global structure that generalizes from a high resource language (i.e., English, EN) to lower resource languages. In both examples, a powerful classifier is trained on the EN embedding (after normalization), and then we demonstrate that after a lower resource language (e.g., German, DE) has been normalized and aligned the analysis task can be directly applied to that language. In particular, adding our normalization (I-C+SN+L) dramatically improves the results over not doing that step, and typically improves on I-C+L normalization.
Cross-lingual Document Classification (CLDC). The CLDC task builds a topic classification using a language model on a high resource language (in our case English EN) across 15 topics. The TED CLDC corpus assembled by (Hermann & Blunsom, 2014) was used for training and evaluation. Following (Glavaš et al., 2019) a simple CNN was used to train. Table 6 summarizes the average F1-score for all topic classifiers on 5 language pairs. The CLWEs induced by PROCSN, PROC-BSN, DLVSN, and RCSLSSN (using I-C+SN+L) outperformed the baseline result (with no normalization) on the CLDC task, greatly improving the best average score from 0.421 to 0.461, and improving over I-C+SN+L for all alignments except RCSLS. Glavaš et al. (2019) used only 12 of 15 topics, but could not confirm which, so we re-ran all baselines using all 15 topics.
Cross-lingual Natural Language Inference (XNLI). We evaluated the CLWE on a cross-lingual natural language inference (XNLI) task. We used a multi-lingual XNLI corpus created by (Conneau et al., 2018b), which is a collection of sentence pairs from the English MultiNLI corpus (Williams et al., 2018) translated into 14 languages. The MultiNLI corpus contains 433k sentence pairs with the labels entailment, contradiction, and neutral. The intersection between XNLI languages and BLI languages results in four XNLI evaluation pairs: EN-DE, EN-FR, EN-TR, and EN-RU. We use the training setup in (Glavaš et al., 2019) with the Enhanced Sequential Inference Model (Chen et al., 2017) on English after normalization. First, we aligned normalized versions of each language onto
8
Published as a conference paper at ICLR 2022
the normalized EN embedding to obtain the shared cross-lingual embedding. Then we used the 5k test pairs from the XNLI corpus to evaluate each language alignment. Table 7 shows the result for PROC, PROC-B, and RCSLS alignments (DLV and VECMAP transform the EN embedding in the process, so were omitted). We compare I-C+L and our I-C+SN+L normalization against the same procedure without normalization from Glavaš et al. (2019). As in other experiments, our normalization improves the average test accuracy with each alignment approach.

heading: 5 CONCLUSION & DISCUSSION
text: We introduce a new way to normalize embeddings, based on spectral normalization, for use in creating cross-lingual word embeddings. Our approach generalizes previous approaches, and when used to individually preprocess monolingual embeddings, it allows alignment procedures to find better alignments: resulting in improved performance on direct translation tasks as well as crosslingual topic classification and natural language inference tasks. Moreover, we demonstrate this improvement is very broadly useful; it holds in contextual embeddings as well as on embeddings of non-language data (on genomic data in Appendix G).
Limitations. Our proposed preprocessing method heavily relies on the approximately isomorphic assumption of the two monolingual embedding spaces to be aligned (Mikolov et al., 2013b; Ruder et al., 2019). Under this condition, it is easier to learn a robust linear map between two monolingual embedding spaces that are approximately isomorphic than spaces that are not. Recent, lines of work (Søgaard et al., 2018; Nakashole & Flauger, 2018; Patra et al., 2018) have questioned this approximately isomorphic assumption and have shown that it does not hold in general. Also, it hinders the performance of some CLWE methods. The goal of our proposed preprocessing method is to canonically preprocess the two monolingual embedding spaces to improve their isomorphism, and we show it is generally effective.
CLWE methods such as Joint CLWE methods (Wang et al., 2019) and Cross-lingual Anchoring (Ormazabal et al., 2021) departs from the approximately isomorphic assumption. They learn new embeddings from scratch, using a different objective function that encourages the alignment of representations or the vector embeddings of the words. As such our proposed preprocessing method does not apply. These methods, however, are less flexible and portable than alignment-based approaches, since they need to learn a new embedding for each pair of languages, and would need to re-train if only one is updated or a different or multiple pairs are considered. We have a preliminary comparison with Ormazabal et al. (2021) in Appendix F and show our methods are competitive.
9
Published as a conference paper at ICLR 2022

heading: 6 ETHICS STATEMENT
text: The vast majority of NLP research and cutting-edge advancements are in English. This disadvantages those who primarily operate in other languages, with less developed models, or less data to train models. As large language models are the cornerstone of most NLP research and development in English, one of this work’s main goals is to inexpensively port these advances to other languages, and those who use them. This will help unlock this technology to many around the world. As with most models, this accuracy and improvement may vary across tasks and languages.
While language models have many positive use cases including improving accessibility, better recommendations, and increased automation, they have some negative effects as well. These include requiring potentially large computational and hence environmental costs, encoding and exacerbating bias, and aiding in automatically generating fake or deceitful content. While this paper is unlikely to change the desire to use embeddings, it aims to reduce the burden of use and increase the effectiveness in lower-resource settings. And in particular to port models trained in English to other languages. This would reduce the cost of retraining in other languages if the English model can be reused, easing environmental costs. We support the maturing efforts in attenuating bias in all such embeddings. And while we acknowledge the possibility of this work aiding in the creation of deceitful content and the harm it can cause, we believe the many benefits outweigh the harms.

heading: 7 REPRODUCIBILITY STATEMENT
text: All existing methods are compared with publicly available codes with publicly available data, with links above or in references. The exception is code for CLDC and XNLI is shared by Glavaš et al. (2019). Everything was run with default parameters; the exception is RCSLS, where we follow the suggested hyperparameter selection strategy (Joulin et al., 2018) (with learning rate in {1, 10, 25, 50} and epoch number in {10, 20}). Our new code for SpecNorm is in Appendix H and here https://github.com/poaboagye/SpecNorm.

heading: ACKNOWLEDGMENTS
text: We thank our support from NSF IIS-1816149, CCF-2115677, and Visa Research.
10
Published as a conference paper at ICLR 2022

heading: A GEOMETRIC MEDIAN NORMALIZATION
text: Iterative Normalization converges towards individual word embeddings having a unit length and each monolingual embedding having a zero mean. Zhang et al. (2019) showed that iterating solutions for these distinct goals will eventually converge to a solution which satisfies both.
In this paper, we observe that both goals can be done in one shot without iterating – by solving the Fermat-Weber problem (Maxwell, 1966; Love et al., 1988). This dates to the 17th century, and corresponds with identifying the geometric median of a point set. Formally, the goal is a point x∗ ∈ Rd that minimizes the sum of distances from n anchor points {a1, . . . , an} ⊂ Rd which are not colinear:
x∗ = min x n∑ i=1 ∥ai − x|. (1)
Several methods (Chatelon et al., 1978; Eyster et al., 1973; Overton, 1983) have been proposed; the most popular is the Weiszfeld’s algorithm (Weiszfeld, see below for details). It is folklore that the solution x∗ satisfies that 0 = ∑n i=1 ai−x∗ ∥ai−x∗∥ ; we do not know of a written proof, so we prove this for completeness. Theorem A.1. If x ∈ Rd is distinct from all the given anchor points, ai, then x is the geometric median is true if and only if (⇐⇒)
0 = n∑ i=1 ai − x ∥ai − x∥
(2)
Proof. (⇒) Suppose x ∈ Rd is distinct from all the given anchor points, ai and x is the geometric median such that x = T̃ (x) (see A.1) then
∇f (x) = n∑
i=1
ai − x ∥ai − x∥ = 0.
(⇐) Suppose x ∈ Rd is distinct from all the given anchor points, ai and x is the unique optimal solution of equation 1 such that ∇f (x) = 0 then solving for x while ignoring the dependency x in∥∥ai − x∥∥ yields:
x =
n∑ i=1 ∥∥ai − x∥∥−1 ai n∑
i=1
∥ai − x∥−1
which is the geometric median.
Using this characteristic of the geometric median, we can simultaneously enforce monolingual word embeddings to have unit-length and zero-mean in just one step. This can be done using the Geometric Median normalization (GeoMediaN) algorithm (as Algorithm 3). Given a monolingual word embedding A, we compute the geometric median x∗, and ”center” the data on this point, and unit length normalizes the centered embedding.
Algorithm 3 Geometric Median Normalization: GeoMediaN(A) 1: x∗ ←Weiszfeld(A) 2: for all ai ∈ A do ai← a
i−x∗ ∥ai−x∗∥
3: return A
After these steps, all vectors are unit length, and because of the folklore property (Theorem A.1), the mean of those points is also 0. As a result, we can state the following property.
17
Published as a conference paper at ICLR 2022
Theorem A.2. The output of GeoMediaN(A) is centered and length normalized.
Despite the Geometric Median Normalization algorithm’s ability to enforce unit-length and zeromean in just one step, we observe that it does not perform especially well on the BLI task. Both GeoMediaN and I-C+L achieve one of many solutions which achieve these joint goals.
A.1 WEISZFELD ALGORITHM
The Weiszfeld algorithm is an iterative method for finding the geometric median of a set of points in Euclidean space based on the reformulation of a stationary point that satisfies∇f (x) = 0. If iteration function T : Rd → Rd is defined by:
T (x) =  T̃ (x) = n∑ i=1 ∥ai−x∥−1ai n∑ i=1 ∥ai−x∥−1 if x /∈ { a1, . . . an } ai if x = ai , i = 1, . . . , n
(3)
then the Weiszfeld algorithm is:
xk+1 = T ( xk ) , k ∈ N (4)
where x0 ∈ Rd is a starting point. When the current iterate, xk /∈ { a1, . . . an } , T ( xk ) = T̃ ( xk ) ;
else, if xk = ai, then T ( xk ) = ai.
The Weiszfeld algorithm is presented in Algorithm 4 below:
Algorithm 4 Weiszfeld algorithm (WA) Input: Anchor points, ( a1, . . . an ) , x0 ∈ Rd and ϵ > 0
1: k← 0 2: while True do 3: xk+1← T ( xk ) 4: if ∥∥xk+1 − xk∥∥ 2 < ϵ then 5: return xk+1 6: k← k + 1

heading: B SPECTRAL STATISTICS AND SPECTRAL ISOMORPHISM MEASURES
text: We also explored other spectral statistics on monolingual embeddings. The numeric rank of A is a smoother analog to rank (where there is noise in low rank components), defined η(A) = ∥A∥2F /∥A∥22. The condition number of A is κ(A) = σ1/σd, and measures how close the matrix is to being truly full rank, smaller is more stable. For two matrices A1 and A2, the condition number harmonic mean is COND-HM(A1, A2) =
2κ(A1)κ(A2) κ(A1)+κ(A2) . Smaller means the matrices are more comparable. Figure 3 plots these measures, and again demonstrates that I-C+SN+L improves these measures on matrices.
We also show the raw numbers used to generate the charts in Figure 2 in the tables below.
18
Published as a conference paper at ICLR 2022
Published as a conference paper at ICLR 2022

heading: C NORMALIZING CONTEXTUAL TYPE-LEVEL EMBEDDINGS
text: Contextual Type-level Embeddings To obtain the contextual type-level embeddings, Fast Align (Dyer et al., 2013) is applied to the source-target parallel corpora to derive silver aligned token pairs. The tokenized parallel corpus is fed into pre-trained BERTs (Wolf et al., 2019; Safaya et al., 2020) of the source and target language. Since there will be multiple occurrences of type-level words and each type-level word will possess a contextual word embedding, for each type-level word, its vector representation is derived from the mean vector of all the type-level words from the monolingual corpus fed into the pre-trained language model. This is done for both source and target language. The resulting type-level contextual embedding from the source and target language is then aligned by solving the Procrustes problem.

heading: D WORD SIMILARITY TASK
text: The word similiarity task was conducted using the following English word similarity benchmarks: WS-3533 (Finkelstein et al., 2001), WS-SIM and WSREL (Agirre et al., 2009), RG-65 (Rubenstein & Goodenough, 1965), MC-30 (Miller & Charles, 1991), MTurk-2875 (Radinsky et al., 2011), MTurk-771 (Halawi et al., 2012), MEN7 (Bruni et al., 2012), YP-130 (Yang & Powers, 2006), Rare Words (Luong et al., 2013).
In addition to a baseline (None, which means no normalization), Tabel 12 shows results comparing against our proposed normalization (I-C+SN+L) and the state of the art (I-C+L). Note that both of these improve the accuracy of the similarity tests. This indicates that they are not distorting the critical information contained in the original embeddings. And our proposed approach (I-C+SN+L) increases the score the most.
In contrast, we show the results of two other spectral adjustments we considered. SSV (SSV means Same Singular Values), performs the SVD on the original embedding and then sets all the singular values to η = √ ∥A∥2F /d. Then we compute U@S@VT to get the new word embedding. So similar to SpecNorm, but it never reaches the else condition. This slightly decreases the scores on the similarity tests. This is a result of accentuating the noise directions. In the other direction, SSVZ
20
Published as a conference paper at ICLR 2022
(SSVZ means Set Singular Values to Zero) only keeps the top 40 singular values and then sets the rest to zero. Then we compute U@S@VT to get the new word embedding. This drastically reduces the similarity score. This shows those noise directions, which tend to be well below the average singular value, are still important and cannot just be removed.

heading: E SUMMARY OF MODEL PERFORMANCE AND SIGNIFICANCE TEST
text: Table 13 summarizes the performance of I-C+SN+L on several supervised and unsupervised projection-based CLWE models across all the 28 language pairs as presented in Appendix L . After preprocessing the monolingual word embeddings with I-C+SN+L, CCASN, PROCSN and DLVSN outperformed CCA, PROC and DLV respectively on 28 of 28 language pairs across all the translation dictionaries. PROC-BSN outperformed PROC-B on 25 of 28 language pairs across 1k and 3k translation dictionaries. The performance of RCSLSSN supersedes RCSLS on 25 of 28 language pairs and 28 of 28 language pairs trained on 3k and 5k translation dictionaries respectively. The unsupervised projection-based CLWE model, VECMAPSN outperformed VECMAP on 27 of 28 language pairs. The lowest performing model was RCSLSSN trained on 1k translation dictionary.
We compare the effectiveness of Iterative Spectral Normalization on n = 28 language pairs. We use a 1-tail Binomial Test to measure the significance of the consistency of the improvement. That is, Iterative Spectral Normalization is a significant improvement over no normalization if it improves the score on well more than 50% of the language pairs. Table 14 shows the p-value derived from the Binomial test with n = 28 language with probability 0.5, for different values k/n language pairs
21
Published as a conference paper at ICLR 2022
with an improvement. For instance, 21/28 language pairs with an improvement have a p-value of 0.006 so is significant at the 0.05 level and the 0.01 level. In the associated Tables 13 and 15 we show those results in bold which are significant at the 0.01 level.
Table 15 summarizes the performance of I-C+L and I-C+SN+L on several supervised and unsupervised projection-based CLWE models across all the 28 language pairs as presented in Appendix K . In all cases, I-C+SN+L outperforms I-C+L except on RCSLS where there was a tie.
Bold signifies that they are significant at a significance level of 0.01 under a Binomial Test, as shown in Table 14.

heading: F P@1 PERFORMANCE FOR MAPPING BASED CLWE METHODS VS JOINT TRAINING
text: Published as a conference paper at ICLR 2022
Joint training methods. The CLWE methods proposed by Wang et al. (2019) and Ormazabal et al. (2021) learn a new embedding from scratch including words from both languages. They do so with a different objective function than standard approaches so that it prevents words entirely clustering by language, and so cross-lingual synonyms are encouraged to be embedded close to each other. Wang et al. (2019) accomplished this by encouraging alignment of representations of words. This was enhanced by Ormazabal et al. (2021) via a mechanism (Cross-lingual Anchoring) that extended the skip-gram to have bilingual target embeddings. Ultimately, these can learn a crosslingual embedding that preserves synonyms, and is optimized over the entire source language data.
Results. Both Ormazabal et al. (2021) and Wang et al. (2019) reported P@1 (precision of the 1st nearest neighbor) BLI results for their method – as opposed to the more robust MAP evaluation we use in the remainder of the paper. So we report the BLI performance on the MUSE dataset on 4 languages before and after I-C+SN+L for a variety of alignment techniques in Table 16. We include their published numbers directly for comparison. We also include reported numbers from various other papers, denoted with ∗. The work of Ormazabal et al. (2021) slightly outperforms I-C+SN+L (with RCSLS (Joulin et al., 2018)) on EN-DE, EN-ES, and EN-FR, but under-performs on EN-RU and as a result has a lower overall average.
The result of Ormazabal et al. (2021) are certainly intriguing, and we hope to be able to compare on a broader set of languages and use the MAP evaluation in future work. We remark, however, that such approaches are less flexible than using a post-processing alignment as they require re-training, so it is less clear how to apply for multiple languages, would need to be re-trained for other languages pairs (e.g., DE-FR), and how it will work for lower resource languages where less training data is available.
After applying our proposed I-C+SN+L normalization, ADV (Conneau et al., 2018b) failed to align EN-DE, EN-ES, EN-RU. This method uses a Generative Adversarial Network (GAN), and this phenomenon is explained by Hartmann et al. (2019). In their works, they showed that unit length normalization makes GAN based CLWE method unstable and deteriorates their performance but other supervised alignments are not affected by it. This explains why the performance on PROC (Conneau et al., 2018a), RCSLS (Joulin et al., 2018) and Wasserstein Procrustes (WP) (Grave et al., 2019) are not affected by I-C+SN+L, and in fact, it leads to an improvement in the BLI performance.

heading: G ALIGNMENT OF SINGLE-CELL MULTI-OMICS DATA
text: Single-cell measurements allow scientists to study the various properties of the genome such as gene expression, chromatin accessibility, DNA methylation etc. Therefore there is the need for data integration of these single-cell measurements so that scientists can understand cell development over time and disease. However, due to the lack of cell-to-cell correspondence between the different types of measurements, it makes the process of data integration a challenging task. Motivated by this, (Demetci et al., 2020) proposed a Single-Cell alignment using Optimal Transport (SCOT), an unsupervised learning algorithm that uses Gromov Wasserstein-based optimal transport to recover the cell-to-cell correspondence between two sequencing domains. * represents results taken directly
23
Published as a conference paper at ICLR 2022
from (Demetci et al., 2020). MMD-MA (Liu et al., 2019) and UnionCom (Cao et al., 2020) are two other unsupervised single-cell alignment methods.
We follow the implementation details from (Demetci et al., 2020). * represents results taken directly from (Demetci et al., 2020). We use two single-cell multi-omics datasets: 1) scGEM assay (Cheow et al., 2016) and 2) SNAREseq assay (Chen et al., 2019). The fraction of samples is closer than the true match, (FOSCTTM) is used to evaluate the alignment performance. A lower average FOSCTTM score indicates a higher ability of SCOT to recover the correct correspondences.
Table 17 summaries the before and after I-C+L and I-C+SN+L SCOT performance based on FOSCTTM. On the SNAREseq dataset SCOT with I-C+SN+L outperforms SCOT with I-C+L but achieves the same performance as SCOT without normalization. However, processing the scGEM dataset with I-C+SN+L before alignment achieved the highest FOSCTTM score of 0.184 followed by SCOT without normalization.

heading: H CODE FOR SPECNORM
text: The code implementing our new algorithm Spectral Normalization is provided here https:// github.com/poaboagye/SpecNorm. As it is quite simple, for completeness we also present it next.
Spectral Normalization (SpecNorm) is below, referred to as SpecNorm.py. Similarly, the code implementation of Iterative Spectral Normalization (I-C+SN+L) referred to as SpecNorm usage.py is also shown below.
24
Published as a conference paper at ICLR 2022
Spectral Normalization (SpecNorm.py)
import numpy as np
def computeSVD(embeds): """ Args:
embeds: Monolingual Embedding
Returns: Singular Value Decomposition """ U, S, VT = np.linalg.svd(embeds,full_matrices=False) return U, S, VT
def SVDConX(embeds, beta): """ Args:
embeds: Monolingual Embedding beta: Use to determine smaller (noisy) singular values to be removed
Returns: Spectral Normalised Embedding """ # Perform SVD on the Monolingual Embedding _, S, VT = computeSVD(embeds) # Compute eta eta = np.sqrt(np.sum(S**2)/len(S)) # Transform diagonal matrix S_prime = 1 / S for idx, sigma in enumerate(S):
if sigma > beta*eta: S_prime[idx] = S_prime[idx] * (beta*eta) else: S_prime[idx] = 1
S_prime = np.eye(len(S)) * S_prime # Compute new monolingual embedding embeds = embeds @ VT.T @ S_prime return embeds
25
Published as a conference paper at ICLR 2022
Iterative Spectral Normalization (SpecNorm usage.py)
import numpy as np from SVDConX import SVDConX from argparse import ArgumentParser
def load_embed(filename, max_vocab=-1): words, embeds = [], [] with open(filename, ’r’) as f:
next(f) for line in f:
word, vector = line.rstrip().split(’ ’, 1) vector = np.fromstring(vector, sep=’ ’) words.append(word) embeds.append(vector) if len(embeds) == max_vocab:
break return words, np.array(embeds)
def saveEmbed(path, words, word_embeds): with open(path, ’w’) as f:
print(word_embeds.shape[0], word_embeds.shape[1], file=f) for word, embed in zip(words, word_embeds):
vector_str = ’ ’.join(str(x) for x in embed) print(word, vector_str, file = f)
def main(): parser = ArgumentParser() parser.add_argument(’--input_file’) parser.add_argument(’--output_file’) parser.add_argument(’--niter’, default=5, type=int) parser.add_argument(’--max_vocab’, default=200000, type=int) parser.add_argument(’--beta’, default=2, type=int) args = parser.parse_args()
words, embeds = load_embed(args.input_file, max_vocab=args.max_vocab) embeds /= np.linalg.norm(embeds, axis=1)[:, np.newaxis] + 1e-8
for i in range(args.niter): # Center Monoligual Embedding embeds -= embeds.mean(axis=0)[np.newaxis, :] # Perform Spectral Normalization embeds = SVDConX(embeds, args.beta) # Unit Length Normalization embeds /= np.linalg.norm(embeds, axis=1)[:, np.newaxis] + 1e-8 saveEmbed(args.output_file, words, embeds)
if __name__ == ’__main__’: main()

heading: I RUNTIME
text: Most of the alignment algorithms run on a CPU except for VecMap, which requires a GPU for faster computation. It takes about 91 seconds to run Iterative Spectral Normalization on a CPU with a β = 2 and five iterations. Hardware specifications are NVIDIA GeForce GTX Titan Xp 12GB, AMD Ryzen 7 1700 eight-core processor, and 62.8GB RAM. All alignment approaches were completed in under 15 minutes, and most less than 5 minutes. Each evaluation (BLI, CLDC, or XNLI) takes under 2 minutes, but the training step for CLDC and XNLI takes about a day each; hence our approach aims only to need to do this once (on a high resource language like English), and then use the faster alignment step to transfer this to other languages.
26
J FULL BLI PERFORMANCE OF VARIOUS NORMALIZATION ALGORITHMS
K FULL BLI RESULTS FOR ALL 28 LANGUAGE PAIRS, TRANSLATION DICTIONARIES, AND MODELS.
L FULL BLI RESULTS FOR ALL 28 LANGUAGE PAIRS, TRANSLATION DICTIONARIES, AND MODELS.

Review summaries:
The paper proposes an Iterative Spectral Normalization method to improve pairwise Cross-Lingual Word Embedding alignment. The work supposes that improvement to spectral properties in monolingual embedding spaces are sufficient to yield improvements to Bi-lingual Lexicon Induction, Cross-Lingual Document Classification and Cross-Lingual NLI. The main contributions are (a) the introduction of a simple, portable algorithm to improve pre-processing of embedding spaces for cross-lingual alignment and (b) empirical improvements in multiple domains of cross-lingual embedding oriented tasks.  

This paper describes three new methods for normalizing word embeddings before cross-lingual alignment. The first method, SpecNorm, caps the singular values of the word embeddings to be twice the average (original) singular values. The second method does mean centering, SpecNorm, and L2 normalization. The third repeats this for a fixed number of iterations (5). Experiments on a wide variety of tasks generally show improvements. 

This paper studies learning cross-lingual word embeddings (CLWE) with alignment: mapping two pre-trained monolingual word embeddings to a shared space with a linear/orthogonal mapping. Previous work shows that normalization methods and the spectral properties of monolingual word embeddings have a big impact on alignment methods. Building on these results, the paper proposes a normalization method that regularizes the spectral properties of monolingual word embeddings. The method is then combined with mean centering and length normalization in an iterative process. Empirically, the proposed normalization method improves alignment and leads to better CLWE (measured by scores on bilingual lexicon induction, cross-lingual document classification, and natural language inference).

The authors propose a normalisation method for cross-lingual text representations.  The goal is to normalise the monolingual embeddings based on spectral normalisation.  The main contributions are: novel method to normalise word embeddings, the proposed method includes different normalization approaches, and the proposed method improves performance on intrinsic and extrinsic evaluation tasks. The study shows that produced text representations keep their meaning and improve performance on downstream tasks. 

This paper presents a new approach to normalizing (static) word embeddings prior to learning the mapping between them to induce an improved shared cross-lingual word embedding (CLWE) space. The main idea is to improve shared cross-lingual spaces not via learning more sophisticated mapping functions or by designing a new CLWE learning paradigm, but rather to affect the properties of independently trained monolingual embeddings to increase their alignability. In other words, normalization can be seen as a pre-processing step that is orthogonal to the actual mapping/projection functions, and improving normalization can have a positive impact on the final shared space. The core contribution of the paper is therefore the spectral normalization method, built on top of the recent advances and insights from prior work, which yields gains over other normalization techniques from prior work in the intrinsic task of Bilingual Lexicon Induction (BLI) and several transfer tasks (which rely on the CLWEs).

The paper proposes a new normalization method for monolingual word embeddings before aligning them between different languages. The normalization is based on spectral properties of the monolingual embeddings and aims to level the contributions of those leading singular values while maintaining those smaller ones at the same time. Further inspired by previous works that centers and units the length of monolingual embeddings, the authors propose an iterative normalization that combines the spectral normalization with centering and unit length normalization. The proposed method achieves better performance in both bilingual lexical induction (BLI) tasks as well as downstream tasks like cross-lingual document classification (CLDC) and cross-lingual natural language inference (XNLI).