Normalization of Language Embeddings for Cross-Lingual Alignment

1 INTRODUCTION . The standard multilingual NLP approaches typically do not jointly learn a single embedding , since words tend to cluster by language , and thus are not useful for translation and cross-lingual learning tasks . Rather , after learning individual embeddings , the usual approach is to map word vectors from multiple languages into a shared cross-lingual word vector space ( Glavaš et al. , 2019 ) . This shared space creates a cross-lingual word embedding ( CLWE ) ( Karan et al. , 2020 ; Wang et al. , 2019 ) . These serve as a valuable tool for transferring data across different languages , understanding crosslinguistic differences , and cross-lingual transfer for downstream tasks , such as direct translation ( Gouws et al. , 2015 ; Heyman et al. , 2017 ; Lample et al. , 2018 ) , cross-lingual information retrieval ( Vulic & Moens , 2015 ) , cross-lingual document classification ( Klementiev et al. , 2012 ) , and crosslingual dependency parsing ( Guo et al. , 2015 ; Ahmad et al. , 2019 ) . A common element of almost all CLWE methods is the use of a rigid , orthogonal transformation mapping one embedding onto another so they inhabit a shared linguistic space . An orthogonal transformation is a special class of transformations that can be interpreted as the space of ( in our case , high-dimensional ) rotations around the origin , and allowing a mirror flip . This family of transformations preserves ( a ) linear and ( b ) angular properties . By linear properties , we mean that the straight-line Euclidean distance between elements is preserved , as are more powerful properties like analogies ( e.g. , Paris - France + Italy ≈ Rome ) . Angular properties refer to measuring angles between pairs of points ( from the origin ) , and as a result , cosine distance is preserved . Given a correspondence between pairs of objects across two embeddings , the classic Procrustes method , provides a closed-form solution which minimizes the sum of Euclidean distances . If the vectors are all first normalized , then this also maximizes the sum of cosine similarities ( Dev et al. , 2021 ) . Under this framework , there has been a flurry of work significantly improving CLWE model performance along with two directions . Semi-supervised and unsupervised models make these approaches require less input , and more amenable to lower-resource languages . For example Bootstrap Procrustes ( PROC-B ) ( Glavaš et al. , 2019 ; Vulić et al. , 2019 ) is semi-supervised in that it starts with a small pairwise correspondence ( of 500-1000 words ) , aligns those to infer a larger correspondence , 1 Published as a conference paper at ICLR 2022 and repeats applying Procrustes alignment . Methods like MUSE ( Conneau et al. , 2018a ) are unsupervised , and use a GAN to estimate a correspondence before applying a Procrustes procedure . The second direction is preprocessing the embeddings before applying the Procrustes alignment . These involve methods like removing the mean , removing principal components , and vector length normalization , discussed later . In principle , these methods aim to remove the geometry of data intrinsic to particular languages ( but not shared across languages ) while preserving similarity properties as assured by orthogonal alignment . The space of transformations allowed under orthogonal alignments is quite large , and we make the point that unless this data geometry is “ normalized ” it inhibits the alignment from optimizing over the entirety of this large space . Finally , we note that methods like Canonical Correlation Analysis ( CCA ) ( Faruqui & Dyer , 2014 ) , Discriminative Latent Variable ( DLV ) ( Ruder et al. , 2018 ) , and Ranking-based optimization ( RCSLS ) ( Joulin et al. , 2018 ) have also been applied towards finding an orthogonal alignment ( or pair of alignments ) which minimizes a different optimization function – since the objective function may not align with sums of squared Euclidean or cosine distance ( Conneau et al. , 2018a ; Smith et al. , 2017 ) . Unlike the others , the RCSLS method notably does not require a rigid transformation . This paper is on embedding preprocessing , and is agnostic to the alignment used afterward . Our contribution . This work proposes a new and general approach to preprocessing word embeddings , subsuming many previous approaches . The key is Spectral Normalization which regularizes the spectral properties of monolingual embeddings by setting all of the top singular vectors to have the same singular value . It leaves alone the smaller singular values ; these capture important information and can not be zeroed out , but making them the same value as the top singular vectors introduces too much noise . Spectral normalization already performs as well as the best previous approaches on alignment and translation tasks , and since it applies a fairly uniform stretching to the embeddings it does not distort monolingual similarity performance . Moreover , we show layering Spectral Normalization within an iterative sequence with also centering and vector length normalization improves results further . We demonstrate this improvement on the standard translation tasks , as well as on downstream cross-lingual use cases of document classification and natural language inference . We provide code at https : //github.com/poaboagye/SpecNorm . Beyond CLWE . This paper focuses on normalization for the alignment of word vector embeddings . This is for a couple of reasons . First , most relevant prior work does the same in the context of language translation , and so it allows for direct comparison . Second , these contexts have generally the largest embeddings , are trained on the most data , and are easy to interpret . However , as foundation models ( Bommasani et al. , 2021 ) become ubiquitous , many other settings induce similar challenges . For instance , alignment of various models from different data sources allows for improved ensemble embeddings ( Dev et al. , 2021 ) , and this could extend to image ( Frome et al. , 2013 ; Kiela & Bottou , 2014 ) or graph embeddings ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) . Or in aligning embeddings of spatial data ( Jenkins et al. , 2019 ) or of merchants ( Wang et al. , 2021 ) , one may want to align data from different geographic regions for tasks in transfer learning and domain adaptation . As an example of generality , we demonstrate an improvement via our normalization approaches in aligning genomic data embeddings ( Demetci et al. , 2020 ) in Appendix G . 2 EXISTING METHODS FOR ORTHOGONAL VECTOR SPACES ALIGNMENT . In an embedded representation of a set of n words , each ith word is associated with a vector ai ∈ Rd , so A = { a1 , . . . , an } is the set of n word vector representation . These vector representations ( derived by methods like word2vec ( Mikolov et al. , 2013a ) , GloVe ( Pennington et al. , 2014 ) , or FastText ( Bojanowski et al. , 2017 ) ) are chosen so words with similar pairwise cosine similarity are found in the similar local context in large text corpora on which they are trained . Higher-level linear structure is shown to emerge , such as concept subspaces and analogies ( Mikolov et al. , 2013 ) . The focus of this paper is on aligning embeddings of two languages L1 and L2 . Each embedding AL1 and AL2 , is only designed to ensure pairwise relationships between its word vectors , but the actual coordinates of those vectors do not have any explicit meaning . Yet , previous work has clearly demonstrated that there exists significant overall structural similarity , and alignment seeks to make correspondences between those structures for translation and joint understanding . 2 Published as a conference paper at ICLR 2022 2.1 PRE-PROCESSING EMBEDDINGS BEFORE ORTHOGONAL ALIGNMENT . It turns out directly aligning embeddings from two languages ( even using the “ optimal ” Procrustes solution ) does not provide the best possible joint embedding for translation tasks . While word meaning appears to hold a similar structure , languages have other properties such as differing word frequency , and this for instance leads to more frequent words having longer vectors in embeddings . This extra language-specific structure tends to interfere with alignment . As a result , a number of techniques have been developed to “ normalize ” the embeddings before Procrustes ( or other ) alignment . This , in some sense , allows the word meaning to dominate the optimization tasks without other confounding factors . We review the most common normalization approaches . Mean Centering ( C ) subtracts the mean of all vectors in an embedding from each vector in that embedding . The result is that the mean of all vectors is 0 . This is a rigid transformation , and so does not change the Euclidean distance between any pair of points in an embedding , and also preserves any linear property like analogies ( e.g. , Paris - France + Italy ≈ Rome ) . Dev et al . ( 2021 ) points out that this is the first step ( followed by the Procrustes orthogonal transformation ) to minimize the sum of squared Euclidean distances among paired words , under any rigid transformation . However , this does change the cosine distance between pairs of points . Length Normalization ( L ) makes each vector have a 2-norm equal to 1 , but retains its direction from the origin ( Artetxe et al. , 2016 ; Xing et al. , 2015 ) . This preprocessing step does not change the cosine distance between any pair of points in an embedding . But , it does change the Euclidean distance between pairs of points . Despite these contrasting goals , these two normalizations each turn out to be individually effective in regularizing the geometry of the embeddings , and allow for better CLWE . Zhang et al . ( 2019 ) , realized doing both was even more effective , and showed that iterating these two steps achieves the state-of-the-art way to preprocess , we denote as I-C+L . Iterative Normalization eventually transforms monolingual word embeddings to have unit-length and zero-mean simultaneously ( in practice they terminate this iterative process after a few steps before it achieves these two goals exactly ) . PCA Removal ( PR ) computes the principal component analysis ( PCA ) of an embedding , and then projects away from the direction of the top principal component , removing it . Mu & Viswanath ( 2018 ) observed these directions typically do not encode essential semantic relationships between words but rather align strongly with word frequency ; and PCA removal before alignment led to improved performance on several tasks . Sachidananda et al . ( 2021 ) also showed this preprocessing improved the BLI task . 2.2 SPECTRAL STATISTICS OF EMBEDDINGS . Dubossarsky et al . ( 2020 ) recently documented how cross-lingual alignment is strongly affected by the spectral statistics of monolingual embeddings . We stack the embedded vectors ai ∈ Rd as rows in a n× d matrix A ∈ Rn×d . The SVD decomposes A into UΣV ⊤ where U and V contain the left and right singular vectors , and the singular values σ1 ≥ σ2 ≥ . . . ≥ σd ≥ 0 are on the diagonal of Σ . The effective rank of A is a smoother analog to rank ( when there is noise in low rank components ) , defined er ( A ) = eH ( Σ ) where H ( Σ ) = − ∑d i=1 σ̄i log σ̄i with σ̄i = σi/ ∑d i=1 σi . The effective condition number κeff ( A ) = σ1/σer ( A ) , which replaces the numerator ( of condition number , σd ) with the more robust singular value at the effective rank . This is desired to be small in stable data sets . The joint effective condition number measures the harmonic mean of the effective condition number across two matrices A , A′ as ECOND-HM ( A , A′ ) = 2κeff ( A ) κeff ( A ′ ) κeff ( A ) +κeff ( A′ ) . The singular value gap measures how similar the singular value sequences are between two matrices as SVG ( A , A′ ) =∑d i=1 ( log σi − log σ′i ) 2 . These should both be smaller , for more comparable data sets . Dubossarsky et al . ( 2020 ) applied these to monolingual embeddings and demonstrated that the performances of several CLWE methods were closely tied to these spectral properties . Basically embeddings align better if they are better jointly conditioned , especially measured via joint effective condition number and the singular value gap . Contextual embeddings have also been shown to suffer from similar challenges ( Ethayarajh , 2019 ; Xu & Koehn , 2021 ) . Motivated by this idea , we propose methods that spectrally normalize embeddings improving these statistics while retaining intra-embedding meaning . 3 Published as a conference paper at ICLR 2022

Summary: This paper describes three new methods for normalizing word embeddings before cross-lingual alignment. The first method, SpecNorm, caps the singular values of the word embeddings to be twice the average (original) singular values. The second method does mean centering, SpecNorm, and L2 normalization. The third repeats this for a fixed number of iterations (5). Experiments on a wide variety of tasks generally show improvements.