Normalization of Language Embeddings for Cross-Lingual Alignment

1 INTRODUCTION . The standard multilingual NLP approaches typically do not jointly learn a single embedding , since words tend to cluster by language , and thus are not useful for translation and cross-lingual learning tasks . Rather , after learning individual embeddings , the usual approach is to map word vectors from multiple languages into a shared cross-lingual word vector space ( Glavaš et al. , 2019 ) . This shared space creates a cross-lingual word embedding ( CLWE ) ( Karan et al. , 2020 ; Wang et al. , 2019 ) . These serve as a valuable tool for transferring data across different languages , understanding crosslinguistic differences , and cross-lingual transfer for downstream tasks , such as direct translation ( Gouws et al. , 2015 ; Heyman et al. , 2017 ; Lample et al. , 2018 ) , cross-lingual information retrieval ( Vulic & Moens , 2015 ) , cross-lingual document classification ( Klementiev et al. , 2012 ) , and crosslingual dependency parsing ( Guo et al. , 2015 ; Ahmad et al. , 2019 ) . A common element of almost all CLWE methods is the use of a rigid , orthogonal transformation mapping one embedding onto another so they inhabit a shared linguistic space . An orthogonal transformation is a special class of transformations that can be interpreted as the space of ( in our case , high-dimensional ) rotations around the origin , and allowing a mirror flip . This family of transformations preserves ( a ) linear and ( b ) angular properties . By linear properties , we mean that the straight-line Euclidean distance between elements is preserved , as are more powerful properties like analogies ( e.g. , Paris - France + Italy ≈ Rome ) . Angular properties refer to measuring angles between pairs of points ( from the origin ) , and as a result , cosine distance is preserved . Given a correspondence between pairs of objects across two embeddings , the classic Procrustes method , provides a closed-form solution which minimizes the sum of Euclidean distances . If the vectors are all first normalized , then this also maximizes the sum of cosine similarities ( Dev et al. , 2021 ) . Under this framework , there has been a flurry of work significantly improving CLWE model performance along with two directions . Semi-supervised and unsupervised models make these approaches require less input , and more amenable to lower-resource languages . For example Bootstrap Procrustes ( PROC-B ) ( Glavaš et al. , 2019 ; Vulić et al. , 2019 ) is semi-supervised in that it starts with a small pairwise correspondence ( of 500-1000 words ) , aligns those to infer a larger correspondence , 1 Published as a conference paper at ICLR 2022 and repeats applying Procrustes alignment . Methods like MUSE ( Conneau et al. , 2018a ) are unsupervised , and use a GAN to estimate a correspondence before applying a Procrustes procedure . The second direction is preprocessing the embeddings before applying the Procrustes alignment . These involve methods like removing the mean , removing principal components , and vector length normalization , discussed later . In principle , these methods aim to remove the geometry of data intrinsic to particular languages ( but not shared across languages ) while preserving similarity properties as assured by orthogonal alignment . The space of transformations allowed under orthogonal alignments is quite large , and we make the point that unless this data geometry is “ normalized ” it inhibits the alignment from optimizing over the entirety of this large space . Finally , we note that methods like Canonical Correlation Analysis ( CCA ) ( Faruqui & Dyer , 2014 ) , Discriminative Latent Variable ( DLV ) ( Ruder et al. , 2018 ) , and Ranking-based optimization ( RCSLS ) ( Joulin et al. , 2018 ) have also been applied towards finding an orthogonal alignment ( or pair of alignments ) which minimizes a different optimization function – since the objective function may not align with sums of squared Euclidean or cosine distance ( Conneau et al. , 2018a ; Smith et al. , 2017 ) . Unlike the others , the RCSLS method notably does not require a rigid transformation . This paper is on embedding preprocessing , and is agnostic to the alignment used afterward . Our contribution . This work proposes a new and general approach to preprocessing word embeddings , subsuming many previous approaches . The key is Spectral Normalization which regularizes the spectral properties of monolingual embeddings by setting all of the top singular vectors to have the same singular value . It leaves alone the smaller singular values ; these capture important information and can not be zeroed out , but making them the same value as the top singular vectors introduces too much noise . Spectral normalization already performs as well as the best previous approaches on alignment and translation tasks , and since it applies a fairly uniform stretching to the embeddings it does not distort monolingual similarity performance . Moreover , we show layering Spectral Normalization within an iterative sequence with also centering and vector length normalization improves results further . We demonstrate this improvement on the standard translation tasks , as well as on downstream cross-lingual use cases of document classification and natural language inference . We provide code at https : //github.com/poaboagye/SpecNorm . Beyond CLWE . This paper focuses on normalization for the alignment of word vector embeddings . This is for a couple of reasons . First , most relevant prior work does the same in the context of language translation , and so it allows for direct comparison . Second , these contexts have generally the largest embeddings , are trained on the most data , and are easy to interpret . However , as foundation models ( Bommasani et al. , 2021 ) become ubiquitous , many other settings induce similar challenges . For instance , alignment of various models from different data sources allows for improved ensemble embeddings ( Dev et al. , 2021 ) , and this could extend to image ( Frome et al. , 2013 ; Kiela & Bottou , 2014 ) or graph embeddings ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) . Or in aligning embeddings of spatial data ( Jenkins et al. , 2019 ) or of merchants ( Wang et al. , 2021 ) , one may want to align data from different geographic regions for tasks in transfer learning and domain adaptation . As an example of generality , we demonstrate an improvement via our normalization approaches in aligning genomic data embeddings ( Demetci et al. , 2020 ) in Appendix G . 2 EXISTING METHODS FOR ORTHOGONAL VECTOR SPACES ALIGNMENT . In an embedded representation of a set of n words , each ith word is associated with a vector ai ∈ Rd , so A = { a1 , . . . , an } is the set of n word vector representation . These vector representations ( derived by methods like word2vec ( Mikolov et al. , 2013a ) , GloVe ( Pennington et al. , 2014 ) , or FastText ( Bojanowski et al. , 2017 ) ) are chosen so words with similar pairwise cosine similarity are found in the similar local context in large text corpora on which they are trained . Higher-level linear structure is shown to emerge , such as concept subspaces and analogies ( Mikolov et al. , 2013 ) . The focus of this paper is on aligning embeddings of two languages L1 and L2 . Each embedding AL1 and AL2 , is only designed to ensure pairwise relationships between its word vectors , but the actual coordinates of those vectors do not have any explicit meaning . Yet , previous work has clearly demonstrated that there exists significant overall structural similarity , and alignment seeks to make correspondences between those structures for translation and joint understanding . 2 Published as a conference paper at ICLR 2022 2.1 PRE-PROCESSING EMBEDDINGS BEFORE ORTHOGONAL ALIGNMENT . It turns out directly aligning embeddings from two languages ( even using the “ optimal ” Procrustes solution ) does not provide the best possible joint embedding for translation tasks . While word meaning appears to hold a similar structure , languages have other properties such as differing word frequency , and this for instance leads to more frequent words having longer vectors in embeddings . This extra language-specific structure tends to interfere with alignment . As a result , a number of techniques have been developed to “ normalize ” the embeddings before Procrustes ( or other ) alignment . This , in some sense , allows the word meaning to dominate the optimization tasks without other confounding factors . We review the most common normalization approaches . Mean Centering ( C ) subtracts the mean of all vectors in an embedding from each vector in that embedding . The result is that the mean of all vectors is 0 . This is a rigid transformation , and so does not change the Euclidean distance between any pair of points in an embedding , and also preserves any linear property like analogies ( e.g. , Paris - France + Italy ≈ Rome ) . Dev et al . ( 2021 ) points out that this is the first step ( followed by the Procrustes orthogonal transformation ) to minimize the sum of squared Euclidean distances among paired words , under any rigid transformation . However , this does change the cosine distance between pairs of points . Length Normalization ( L ) makes each vector have a 2-norm equal to 1 , but retains its direction from the origin ( Artetxe et al. , 2016 ; Xing et al. , 2015 ) . This preprocessing step does not change the cosine distance between any pair of points in an embedding . But , it does change the Euclidean distance between pairs of points . Despite these contrasting goals , these two normalizations each turn out to be individually effective in regularizing the geometry of the embeddings , and allow for better CLWE . Zhang et al . ( 2019 ) , realized doing both was even more effective , and showed that iterating these two steps achieves the state-of-the-art way to preprocess , we denote as I-C+L . Iterative Normalization eventually transforms monolingual word embeddings to have unit-length and zero-mean simultaneously ( in practice they terminate this iterative process after a few steps before it achieves these two goals exactly ) . PCA Removal ( PR ) computes the principal component analysis ( PCA ) of an embedding , and then projects away from the direction of the top principal component , removing it . Mu & Viswanath ( 2018 ) observed these directions typically do not encode essential semantic relationships between words but rather align strongly with word frequency ; and PCA removal before alignment led to improved performance on several tasks . Sachidananda et al . ( 2021 ) also showed this preprocessing improved the BLI task . 2.2 SPECTRAL STATISTICS OF EMBEDDINGS . Dubossarsky et al . ( 2020 ) recently documented how cross-lingual alignment is strongly affected by the spectral statistics of monolingual embeddings . We stack the embedded vectors ai ∈ Rd as rows in a n× d matrix A ∈ Rn×d . The SVD decomposes A into UΣV ⊤ where U and V contain the left and right singular vectors , and the singular values σ1 ≥ σ2 ≥ . . . ≥ σd ≥ 0 are on the diagonal of Σ . The effective rank of A is a smoother analog to rank ( when there is noise in low rank components ) , defined er ( A ) = eH ( Σ ) where H ( Σ ) = − ∑d i=1 σ̄i log σ̄i with σ̄i = σi/ ∑d i=1 σi . The effective condition number κeff ( A ) = σ1/σer ( A ) , which replaces the numerator ( of condition number , σd ) with the more robust singular value at the effective rank . This is desired to be small in stable data sets . The joint effective condition number measures the harmonic mean of the effective condition number across two matrices A , A′ as ECOND-HM ( A , A′ ) = 2κeff ( A ) κeff ( A ′ ) κeff ( A ) +κeff ( A′ ) . The singular value gap measures how similar the singular value sequences are between two matrices as SVG ( A , A′ ) =∑d i=1 ( log σi − log σ′i ) 2 . These should both be smaller , for more comparable data sets . Dubossarsky et al . ( 2020 ) applied these to monolingual embeddings and demonstrated that the performances of several CLWE methods were closely tied to these spectral properties . Basically embeddings align better if they are better jointly conditioned , especially measured via joint effective condition number and the singular value gap . Contextual embeddings have also been shown to suffer from similar challenges ( Ethayarajh , 2019 ; Xu & Koehn , 2021 ) . Motivated by this idea , we propose methods that spectrally normalize embeddings improving these statistics while retaining intra-embedding meaning . 3 Published as a conference paper at ICLR 2022

Summary: The paper proposes an Iterative Spectral Normalization method to improve pairwise Cross-Lingual Word Embedding alignment. The work supposes that improvement to spectral properties in monolingual embedding spaces are sufficient to yield improvements to Bi-lingual Lexicon Induction, Cross-Lingual Document Classification and Cross-Lingual NLI. The main contributions are (a) the introduction of a simple, portable algorithm to improve pre-processing of embedding spaces for cross-lingual alignment and (b) empirical improvements in multiple domains of cross-lingual embedding oriented tasks.