Filtered Inner Product Projection for Crosslingual Embedding Alignment

1 INTRODUCTION . The problem of aligning sets of embeddings , or high dimensional real valued vectors , is of great interest in natural language processing , with applications in machine translation and transfer learning , and shares connections to graph matching and assignment problems ( Grave et al. , 2019 ; Gold & Rangarajan , 1996 ) . Aligning embeddings trained on corpora from different languages has led to improved performance of supervised and unsupervised word and sentence translation ( Zou et al. , 2013 ) , sequence labeling ( Zhang et al. , 2016 ; Mayhew et al. , 2017 ) , and information retrieval ( Vulić & Moens , 2015 ) . Additionally , linguistic patterns have been studied using embedding alignment algorithms ( Schlechtweg et al. , 2019 ; Lauscher & Glavaš , 2019 ) . Embedding alignments have also been shown to improve the performance of multilingual contextual representation models ( i.e . mBERT ) , when used during intialization , on certain tasks such as multilingual document classification ( Artetxe et al. , 2020 ) Recently , algorithms using embedding alignments on the input token representations of contextual embedding models have been shown to provide efficient domain adaptation ( Poerner et al. , 2020 ) . Lastly , aligned source and target input embeddings have been shown to improve the transferability of models learned on a source domain to a target domain ( Artetxe et al. , 2018a ; Wang et al. , 2018 ; Mogadala & Rettinger , 2016 ) . In the bilingual lexicon induction task , one seeks to learn a transformation on the embeddings of a source and a target language so that translated word pairs lie close to one another in the shared representation space . Specifically , one is given a small seed dictionary D containing c pairs of translated words , and embeddings for these word pairs in a source and a target language , Xs ∈ Rc×d and Xt ∈ Rc×d . Using this seed dictionary , a transformation is learned on Xs and Xt with the objective that unseen translation pairs can be induced , often through nearest neighbors search . Previous literature on this topic has focused on aligning embeddings by minimizing matrix or distributional distances ( Grave et al. , 2019 ; Jawanpuria et al. , 2019 ; Joulin et al. , 2018a ) . For instance , Mikolov et al . ( 2013a ) proposed using Stochastic Gradient Descent ( SGD ) to learn a mapping , Ω , to minimize the sum of squared distances between pairs of words in the seed dictionary ‖XDs Ω−XDt ‖2F , which achieves high word translation accuracy for similar languages . Smith et al . ( 2017 ) and Artetxe et al . ( 2017 ) independently showed that a mapping with an additional orthogonality constraint , to preserve the geometry of the original spaces , can be solved with the closed form solution to the Orthogonal Procrustes problem , Ω∗ = arg minΩ∈O ( d ) ‖XDs Ω − XDt ‖F where O ( d ) denotes the group of d dimensional orthogonal matrices . However , these methods usually require the dimensions of the source and target language embeddings to be the same , which often may not hold . Furthermore , due to semantic shifts across languages , it ’ s often the case that a word and its translation may not co-occur with the same sets of words ( Gulordava & Baroni , 2011 ) . Therefore , seeking an alignment which minimizes all pairwise distances among translated pairs results in using information not common to both the source and target embeddings . To address these problems , we propose Filtered Inner Product Projection ( FIPP ) for mapping embeddings from different languages to a shared representation space . FIPP aligns a source embedding Xs ∈ Rn×d1 to a target embedding Xt ∈ Rm×d2 and maps vectors in Xs to the Rd2 space of Xt . Instead of word-level information , FIPP focuses on pairwise distance information , specified by the Gram matrices XsXTs and XtX T t , where the rows of Xs and Xt correspond to embeddings for the c pairs of source and target words from the seed dictionary . During alignment , FIPP tries to achieve the following two goals . First , it is desired that the aligned source embedding FIPP ( Xs ) = X̃s ∈ Rc×d2 be structurally close to the original source embedding to ensure that semantic information is retained and prevent against overfitting on the seed dictionary . This goal is reflected in the minimization of the reconstruction loss : ‖X̃sX̃Ts −XsXTs ‖2F . Second , as the usage of words and their translations vary across languages , instead of requiring X̃s to use all of the distance information fromXt , FIPP selects a filtered setK of word pairs that have similar distances in both the source and target languages : K = { ( i , j ) ∈ D : { |XsXTs −XtXTt |ij ≤ } . FIPP then minimizes a transfer loss on this set K , the squared difference in distances between the aligned source embeddings and the target embeddings : ∑ ( i , j ) ∈K ( X̃s [ i ] X̃s [ j ] T −Xt [ i ] Xt [ j ] T ) 2 . 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 Filtered Inner Product Projection for Multilingual Embedding Alignment FIPP ( Xs ) Id2 Rd2 Rd1 Rd2 Xs Xt Figure 1 . FIPP alignment of source and target embeddings , Xs and Xt , to a common representation space . Note that Xs is modified using information from Xt and mapped to Rd2 while Xt is unchanged . transformations as FIPP ’ s alignment stems directly from inner product information . There are numerous benefits to aligning embeddings based on inner products in terms of applicability , interpretability , and efficiency . First , since FIPP ’ s alignment between the source and target embeddings is performed on the inner product matrices , XsX T s and XtX T t 2 Rn⇥n , embeddings are not required to be of the same dimensions . This is particularly helpful for transferring relevant knowledge from embeddings trained on large corpuses to smaller/domain-specific embeddings such as in the case of machine translation for low resource languages . Additionally , dimensionality-invariant alignment can be helpful in compressing input representations for memory and computation sensitive settings . Secondly , alignment modifications on pairwise inner products provide more granularity and increased interpretability in comparison to dimension based approaches . This is because FIPP ’ s conversion of source embeddings is explicitly linked to specific changes in inner products between words . Additionally , FIPP allows better control over alignment as it ’ s transfer loss can be altered to only modify angles between specified word pairs . Lastly , since alignment can be parallelized on a word-pair basis we find that FIPP provides improvements in computational efficiency . We conduct a thorough evaluation of FIPP for multilingual embedding alignment on the MUSE word translation task set . In comparison to existing approaches , we find that FIPP is more computationally efficient , can be applied to more general settings , and achieves better word translation performance for dissimilar language pairs . We also discuss how FIPP can be directly applied in other applications such as embedding compression and transfer learning in natural language processing . The rest of this paper is organized as follows . We discuss related work in Section 2 . We introduce our FIPP model in Section 3 . We present experimental results in Section 4 and further discuss findings in Section 5 . Lastly , we discuss related applications of FIPP in Section 6 and conclude our paper in Section 7 . 2 . Related Work In this section , we discuss related work in quantifying semantic shifts in embeddings and the task of multilingual embedding alignment . 2.1 . Distributional Methods for Quantifying Semantic Shifts Prior work has shown monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax ( Hamilton et al. , 2016a ; b ) . In order to find linguistic shifts in different corpora , distributional methods make comparisons on word co-occurence distributions . These approaches have been well studied to classify the introduction of new words and senses ( Jatowt & Duh , 2014 ) . Word embeddings ( Mikolov et al. , 2013b ; Pennington et al. , 2014 ; Bojanowski et al. , 2017 ) map words to representations in a continuous space . By comparing pairwise distances in embeddings trained on separate corpora , one can determine semantic shifts associated with biases and cultural norms ( Gulordava & Baroni , 2011 ; Sagi et al . ; Kim et al. , 2014 ) . Embedding translation has been an active research direction in many fields , e.g . including the Out-Of-Vocabulary problem ( Yang et al. , 2019 ) and neural machine translation ( Zou et al. , 2013 ) . Recently proposed methods extend this notion by comparing all pairs of words and more specifically the pairwise inner products ( Yin et al. , 2018 ) . However , these approaches for quantifying distributional language variations have not yet been explored in the context of shared representation learning or multilingual embeddings . 2.2 . Word Translation with Multilingual Embedding Alignment The problem of word translation from multilingual embeddings has been studied extensively in natural language processing , under both supervised and unsupervised learning settings . For unsupervised word translation ( Artetxe et al. , 2017 ) , one seeks to align word embeddings trained on a source and target embedding , Xs and Xt respectively , such that for any source word , the closest word embedding in the target embedding space is the source word ’ s translation . The supervised learning setting , commonly known as the i ure 1 : FIPP alignment of source and target embeddings , Xs and Xt , to a common representation space . NoteXs is modified using information from Xt and mapped to Rd2 while Xt is unchanged . We show FIPP can be efficiently solved using either low-rank semidefinite approximations or with stochastic gradient descent . Also , we formulate a least squares projection to infer aligned representations for words outside the seed dictionary and present a weighted Procrustes objective which recovers an orthogonal operator that takes into consideration the degree of structural similarity among translation pairs . The method is illustrated in Figure 1 . Compared to previous approaches , FIPP has improved generality , stability , and efficiency . First , since FIPP ’ s alignment between the source and target embeddings is performed on Gram matrices , i.e . XsXTs and XtX T t ∈ Rc×c , embeddings are not required to be of the same dimension and are projected to isomorphic vector spaces . This is particularly helpful for aligning embeddings trained on smaller corpora , such as in low resource domains , or compute-intensive settings where embeddings may have been compressed to lower di ensions . Secondly , alignment modifications made on filtered Gram matrices can incorporate varying con tr ints o alignment at the most granular level , pairwise distances . Lastly , FIPP is asy to implement as it involve only matrix 2 operations , is deterministic , and takes an order of magnitude less time to compute than either the best supervised ( Joulin et al. , 2018b ) or unsupervised approach ( Artetxe et al. , 2018c ) compared against . We conduct a thorough evaluation of FIPP using baselines outlined in Glavaš et al . ( 2019 ) including bilingual lexicon induction with 5K and 1K supervision sets and downstream evaluation on MNLI Natural Language Inference and Ted CLDC Document Classification tasks . The rest of this paper is organized as follows . We discuss related work in Section 2 . We introduce our FIPP model in Section 3 and usage for inference in Section 4 . We present experimental results in Section 5 and further discuss findings in Section 6 . We conclude the paper in Section 7 .

Summary: This paper proposes an approach to align word vector spaces based on a dictionary of pairs (e.g. translations) that retains "common geometric structure" (inner products within each vector space for a subset of word pairs). In essence the approach applies preprocessing to each vector space, finds a projection of one vector spaces to the (lower-dimensional) vector space minimising a combination of a reconstruction and transfer loss, and then aligns the two using weighted procrustes to resolve rotational symmetries.