FILTERED INNER PRODUCT PROJECTION FOR CROSSLINGUAL EMBEDDING ALIGNMENT
Year: 2021

Abstract:
Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space. Recently, these algorithms have been studied in the setting of bilingual lexicon induction where one seeks to align the embeddings of a source and a target language such that translated word pairs lie close to one another in a common representation space. In this paper, we propose a method, Filtered Inner Product Projection (FIPP), for mapping embeddings to a common representation space. As semantic shifts are pervasive across languages and domains, FIPP first identifies the common geometric structure in both embeddings and then, only on the common structure, aligns the Gram matrices of these embeddings. FIPP aligns embeddings to isomorphic vector spaces even when the source and target embeddings are of differing dimensionalities. Additionally, FIPP provides computational benefits in ease of implementation and is faster to compute than current approaches. Following the baselines in Glavaš et al. (2019), we evaluate FIPP in the context of bilingual lexicon induction and downstream language tasks. We show that FIPP outperforms existing methods on the XLING (5K) BLI dataset and the XLING (1K) BLI dataset, when using a self-learning approach, while also providing robust performance across downstream tasks.

heading: 1 INTRODUCTION
text: The problem of aligning sets of embeddings, or high dimensional real valued vectors, is of great interest in natural language processing, with applications in machine translation and transfer learning, and shares connections to graph matching and assignment problems (Grave et al., 2019; Gold & Rangarajan, 1996). Aligning embeddings trained on corpora from different languages has led to improved performance of supervised and unsupervised word and sentence translation (Zou et al., 2013), sequence labeling (Zhang et al., 2016; Mayhew et al., 2017), and information retrieval (Vulić & Moens, 2015). Additionally, linguistic patterns have been studied using embedding alignment algorithms (Schlechtweg et al., 2019; Lauscher & Glavaš, 2019). Embedding alignments have also been shown to improve the performance of multilingual contextual representation models (i.e. mBERT), when used during intialization, on certain tasks such as multilingual document classification (Artetxe et al., 2020) Recently, algorithms using embedding alignments on the input token representations of contextual embedding models have been shown to provide efficient domain adaptation (Poerner et al., 2020). Lastly, aligned source and target input embeddings have been shown to improve the transferability of models learned on a source domain to a target domain (Artetxe et al., 2018a; Wang et al., 2018; Mogadala & Rettinger, 2016).
In the bilingual lexicon induction task, one seeks to learn a transformation on the embeddings of a source and a target language so that translated word pairs lie close to one another in the shared representation space. Specifically, one is given a small seed dictionary D containing c pairs of translated words, and embeddings for these word pairs in a source and a target language, Xs ∈ Rc×d
and Xt ∈ Rc×d. Using this seed dictionary, a transformation is learned on Xs and Xt with the objective that unseen translation pairs can be induced, often through nearest neighbors search.
Previous literature on this topic has focused on aligning embeddings by minimizing matrix or distributional distances (Grave et al., 2019; Jawanpuria et al., 2019; Joulin et al., 2018a). For instance, Mikolov et al. (2013a) proposed using Stochastic Gradient Descent (SGD) to learn a mapping, Ω, to minimize the sum of squared distances between pairs of words in the seed dictionary ‖XDs Ω−XDt ‖2F , which achieves high word translation accuracy for similar languages. Smith et al. (2017) and Artetxe et al. (2017) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the Orthogonal Procrustes problem, Ω∗ = arg minΩ∈O(d) ‖XDs Ω − XDt ‖F where O(d) denotes the group of d dimensional orthogonal matrices. However, these methods usually require the dimensions of the source and target language embeddings to be the same, which often may not hold. Furthermore, due to semantic shifts across languages, it’s often the case that a word and its translation may not co-occur with the same sets of words (Gulordava & Baroni, 2011). Therefore, seeking an alignment which minimizes all pairwise distances among translated pairs results in using information not common to both the source and target embeddings.
To address these problems, we propose Filtered Inner Product Projection (FIPP) for mapping embeddings from different languages to a shared representation space. FIPP aligns a source embedding Xs ∈ Rn×d1 to a target embedding Xt ∈ Rm×d2 and maps vectors in Xs to the Rd2 space of Xt. Instead of word-level information, FIPP focuses on pairwise distance information, specified by the Gram matrices XsXTs and XtX T t , where the rows of Xs and Xt correspond to embeddings for the c pairs of source and target words from the seed dictionary. During alignment, FIPP tries to achieve the following two goals. First, it is desired that the aligned source embedding FIPP(Xs) = X̃s ∈ Rc×d2 be structurally close to the original source embedding to ensure that semantic information is retained and prevent against overfitting on the seed dictionary. This goal is reflected in the minimization of the reconstruction loss: ‖X̃sX̃Ts −XsXTs ‖2F . Second, as the usage of words and their translations vary across languages, instead of requiring X̃s to use all of the distance information fromXt, FIPP selects a filtered setK of word pairs that have similar distances in both the source and target languages: K = {(i, j) ∈ D : {|XsXTs −XtXTt |ij ≤ }. FIPP then minimizes a transfer loss on this set K, the squared difference in distances between the aligned source embeddings and the target embeddings: ∑ (i,j)∈K(X̃s[i]X̃s[j] T −Xt[i]Xt[j]T )2.
055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108
Filtered Inner Product Projection for Multilingual Embedding Alignment
FIPP(Xs) Id2
Rd2
Rd1 Rd2
Xs Xt
Figure 1. FIPP alignment of source and target embeddings, Xs and Xt, to a common representation space. Note that Xs is modified using information from Xt and mapped to Rd2 while Xt is unchanged.
transformations as FIPP’s alignment stems directly from inner product information. There are numerous benefits to aligning embeddings based on inner products in terms of applicability, interpretability, and efficiency.
First, since FIPP’s alignment between the source and target embeddings is performed on the inner product matrices, XsX T s and XtX T t 2 Rn⇥n, embeddings are not required to be of the same dimensions. This is particularly helpful for transferring relevant knowledge from embeddings trained on large corpuses to smaller/domain-specific embeddings such as in the case of machine translation for low resource languages. Additionally, dimensionality-invariant alignment can be helpful in compressing input representations for memory and computation sensitive settings.
Secondly, alignment modifications on pairwise inner products provide more granularity and increased interpretability in comparison to dimension based approaches. This is because FIPP’s conversion of source embeddings is explicitly linked to specific changes in inner products between words. Additionally, FIPP allows better control over alignment as it’s transfer loss can be altered to only modify angles between specified word pairs. Lastly, since alignment can be parallelized on a word-pair basis we find that FIPP provides improvements in computational efficiency.
We conduct a thorough evaluation of FIPP for multilingual embedding alignment on the MUSE word translation task set. In comparison to existing approaches, we find that FIPP is more computationally efficient, can be applied to more general settings, and achieves better word translation performance for dissimilar language pairs.
We also discuss how FIPP can be directly applied in other applications such as embedding compression and transfer
learning in natural language processing.
The rest of this paper is organized as follows. We discuss related work in Section 2. We introduce our FIPP model in Section 3. We present experimental results in Section 4 and further discuss findings in Section 5. Lastly, we discuss related applications of FIPP in Section 6 and conclude our paper in Section 7.
2. Related Work In this section, we discuss related work in quantifying semantic shifts in embeddings and the task of multilingual embedding alignment.
2.1. Distributional Methods for Quantifying Semantic Shifts
Prior work has shown monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax (Hamilton et al., 2016a;b). In order to find linguistic shifts in different corpora, distributional methods make comparisons on word co-occurence distributions. These approaches have been well studied to classify the introduction of new words and senses (Jatowt & Duh, 2014).
Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017) map words to representations in a continuous space. By comparing pairwise distances in embeddings trained on separate corpora, one can determine semantic shifts associated with biases and cultural norms (Gulordava & Baroni, 2011; Sagi et al.; Kim et al., 2014). Embedding translation has been an active research direction in many fields, e.g. including the Out-Of-Vocabulary problem (Yang et al., 2019) and neural machine translation (Zou et al., 2013). Recently proposed methods extend this notion by comparing all pairs of words and more specifically the pairwise inner products (Yin et al., 2018). However, these approaches for quantifying distributional language variations have not yet been explored in the context of shared representation learning or multilingual embeddings.
2.2. Word Translation with Multilingual Embedding Alignment
The problem of word translation from multilingual embeddings has been studied extensively in natural language processing, under both supervised and unsupervised learning settings. For unsupervised word translation (Artetxe et al., 2017), one seeks to align word embeddings trained on a source and target embedding, Xs and Xt respectively, such that for any source word, the closest word embedding in the target embedding space is the source word’s translation.
The supervised learning setting, commonly known as the
i ure 1: FIPP alignment of source and target embeddings, Xs and Xt, to a common representation space. NoteXs is modified using information from Xt and mapped to Rd2 while Xt is unchanged.
We show FIPP can be efficiently solved using either low-rank semidefinite approximations or with stochastic gradient descent. Also, we formulate a least squares projection to infer aligned representations for words outside the seed dictionary and present a weighted Procrustes objective which recovers an orthogonal operator that takes into consideration the degree of structural similarity among translation pairs. The method is illustrated in Figure 1.
Compared to previous approaches, FIPP has improved generality, stability, and efficiency. First, since FIPP’s alignment between the source and target embeddings is performed on Gram matrices, i.e. XsXTs and XtX T t ∈ Rc×c, embeddings are not required to be of the same dimension and are projected to isomorphic vector spaces. This is particularly helpful for aligning embeddings trained on smaller corpora, such as in low resource domains, or compute-intensive
settings where embeddings may have been compressed to lower di ensions. Secondly, alignment modifications made on filtered Gram matrices can incorporate varying con tr ints o alignment at the most granular level, pairwise distances. Lastly, FIPP is asy to implement as it involve only matrix
2
operations, is deterministic, and takes an order of magnitude less time to compute than either the best supervised (Joulin et al., 2018b) or unsupervised approach (Artetxe et al., 2018c) compared against.
We conduct a thorough evaluation of FIPP using baselines outlined in Glavaš et al. (2019) including bilingual lexicon induction with 5K and 1K supervision sets and downstream evaluation on MNLI Natural Language Inference and Ted CLDC Document Classification tasks. The rest of this paper is organized as follows. We discuss related work in Section 2. We introduce our FIPP model in Section 3 and usage for inference in Section 4. We present experimental results in Section 5 and further discuss findings in Section 6. We conclude the paper in Section 7.

heading: 2 RELATED WORK

heading: 2.1 DISTRIBUTIONAL METHODS FOR QUANTIFYING SEMANTIC SHIFTS
text: Prior work has shown that monolingual text corpora from different communities or time periods exhibit variations in semantics and syntax (Hamilton et al., 2016a;b). Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017) map words to representations in a continuous space with the objective that the inner product between any two words representations is approximately proportional to their probability of co-occurrence. By comparing pairwise distances in monolingual embeddings trained on separate corpora, one can quantify semantic shifts associated with biases, cultural norms, and temporal differences (Gulordava & Baroni, 2011; Sagi et al., 2011; Kim et al., 2014). Recently proposed metrics on embeddings compare all pairwise inner products of two embeddings, E and F , of the form ‖EET − FFT ‖F (Yin et al., 2018). While these metrics have been applied in quantifying monolingual semantic variation, they have not been explored in context of mapping embeddings to a common representation space or in multilingual settings.

heading: 2.2 CROSSLINGUAL EMBEDDING ALIGNMENT
text: The first work on this topic is by Mikolov et al. (2013a) who proposed using Stochastic Gradient Descent (SGD) to learn a mapping, Ω, to minimize the sum of squared distances between pairs of words in the seed dictionary ‖XsΩ−Xt‖2F , which achieves high word translation accuracy for similar languages. Smith et al. (2017) and Artetxe et al. (2017) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the Orthogonal Procrustes problem, Ω∗ = arg minΩ∈O(d) ‖XsΩ−Xt‖F . Dinu & Baroni (2015) worked on corrections to the "hubness" problem in embedding alignment, where certain word vectors may be close to many other word vectors, arising due to nonuniform density of vectors in the Rd space. Smith et al. (2017) proposed the inverted softmax metric for inducing matchings between words in embeddings of different languages. Artetxe et al. (2016) studied the impact of normalization, centering and orthogonality constraints in linear alignment functions. Jawanpuria et al. (2019) presented a composition of orthogonal operators and a Mahalanobis metric of the form UBV T , U, V T ∈ O(d), B 0 to account for observed correlations and moment differences between dimensions (Søgaard et al., 2018). Joulin et al. (2018a) proposed an alignment based on neighborhood information to account for differences in density and shape of embeddings in their respective Rd spaces. Artetxe et al. (2018c) outlined a framework which unifies many existing alignment approaches as compositions of matrix operations such as Orthogonal mappings, Whitening, and Dimensionality Reduction. Nakashole & Flauger (2018) found that locally linear maps vary between different neighborhoods in bilingual embedding spaces which suggests that nonlinearity is beneficial in global alignments. Nakashole (2018) proposed an alignment method using neighborhood sensitive maps which shows strong performance on dissimilar language pairs. Patra et al. (2019) proposed a novel hub filtering method and a semi-supervised alignment approach based on distributional matching. Mohiuddin et al. (2020) learned a non-linear mapping in the latent space of two independently pre-trained autoencoders which provide strong performance on well-studied BLI tasks. A recent method, most similar to ours, Glavaš & Vulić (2020) utilizes non-linear mappings to find a translation vector for each source and target embedding using the cosine similarity and euclidean distances between nearest neighbors and corresponding translations. In the unsupervised setting, where a bilingual seed dictionary is not provided, approaches using adversarial learning, distributional matching, and noisy self-supervision have been used to concurrently learn a matching and an alignment between embeddings (Cao et al., 2016; Zhang et al., 2017; Hoshen
& Wolf, 2018; Grave et al., 2019; Artetxe et al., 2017; 2018b; Alvarez-Melis & Jaakkola, 2018). Discussion on unsupervised approaches is included in Appendix Section I.

heading: 3 FILTERED INNER PRODUCT PROJECTION (FIPP)

heading: 3.1 FILTERED INNER PRODUCT PROJECTION OBJECTIVE
text: In this section, we introduce Filtered Inner Product Projection (FIPP), a method for aligning embeddings in a shared representation space. FIPP aligns a source embedding Xs ∈ Rn×d1 to a target embedding Xt ∈ Rm×d2 and projects vectors in Xs to X̃s ∈ Rn×d2 . Let Xs ∈ Rc×d1 and Xt ∈ Rc×d2 be the source and target embeddings for pairs in the seed dictionary D, |D| = c min(n,m). FIPP’s objective is to minimize a linear combination of a reconstruction loss, which regularizes changes in the pairwise inner products of the source embedding, and a transfer loss, which aligns the source and target embeddings on common portions of their geometries.
min X̃s∈Rc×d2 Reconstruction Loss︷ ︸︸ ︷ ‖X̃sX̃Ts −XsXTs ‖2F +λ ‖∆ ◦ (X̃sX̃Ts −XtXTt )‖2F︸ ︷︷ ︸
Transfer Loss
(1)
where λ, ∈ R+ are tunable scalar hyperparameters whose effects are discussed in Section E, ◦ is the Hadamard product, and ∆ is a binary matrix discussed in 3.1.2.

heading: 3.1.1 RECONSTRUCTION LOSS
text: Due to the limited, noisy supervision in our problem setting, an alignment should be regularized against overfitting. Specifically, the aligned space needs to retain a similar geometric structure to the original source embeddings; this has been enforced in previous works by ensuring that alignments are close to orthogonal mappings (Mikolov et al., 2013a; Joulin et al., 2018a; Jawanpuria et al., 2019). As X̃s and Xs can be of differing dimensionality, we check structural similarity by comparing pairwise inner products, captured by a reconstruction loss known as the PIP distance or Global Anchor Metric: ‖X̃sX̃Ts −XsXTs ‖2F (Yin & Shen, 2018; Yin et al., 2018). Theorem 1. Suppose E ∈ Rn×d, F ∈ Rn×d are two matrices with orthonormal columns and Ω∗ = arg minΩ∈O(d) ‖EΩ− F‖F . It follows that (Yin et al., 2018):
‖EΩ∗ − F‖ ≤ ‖EET − FFT ‖ ≤ √ 2‖EΩ∗ − F‖. (2)
This metric has been used in quantifying semantic shifts and has been shown Yin et al. (2018) to be equivalent to the residual of the Orthogonal Procrustes problem up to a small constant factor, as seen in Theorem 1. Note that the PIP distance is invariant to orthogonal operations such as rotations which are known to be present in unaligned embeddings.

heading: 3.1.2 TRANSFER LOSS
text: In aligning Xs to Xt, we should seek to only utilize common geometric information between the two embedding spaces. We propose a simple approach, although FIPP can admit other forms of filtering mechanisms, denoted as inner product filtering where we only utilize pairwise distances similar in both embedding spaces as defined by a threshold . Specifically, compute a matrix ∆ ∈ {0, 1}c×c where ∆ ij is an indicator on whether |Xs,iXTs,j −Xt,iXTt,j | < . In this form, is a hyperparameter which determines how close pairwise distances must be in the source and target embeddings to be deemed similar. We then define a transfer loss as being the squared difference between the converted source embedding X̃s and target embedding Xt, but only on pairs of words in K: ‖∆ ◦ (X̃sX̃Ts −XtXTt )‖2F , where ◦ is the Hadamard product. The FIPP objective is a linear combination of the reconstruction and transfer losses.

heading: 3.2 APPROXIMATE SOLUTIONS TO THE FIPP OBJECTIVE

heading: 3.2.1 SOLUTIONS USING LOW-RANK SEMIDEFINITE APPROXIMATIONS
text: Denote the Gram matrices Gs , XsXTs , G t , XtXTt and G̃ s , X̃sX̃Ts .
Lemma 2. The matrix G∗ which minimizes the FIPP objective for a fixed λ and has entries:
G∗ij =
{ (XsX T s )ij+λ(XtX T t )ij
1+λ , if (i, j) ∈ K (XsX T s )ij , otherwise
(3)
Proof. For a fixed λ and , LFIPP,λ, (X̃sX̃Ts ) can be decomposed as follows:
LFIPP,λ, (X̃sX̃Ts ) =‖X̃sX̃Ts −XsXTs ‖2F + λ‖∆ ◦ (X̃sX̃Ts −XtXTt )‖2F = ∑ i,j∈K ((G̃sij −Gsij)2 + λ(G̃sij −Gtij)2) + ∑ i,j /∈K (G̃sij −Gsij)2 (4)
By taking derivatives with respect to G̃sij , the matrix G ∗ which minimizes LFIPP,λ, (·) is:
G∗ = arg min X̃sX̃Ts ∈Rc×c
LFIPP,λ, (X̃sX̃Ts ), G∗ij = { (XsX T s )ij+λ(XtX T t )ij
1+λ , if (i, j) ∈ K (XsX T s )ij , otherwise
(5)
We now have the matrix G∗ ∈ Rc×c which minimizes the FIPP objective. However, for G∗ to be a valid Gram matrix, it is required that G∗ ∈ Sc×c+ , the set of symmetric Positive Semidefinite matrices. Additionally, to recover an X̃s ∈ Rc×d2 such that X̃sX̃Ts = G∗, we must have Rank(G∗) ≤ d2. Note that G∗ is symmetric by construction since the set K is commutative and Gs, Gt are symmetric. However, G∗ is not necessarily positive semidefinite nor is it necessarily true that Rank(G∗) ≤ d2. Therefore, to recover an aligned embedding X̃s ∈ Rc×d2 , we perform a rank-constrained semidefinite approximation to find minX̃s∈Rc×d2 ‖X̃sX̃Ts −G∗‖F . Theorem 3. Let G∗ = QΛQT be the Eigendecomposition of G∗. A matrix X̃s ∈ Rm×d2 which minimizes ‖X̃sX̃Ts −G∗‖F is given by ∑d2 i=1,λi≥0 λ 1 2 i qi, where λi and qi are the i
th largest eigenvalue and corresponding eigenvector.
Proof. Since G∗ ∈ Sc×c, its Eigendecomposition is G∗ = QΛQT where Q is orthonormal. Let λ̃, q̃ be the d2 largest nonnegative eigenvalues in Λ and their corresponding eigenvectors; additionally, denote the complementary eigenvalues and associated eigenvectors as λ̃⊥ = Λ \ λ̃, q̃⊥ = Q \ q̃. Using the Eckart–Young–Mirsky Theorem for the Frobenius norm (Kishore Kumar & Schneider, 2017), note that for G ∈ Sc×c+ , Rank(G) ≤ d2; ‖G∗ −G‖F ≥ ‖q̃⊥λ̃⊥q̃⊥T ‖F = ∑ λi∈λ̃⊥ λ 1 2 i and that ‖G∗ −G‖F is minimized for G = q̃λ̃q̃T . Using this result, we can recover X̃s:
arg min G∈Sc×c+ ,
Rank(G)≤d2
‖G∗ −G‖F = ∑ λi∈λ̃ (λ 1 2 i qi)(λ 1 2 i qi) T = X̃sX̃ T s (6)
Using the above matrix approximation, we find our aligned embedding X̃s = ∑ λi∈λ̃ λ 1 2 i qi, a minimizer of ‖X̃sX̃Ts −G∗‖F .
Due to the rank constraint onG, we are only interested in the d2 largest eigenvalues and corresponding eigenvectors which incurs a complexity of O(d2c2) using power iteration (Panju, 2011).

heading: 3.2.2 SOLUTIONS USING STOCHASTIC GRADIENT DESCENT
text: Alternatively, solutions to the FIPP objective can be obtained using Stochastic Gradient Descent (SGD). This requires defining a single variable X̃s ∈ Rc×d2 over which to optimize. We find that the solutions obtained after convergence of SGD are close, with respect to the Frobenius norm, to those obtained with low rank PSD approximations up to a rotation. However, the complexity of solving FIPP using SGD is O(Tc2), where T is the number of training epochs. Empirically we find T > d2 for SGD convergence and, as a result, this approach incurs a complexity greater than that of low-rank semidefinite approximations.

heading: 3.3 ISOTROPIC PREPROCESSING
text: Common preprocessing steps used by previous approaches (Joulin et al., 2018a; Artetxe et al., 2018a), involve normalizing the rows of Xs,Xt to have an `2 norm of 1 and demeaning columns. The transfer loss of the FIPP objective makes direct comparisons on the Gram matrices, XsXTs and XtX T t , of the source and target embeddings for words in the seed dictionary. To reduce the influence of dimensional biases between Xs and Xt and ensure words are weighted equally during alignment, it is desired that Xs and Xt be isotropic - i.e. Z(c) = ∑ i∈Xs exp c
TXs[i] is approximately constant for any unit vector c (Arora et al., 2016). Mu & Viswanath (2018) find that a first order approximation to enforce isotropic behavior is achieved by column demeaning while a second order approximation is obtained by the removal of the top PCA components. In FIPP, we apply this simple pre-processing approach by removing the top PCA component of Xs and Xt. Empirically, the distributions of inner products between a source and target embedding can differ substantially when not preprocessed, rendering a substandard alignment, which is discussed further in the Appendix Section G.

heading: 4 INFERENCE AND ALIGNMENT

heading: 4.1 INFERENCE WITH LEAST SQUARES PROJECTION
text: To infer aligned source embeddings for words outside of the supervision dictionary, we make the assumption that source words not used for alignment should preserve their distances to those in the seed dictionary in their respective spaces, i.e., XsXTs ≈ X̃sX̃Ts . Using this assumption, we formulate a least squares projection (Boyd & Vandenberghe, 2004) on an overdetermined system of equations to recover X̃Ts : X̃ T s = (X̃ T s X̃s) −1X̃Ts XsX T s .

heading: 4.2 WEIGHTED ORTHOGONAL PROCRUSTES
text: As X̃s ∈ Rc×d2 has been optimized only with concern for its inner products, X̃s must be rotated so that it’s basis matches that of Xt. We propose a weighted variant of the Orthogonal Procrustes solution to account for differing levels of translation uncertainty among pairs in the seed dictionary, which may arise due to polysemy, semantic shifts and translation errors. In Weighted Least Squares problems, an inverse variance-covariance weighting W is used (Strutz, 2010; Brignell et al., 2015) to account for differing levels of measurement uncertainty among samples. We solve a weighted Procrustes objective, where measurement error is approximated as the transfer loss for each translation pair, W−1ii = ‖X̃s[i]XTs −Xt[i]XTt ‖2:
SVD((WXt)TWX̃s) =UΣV T ,ΩW = arg min Ω∈O(d2) ‖W (X̃sΩ−Xt)‖2F = UV T , (7)
where O(d2) is the group of d2 × d2 orthogonal matrices. The rotation ΩW is then applied to X̃s.

heading: 5 EXPERIMENTATION
text: In this section, we report bilingual lexicon induction results from the XLING dataset and downstream experiments performed on the MNLI Natural Language Inference and TED CLDC tasks.

heading: 5.1 XLING BILINGUAL LEXICON INDUCTION
text: The XLing BLI task dictionaries constructed by Glavaš et al. (2019) include all 28 pairs between 8 languages in different language families, Croatian (HR), English (EN), Finnish (FI), French (FR), German (DE), Italian (IT), Russian (RU), and Turkish (TR). The dictionaries use the same vocabulary across languages and are constructed based on word frequency, to reduce biases known to exist in other datasets (Kementchedjhieva et al., 2019). We evaluate FIPP across the all language pairs using a supervision dictionary of size 5K and 1K. On 5K dictionaries, FIPP outperforms other approaches on 22 of 28 language pairs. On 1K dictionaries, FIPP outperforms other approaches on 23 of 28
language pairs when used along with a Self-Learning Framework (denoted as FIPP + SL) discussed in Appendix Section C. Our code for FIPP is open-source and available on Github 1.

heading: 5.1.1 BILINGUAL LEXICON INDUCTION: XLING 1K

heading: 5.1.2 BILINGUAL LEXICON INDUCTION: XLING 5K

heading: 5.2 DOWNSTREAM EVALUATIONS

heading: 5.2.1 TED CLDC - DOCUMENT CLASSIFICATION
text: The TED-CLDC corpus (Hermann & Blunsom, 2014) contains cross-lingual documents across 15 topics and 12 language pairs. Following the evaluation of Glavaš et al. (2019), a simple CNN based classifier is trained and evaluated over each topic for language pairs included in our BLI evaluations (EN-DE, EN-FR, EN-IT, EN-RU, and EN-TR). RCSLS outperforms other methods overall and on DE, FR, and RU while FIPP performs best on IT and TR.
1https://github.com/vinsachi/FIPPCLE

heading: 5.2.2 MNLI - NATURAL LANGUAGE INFERENCE
text: The multilingual XNLI corpus introduced by Conneau et al. (2018), based off the English only MultiNLI (Williams et al., 2018), includes 5 of the 8 languages used in BLI: EN, DE, FR, RU, and TR. We perform the same evaluation as Glavaš et al. (2019) by training an ESIM model (Chen et al., 2017) using EN word embeddings from a shared EN-L2 embedding space for L2 ∈ {DE, FR, RU, TR}. The trained model is then evaluated without further training on the L2 XNLI test set using L2 embeddings from the shared space. The bootstrap Procrustes approach (Glavaš et al., 2019) outperforms other methods narrowly while RCSLS performs worst despite having high BLI accuracy.

heading: 6 DISCUSSION

heading: 6.1 RUNTIME COMPARISON
text: Method FIPP FIPP+SL VecMap RCSLS Proc-B
CPU 23s - 15,896s 387s 885s GPU - 176s 612s - -
method which performs best out of the compared approaches on 1K seed dictionaries. The average execution time of alignment on 3 runs of the ’EN-DE’ XLING 5K dictionary is provided in Table 4. The implementation used for RCSLS is from Facebook’s fastText repo 2 with default parameters and the VecMap implementation is from the author’s repo 3 with and without the ’cuda’ flag. Proc-B is implemented from the XLING-Eval repo 4 with default parameters. Hardware specifications are 2 Nvidia GeForce GTX 1080 Ti GPUs, 12 Intel Core i7-6800K processors, and 112GB RAM.
2https://github.com/facebookresearch/fastText/tree/master/alignment 3https://github.com/artetxem/vecmap 4https://github.com/codogogo/xling-eval

heading: 6.2 ALIGNMENT OF EMBEDDINGS OF DIFFERING DIMENSIONALITY

heading: 6.2.1 COMPARISON OF GRAM MATRIX SPECTRUM
text: In this section, we compute alignments with previous methods, a modification of Procrustes and RCSLS (Joulin et al., 2018b), and FIPP on an English Embedding ∈ R200 to a German Embedding ∈ R300. In Figure 2, we plot the spectrum of the Gram Matrices for the aligned embeddings from each method and the target German Embedding. While the FIPP aligned embedding is isomorphic to the target German Embedding ∈ R300, other methods produce a rank deficient aligned embedding whose spectrum deviates from the target embedding. The null space of aligned source embeddings, for methods other than FIPP, is of dimension d2−d1. We note that issues can arise when learning and transferring models on embeddings from different rank vector spaces. For regularized models transferred from the source to the target space, at least d2−d1 column features of the target embedding will not be utilized. Meanwhile, models
transferred from the target space to the source space will exhibit bias associated with model parameters corresponding to the source embedding’s null space.

heading: 6.2.2 BLI PERFORMANCE
text: d1 Linear CCA FIPP

heading: 300 0.544 0.542 0.590

heading: 250 0.524 0.529 0.574

heading: 200 0.486 0.501 0.543
text: of all methods decreases as d2 − d1 increases, the relative performance gap between FIPP and other approaches is maintained.

heading: 7 CONCLUSION
text: In this paper, we introduced Filtered Inner Product Projection (FIPP), a method for aligning multiple embeddings to a common representation space using pairwise inner product information. FIPP accounts for semantic shifts and aligns embeddings only on common portions of their geometries. Unlike previous approaches, FIPP aligns embeddings to equivalent rank vector spaces regardless of their dimensions. We provide two methods for finding approximate solutions to the FIPP objective and show that it can be efficiently solved even in the case of large seed dictionaries. We evaluate FIPP on the task of bilingual lexicon induction using the XLING (5K) dataset and the XLING (1K) dataset, on which it achieves state-of-the-art performance on most language pairs. Our method provides a novel efficient approach to the problem of shared representation learning.
5https://scikit-learn.org/stable/modules/generated/sklearn.cross_decomposition.CCA.html

heading: A FULL BLI EXPERIMENTATION - XLING (1K) AND XLING (5K)
text: In Table 6 below, we provide experimental results for FIPP using 1K seed dictionaries. Unlike in case of a 5K supervision set, FIPP is outperformed by the bootstrapped Procrustes method (Glavaš et al., 2019) and the unsupervised VecMap approach (Artetxe et al., 2018c). However, the addition of a self learning framework, detailed in Section C, to FIPP (FIPP + SL) results in performance greater than compared methods albeit at the cost of close to a 8x increase in computation time, from 23s to 176s, and the requirement of a GPU. Other well performing methods for XLING 1K, Proc-B (Glavaš et al., 2019) and VecMap (Artetxe et al., 2018c), also use self-learning frameworks; further analysis is required to understand the importance of self-learning frameworks in the case of small (or no) seed dictionary.
The methods compared against were originally proposed in: VecMap (Artetxe et al., 2018c), MUSE (Lample et al., 2018), ICP (Hoshen & Wolf, 2018), CCA (Faruqui & Dyer, 2014), GWA (AlvarezMelis & Jaakkola, 2018), PROC (Mikolov et al., 2013a), PROC-B (Glavaš et al., 2019), DLV (Ruder et al., 2018), and RCSLS (Joulin et al., 2018b).

heading: B EFFECTS OF RUNNING MULTIPLE ITERATIONS OF FIPP OPTIMIZATION
text: Although other alignment approaches (Joulin et al., 2018b; Artetxe et al., 2018c) run multiple iterations of their alignment objective, we find that running multiple iterations of the FIPP Optimization does not improve performance. We run between 1 and 5 iterations of the FIPP objective. For 1K seed dictionaries, 26/28 language pairs perform best with 1 iteration while 26/28 language pairs perform best with 1 iteration for 5K seed dictionaries. In the case of 1K seed dictionaries, (EN, FI) with 2 iterations and (RU, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.001. For 5K seed dictionaries, (EN, FI) with 3 iterations and (EN, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.002. Due to limited set of language pairs on which performance improvements were achieved, these results have not been included in Tables 1 and 6.

heading: C SELF-LEARNING FRAMEWORK
text: A self-learning framework, used for augmenting the number of available training pairs without direct supervision, has been found to be effective (Glavaš et al., 2019; Artetxe et al., 2018c) both in the case of small seed dictionaries or in an unsupervised setting. We detail a self-learning framework which improves the BLI performance of FIPP in the XLING 1K setting but not in the case of 5K seed pairs.
Cos. Sim. Rank English word French word
1 oversee superviser 2 renegotiation renégociation 3 inform informer 4 buy acheter 5 optimize optimiser 6 participate participer 7 conceptualization conceptualisation 8 internationalization internationalisation 9 renegotiate renégocier
10 interactivity interactivité
Let Xs ∈ Rc×d1 and Xt ∈ Rc×d2 be the source and target embeddings for pairs in the seed dictionary D. Additionally, Xs ∈ Rn×d1 and Xt ∈ Rm×d2 are the source and target embeddings for the entire vocabulary. Assume all vectors have been normalized to have an `2 norm of 1. Each source vector Xs,i and target vector Xt,j can be rewritten as d dimensional row vectors of inner products with their corresponding seed dictionaries: As,i = Xs,iX T s ∈ R1×c and As,j = Xt,jX T s ∈ R1×c.
product for vectors with norm of 1, in this d dimensional space as (i, j) = arg maxj As,iA T t,j . For XLING BLI 1K experiments, we find the 14K (i, j) pairs with the largest cosine similarity and augment our seed dictionaries with these pairs. In Table 7, the top 10 translation pairings, sorted by cosine similarity, obtained using this self-learning framework are shown for the English to French (EN-FR) language pair.

heading: D COMPARISON OF FIPP SOLUTION TO ORTHOGONAL ALIGNMENTS
text: In this section, we conduct experimentation to quantify the degree to which the FIPP alignment differs from an orthogonal solution and compare performance on monolingual tasks before and after FIPP alignment.
D.1 DEVIATION OF FIPP FROM THE CLOSEST ORTHOGONAL SOLUTION
For each language pair, we quantify the deviation of FIPP from an Orthogonal solution by first calculating the FIPP alignment before rotation, X̃s, on the seed dictionary. We then compute the relative deviation of the FIPP alignment with the closest orthogonal alignment on the original embedding, Xs. This is equal to D = ‖X̃s−Ω∗Xs‖F ‖Xs‖F where Ω
∗ = arg minΩ∈O(d2) ‖XsΩ − X̃s‖F . The average of these deviations for 1K seed dictionaries is 0.292 and for 5K seed dictionaries is 0.115. Additionally, the 3 language pairs with the largest and smallest deviations from orthogonal solutions are presented in the Table below. We find that in most cases, small deviations from orthogonal solutions are observed between languages in the same language family (i.e. Indo-European -> IndoEuropean) while those in different Language families tend to have larger deviations (i.e. Turkic -> Indo-European). A notable exception to this observation is English and Finnish which belong to different language families, Indo-European and Uralic respectively, yet have small deviations in their FIPP solution compared to an orthogonal alignment.
D.2 EFFECT OF INNER PRODUCT FILTERING ON WORD-LEVEL ALIGNMENT
In FIPP, Inner Product Filtering is used to find common geometric information by comparing pairwise distances between a source and target language. To illustrate this step with translation word pairs, in the Table below we show the 5 words with the largest and smallest fraction of zeros, i.e. the "least and most filtered", in the binary filtering matrix ∆ during alignment between English (En) and Italian (It). The words which are least filtered tend to have an individual word sense, i.e. proper nouns, while those which are most filtered are somewhat ambiguous translations. For instance, while the English word "securing" can be translated to the Italian word "fissagio", depending on the context the Italian words "garantire", "assicurare" or "fissare" may be more appropriate.
D.3 MONOLINGUAL TASK PERFORMANCE OF ALIGNED EMBEDDINGS
As FIPP does not perform an orthogonal transform, it modifies the inner products of word vectors in the source embedding which can impact performance on monolingual task accuracy. We evaluate the aligned embedding learned using FIPP, X̃s, on monolingual word analogy tasks and compare these results to the original fastText embeddings Xs. In Table 3, we compare monolingual English word analogy results for English embeddings X̃s which have been aligned to a Turkish target embedding using FIPP. Evaluation of the aligned and original source embeddings on multiple English word analogy experiments show that aligned FIPP embeddings retain performance on monolingual analogy tasks.

heading: E EFFECT OF HYPERPARAMETERS ON BLI PERFORMANCE
text: In our experiments, we tune the hyperparameters and λ which signify the level of discrimination in the inner product filtering step and the weight of the transfer loss respectively. When tuning, we account for the sparsity of ∆ by scaling λ in the transfer loss by γ = c 2
NNZ(∆ ) where NNZ(∆ )
is the number of nonzeroes in ∆ . Values of used in tuning were [0.01, 0.025, 0.05, 0.10, 0.15] and scaled values of λ used in tuning were [0.25, 0.5, 0.75, 1.0, 1.25]. Values of ( , λ) which are close to one another result in similar performance and results are deterministic across reruns of the same experiment. As no validation set is provided, hyperparameters are tuned by holding out 20% of the training set.

heading: F DIFFERENCE IN SOLUTIONS OBTAINED USING LOW RANK APPROXIMATIONS AND SGD
text: Figure 3: Comparison of FIPP Objective Loss for (FI-FR) for solutions obtained using SGD vs LRA
As detailed in Section 3, solutions to FIPP can be calculated either using Low Rank Approximations or Stochastic Gradient Descent (SGD). In this section, we show the error on the FIPP objective for SGD trained over 10,000 epochs on alignment of a Finnish (FI) embedding to a French (FR) embedding. The Adam (Kingma & Ba, 2015) optimizer is used with a learning rate of 1e−3 and the variable being optimized X̃SGDs is initialized to the original Finnish embedding Xs. We find that the SGD solution X̃SGDs approaches the error of the Low Rank Approximation X̃LRAs , which is the global minima of the FIPP objective, as shown in the Figure below but is not equivalent. While the deviation between SGD and the Low Rank Approximation, DSGD,LRA =
‖X̃SGDs −X̃LRAs ‖F ‖X̃LRAs ‖F
= 0.041, is smaller than the deviation between the original embedding and the Low Rank Approximation, DOrig,LRA =
‖Xs−X̃LRAs ‖F ‖X̃LRAs ‖F = 0.343 we note that the solutions are close but not equivalent.

heading: G EFFECTS OF PREPROCESSING ON INNER PRODUCT DISTRIBUTIONS
text: We plot the distributions of inner products, entries of XsXTs and XtX T t , for English (En) and Turkish (Tr) words in the XLING 5K training dictionary before and after preprocessing in Figure below. All embeddings used in experimentation are fastText word vectors trained on Wikipedia (Bojanowski et al., 2017). Since inner products between Xs and Xt are compared directly, the isotropic preprocessing utilized in FIPP is necessary for removing biases caused by variations in scaling, shifts, and point density across embedding spaces.

heading: H COMPLEXITY ANALYSIS
text: In this section, we provide the computational and space complexity of the FIPP method as described in the paper for computing X̃s. We split the complexity for each step in the method. We leave out steps (i.e. preprocessing) which do not contribute significantly to the runtime or memory footprint. The computational complexity of matrix multiplication between two matrices A1A2, where A1 ∈ Rm×nA2 ∈ Rn×p, is denoted as MM(m,n, p) which is upper bounded by 2mnp operations.
H.1 FIPP OPTIMIZATION
The complexity for solving the FIPP objective is detailed below:
Space Complexity = 4c2︸︷︷︸ ∆ , Gs, Gt, G̃s + 3cd2︸︷︷︸ Xs, Xt, X̃s
Time Complexity = O(d2c2)︸ ︷︷ ︸ Compute X̃sX̃Ts w/ Power Iteration + 3c2︸︷︷︸ ‖X̃sX̃Ts −XsXTs ‖F + 5c2︸︷︷︸ λ‖∆ ◦ (X̃sX̃Ts −XtXTt )‖F = O(d2c2) (8)
H.2 SVD ALIGNMENT AND LEAST SQUARE PROJECTION
The complexity of the alignment using the Orthogonal Procrustes solution and the Least Squares Projection is as follows:
Space Complexity = 3c2︸︷︷︸ UΣV T + cn︸︷︷︸ S + nd2︸︷︷︸ X̃Ts
Time Complexity =MM(c, d2, d2) +MM(d2, d2, d2)︸ ︷︷ ︸ X̃sV U T + d32︸︷︷︸ SV D(XTt X̃s)
+MM(c, d1, n) +MM(c, d2, c) + c 3 +MM(d2, c, n)︸ ︷︷ ︸
Least Squares
≤2(cd22 + 3
2 d32 + cd1n+ c 2d2 + cd2n) + c 3
(9)
H.3 DISCUSSION
In performing our analysis, we note that the majority of operations performed are quadratic in the training set size c. While we incur a time complexity of O(c3) during our Least Squares Projection due to the matrix inversion in the normal equation, this inversion is a one time cost. The space complexity of FIPP is O(c2) which is tractable as c is at most 5K. Empirically, FIPP is fast to compute taking less than 30 seconds for a seed dictionary of size 5K which is more efficient than competing methods.

heading: I RELATED WORKS: UNSUPERVISED ALIGNMENT METHODS
text: Cao et al. (2016) studied aligning the first two moments of sets of embeddings under Gaussian distribution assumptions. In Zhang et al. (2017), an alignment between embeddings of different languages is found by matching distributions using an adversarial autoencoder with an orthogonal regularizer. Artetxe et al. (2017) proposes an alignment approach which jointly bootstraps a small seed dictionary and learns an alignment in a self-learning framework. Hoshen & Wolf (2018) first projects embeddings to a subspace spanned by the top p principle components and then learns an alignment by matching the distributions of the projected embeddings. Grave et al. (2019) proposes an unsupervised variant to the Orthogonal Procrustes alignment which jointly learns an orthogonal transform Ω ∈ Rd×d and an assignment matrix P ∈ {0, 1}n×n to minimize the Wasserstein distance between the embeddings subject to an unknown rotation. Three approaches utilize the Gram matrices of embeddings in computing alignment initializations and matchings. Artetxe et al. (2018b) studied the alignment of dissimilar language pairs using a Gram matrix based initialization and robust selflearning. Alvarez-Melis & Jaakkola (2018) proposed an Optimal Transport based approach using the Gromov-Wasserstein distance, GW (C,C ′, p, q) = minΓ∈Π(p,q) ∑ i,j,k,l L(Cik, C ′ jl)ΓijΓkl where C,C ′ are Gram matrices for normalized embeddings and Γ is an assignment. Aldarmaki et al. (2018) learns a unsupervised linear mapping between a source and target language with a loss at each iteration equal to the sum of squares of proposed source and target Gram matrices.

heading: J ABLATION STUDIES
text: J.1 AGGREGATE MAP PERFORMANCE FOR BLI TASKS
We provide an ablation study to quantify improvements associated with the three modifications of our alignment approach compared to a standard Procrustes alignment: (i) isotropic pre-processing (IP), (ii) inner product filtering (IPF) and (iii) weighted procrustes objectives (WP), on the XLING 1K and 5K BLI tasks. For seed dictionaries of size 1K, improvements associated with each portion of FIPP are approximately the same while for seed dictionaries of size 5K,
a larger improvement is obtained due to isotropic pre-processing than inner product filtering or weighted procrustes alignment.
J.2 PREPROCESSING ABLATION
We perform an ablation study to understand the impact of preprocessing on XLING 1K and 5K BLI performance both on Procrustes and FIPP. Additionally, we compare the isotropic preprocessing used in our previous experimentation with iterative normalization, a well performing preprocessing method proposed by Zhang et al. (2017).
For training dictionaries of size 1K, iterative normalization and isotropic preprocessing before running procrustes result in equivalent aggregate MAP performance of 0.316. We find that iterative normalization and isotropic preprocessing each achieve better performance on 11 of 28 and 12 of 28 language pairs respectively.
Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 1K training dictionaries (0.344) when compared to iterative normalization (0.331). In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.
Method Proc. + IN Proc. + IP FIPP + IN FIPP + IP
DE-FI 0.278 0.264 0.291 0.296 DE-FR 0.421 0.422 0.438 0.463 DE-HR 0.240 0.239 0.264 0.268 DE-IT 0.452 0.458 0.468 0.482 DE-RU 0.336 0.331 0.304 0.359 DE-TR 0.191 0.182 0.212 0.215 EN-DE 0.484 0.490 0.490 0.513 EN-FI 0.301 0.299 0.304 0.314 EN-FR 0.578 0.585 0.582 0.601 EN-HR 0.250 0.258 0.261 0.275 EN-IT 0.562 0.563 0.575 0.591 EN-RU 0.372 0.375 0.381 0.399 EN-TR 0.262 0.267 0.275 0.292 FI-FR 0.245 0.245 0.256 0.274 FI-HR 0.214 0.214 0.239 0.243 FI-IT 0.270 0.272 0.287 0.309 FI-RU 0.249 0.251 0.274 0.285 HR-FR 0.255 0.251 0.270 0.283 HR-IT 0.273 0.273 0.296 0.318 HR-RU 0.285 0.279 0.310 0.318
For training dictionaries of size 5K, iterative normalization and isotropic preprocessing before running procrustes result in comparable aggregate MAP performances of 0.422 and 0.424 respectively. We find that iterative normalization and isotropic preprocessing each achieve better performance on 8 of 28 and 16 of 28 language pairs respectively.
Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 5K training dictionaries (0.442) when compared to iterative normalization (0.425). In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.
Method Proc. + IN Proc. + IP FIPP + IN FIPP + IP
DE-FI 0.378 0.370 0.383 0.389 DE-FR 0.519 0.526 0.521 0.543 DE-HR 0.345 0.344 0.349 0.360 DE-IT 0.523 0.523 0.522 0.533 DE-RU 0.432 0.428 0.435 0.449 DE-TR 0.309 0.300 0.314 0.321 EN-DE 0.562 0.574 0.562 0.590 EN-FI 0.420 0.424 0.425 0.439 EN-FR 0.660 0.667 0.660 0.679 EN-HR 0.361 0.370 0.360 0.382 EN-IT 0.641 0.642 0.643 0.649 EN-RU 0.483 0.489 0.482 0.502 EN-TR 0.370 0.370 0.368 0.407 FI-FR 0.377 0.383 0.386 0.407 FI-HR 0.315 0.316 0.318 0.335 FI-IT 0.382 0.382 0.386 0.407 FI-RU 0.362 0.361 0.366 0.379 HR-FR 0.397 0.400 0.402 0.426 HR-IT 0.396 0.397 0.396 0.415 HR-RU 0.394 0.390 0.397 0.408 IT-FR 0.671 0.674 0.671 0.684 RU-FR 0.481 0.482 0.485 0.497 RU-IT 0.488 0.489 0.490 0.503 TR-FI 0.288 0.286 0.292 0.306 TR-FR 0.352 0.355 0.354 0.380 TR-HR 0.267 0.272 0.274 0.288 TR-IT 0.353 0.353 0.354 0.372 TR-RU 0.304 0.303 0.304 0.319 AVG 0.422 0.424 0.425 0.442
Table 15: Mean Average Precision (MAP) of alignment methods on XLING with 5K supervision dictionaries using either Iterative Normalization (IN) (Zhang et al., 2017) or Isotropic Preprocessing (IP).
J.3 WEIGHTED PROCRUSTES ABLATION
In order to measure the effect of a Weighted Procrustes rotation on BLI performance, we perform an ablation on both the 1K and 5K XLING BLI datasets against the standard Procrustes formulation.
For training dictionaries of size 1K, weighted procrustes achieves a marginally better aggregate MAP performance when compared to standard procrustes - 0.344 vs 0.336 respectively. In 27 of 28 language pairs, weighted procrustes provides improved MAP performance over standard procrustes.
With training dictionaries of size 5K, weighted procrustes achieves a marginally better aggregate MAP performance when compared to standard procrustes - 0.442 vs 0.440 respectively. In 21 of 28 language pairs, weighted procrustes provides improved MAP performance over standard procrustes.

heading: K BLI PERFORMANCE WITH ALTERNATIVE RETRIEVAL CRITERIA
text: In order to measure the effect on BLI performance of different retrieval criteria, we perform experimentation using CSLS and Nearest Neighbors on both the 1K and 5K XLING BLI datasets.
We find that the CSLS retrieval criterion provides significant performance improvements, compared to Nearest Neighbors search, both for Procrustes and FIPP on both 1K and 5K training dictionaries. For 1K training dictionaries, CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.051 and 0.043 for Procrustes and FIPP respectively. In the case of 5K training dictionaries, CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.049 and 0.031 for Procrustes and FIPP respectively.
Method Proc. + NN (5K) Proc. + CSLS (5K) FIPP + NN (5K) FIPP + CSLS (5K)
DE-FI 0.359 0.433 0.389 0.440 DE-FR 0.511 0.560 0.543 0.576 DE-HR 0.329 0.386 0.360 0.400 DE-IT 0.510 0.555 0.533 0.568 DE-RU 0.425 0.456 0.449 0.461 DE-TR 0.284 0.334 0.321 0.357 EN-DE 0.544 0.584 0.590 0.608

Review summaries:
This paper proposes a new approach for Bilingual Lexicon Induction (BLI). To do this, the paper proposes the use of a FIPP objective, which consists of the linear combination of reconstruction and transfer loss terms. It derives the Gram matrix that minimizes this loss, followed by a rank-constrained semi-definite approximation to obtain an aligned embedding whose Gram matrix is close to the one minimizing the FIPP. The proposed approach is deterministic and has been shown to be efficient. Experiments also demonstrate the method works well, both on BLI and on downstream tasks.

This paper proposes an approach to align word vector spaces based on a dictionary of pairs (e.g. translations) that retains "common geometric structure" (inner products within each vector space for a subset of word pairs). In essence the approach applies preprocessing to each vector space, finds a projection of one vector spaces to the (lower-dimensional) vector space minimising a combination of a reconstruction and transfer loss, and then aligns the two using weighted procrustes to resolve rotational symmetries.