Filtered Inner Product Projection for Crosslingual Embedding Alignment
Authors: Vin Sachidananda
Ziyi Yang
Chenguang Zhu
Venue: ICLR 2021 Poster (2021)
Keywords: multilingual representations
word embeddings
natural language processing

Abstract: Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space. Recently, these algorithms have been studied in the setting of bilingual lexicon induction where one seeks to align the embeddings of a source and a target language such that translated word pairs lie close to one another in a common representation space. In this paper, we propose a method, Filtered Inner Product Projection (FIPP), for mapping embeddings to a common representation space. As semantic shifts are pervasive across languages and domains, FIPP first identifies the common geometric structure in both embeddings and then, only on the common structure, aligns the Gram matrices of these embeddings. FIPP is applicable even when the source and target embeddings are of differing dimensionalities. Additionally, FIPP provides computational benefits in ease of implementation and is faster to compute than current approaches. Following the baselines in Glavas et al. 2019, we evaluate FIPP both in the context of bilingual lexicon induction and downstream language tasks. We show that FIPP outperforms existing methods on the XLING BLI dataset for most language pairs while also providing robust performance across downstream tasks. 

Published as a conference paper at ICLR 2021
FILTERED INNER PRODUCT PROJECTION
FOR
CROSSLINGUAL EMBEDDING ALIGNMENT
Vin Sachidananda Stanford University
Ziyi Yang Stanford University z ziyi -yaggetttnoor.dec....
vsachi @stanford..
Chenguang Zhu
Microsoft Research chezhu@mmccrosoft.comm
ABSTRACT
Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space. Recently, these algorithms have been studied in the setting of bilingual lexicon induction where seeks align the embeddings of source and target language such that translated word pairs lie close to one another a common representation space. In this paper, we propose method, Filtered Inner Product Projection (FIPP), for mapping embeddings to common representation space. As semantic shifts are pervasive across languages and domains, FIPP first identifies the common geometric structure in both embeddings and then, only on the common structure, aligns the Gram matrices of these embeddings. FIPP aligns embeddings to isomorphic vector spaces even when the source and target embeddings are of differing dimensionalities. Additionally, FIPP provides computational benefits ease of implementation and is faster to compute than current approaches. Following the baselines in Glavaš et al. (2019), we evaluate FIPP in the context of bilingual lexicon induction and downstream language tasks. We show that FIPP outperforms existing methods on the XLING (5K) BLI dataset and the XLING (1K) BLI dataset, when using a self-learning approach, while also providing robust performance across downstream tasks.
INTRODUCTION
The problem of aligning sets of embeddings, or high dimensional real valued vectors, is great interest in natural language processing, with applications in machine translation and transfer learning. and shares connections to graph matching and assignment problems (Grave et al. 2019 Gold & Rangarajan 1996). Aligning embeddings trained corpora from different languages has led to improved performance of supervised and unsupervised word and sentence translation (Zou et al. 2013), sequence labeling (Zhang al. 2016 Mayhew et 2017), and information retrieval (Vulic & Moens 2015). Additionally. linguistic patterns have been studied using embedding alignment algorithms (Schlechtweg et al. 2019 Lauscher Glavaš| 2019). Embedding alignments have also been shown to improve the performance of multilingual contextual representation models (i.e. mBERT), when used during intialization, on certain tasks such multilingual document classification (Artetxe et al.) 2020) Recently, algorithms using embedding alignments on the input token representations of contextual embedding models have been shown to provide efficient domain adaptation (Poerner et al. 2020). Lastly, aligned source and target input embeddings have been shown to improve the transfcrability models learned on a source domain to a target domain (Artetxe et al.
201 Sal Wang al. 2018 Mogadala & Rettinger 2016).
In the bilingual lexicon induction task, one seeks to learn a transformation on the embeddings of source and target language so that translated word pairs lie close to another in the shared representation space. Specifically, one is given small seed dictionary D containing pairs of translated words, and embeddings for these word pairs source and target language, X, Rexd

---

Published as a conference paper at ICLR 2021
and X, Roxd Using this seed dictionary, transformation is learned on X, and X, with the objective that unseen tramslation pairs can be induced, often through nearest neighbors search.
Previous literature this topic has focused on aligning embeddings by minimizing matrix or distributional distances (rrae et al. 2019 Jawanpuria et al. 2019: Joulin et al. 2018a). For instance, Mikolov et al. (2013a) proposed using Stochastic Gradient Descent (SGD) learn mapping. Q, to minimize the sum of squared distances between pairs of words the seed dictionary |X 1
which achieves high word translation accuracy for similar languages. /Smith al. (2017) and Artetxe et al. (0177) independently showed that a mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the Orthogonal Procrustes problem, ! = arg mid) X| x," || where O(d) denotes the group of d dimensional orthogonal matrices. However, these methods usually require the dimensions of the source and target language embeddings to be the same, which often may not hold. Furthermore, due to semantic shifts across languages, it's often the case that word and its translation may not co-occur with the same sets of words (Gulordava & Baroni 201T). Therefore, seeking an alignment which minimizes all pairwise distances among translated pairs results in using information not common to both the source and target embeddings.
To address these problems, we propose Filtered Inner Product Projection (FIPP) for mapping embed- dings from different languages to a shared representation space. FTPP aligns a source embedding I, nxx to target embedding I, E 2m and maps vectors in X, to the R& space of X,- Instead of word-level information, FIPP focuses on pairwise distance information, specified by the Gram matrices X, and X,X,, where the rows of X, and X, correspond to embeddings for the pairs of source and target words from the seed dictionary. During aligmment, FIPP tries to achieve the following two goals. First, it is desired that the aligned source embedding FIPP(X,) = X, E Rexda be structurally close to the original source emhedding to ensure that semantic information is retained and prevent against overfitting on the seed dictionary. This goal is reflected the minimization of the reconstruction loss: 1 x,X X 1
Second, as the usage of words and their translations vary across languages, instead of requiring X, to use all of the distance information from X,, FIPP selects a filtered set K of word pairs that have similar X,X7 laj S c). FIPP then minimizes transfer loss on this K, the squared difference in distances between the aligned source embeddings and the target embeddings:
distances in both the source and target languages: K {(, E D fIx,X"
E(.je (X. (x,[1 [[,)" - [ (A(xx,00 ).
We show FIPP can be efficiently solved using either low-rank semidefinite approximations or with stochastic gradient descent. Also, we for- mulate least squares projection to infer aligned representations for words outside the seed dictio- nary and present weighted Procrustes objective which recovers an orthogonal operator that takes into consideration the degree of structural sim- ilarity among translation pairs. The method is illustrated in Figure
R4
R
FIP)(X))
Compared to previous approaches, FIPP has im- proved generality, stability, and efficiency. First, since FIPP's alignment between the source and target embeddings is performed on Gram matri- ces, i.e. X, and X,X! E Rexx, embed- dings are not required be of the same di- mension and are projected to isomorphic vector spaces. This particularly helpful for aligning embeddings trained on smaller corpora, such as in low resource domains, or compute-intensive settings where embeddings may have been compressed to lower dimensions. Secondly, alignment modifications made on filtered Gram matrices can incorporate varying constraints on alignment at the most granular level, pairwise distances. Lastly, FIPP is easy implement involves only matrix
X
Figure I: FIPP alignment of source and target em- beddings, X. and Xt, to a common representation space. Note X, is modified using information from X, and mapped to de while X; is unchanged.
2

---

Published as a conference paper at ICLR 2021
operations, is deterministic, and takes order of magnitude less time to compute than either the best supervised (Joulin et al. (0188) or unsupervised approach (Artetxe et al. 2018c) compared against.
We conduct a thorough evaluation of FIPP using baselines outlined in Glavaš et al. (2019) including bilingual lexicon induction with 5K and 1K supervision sets and downstream evaluation on MNLI Natural Language Inference and Ted CLDC Document Classification tasks. The rest of this paper is
organized as follows. We discuss related work in Section 2 We introduce our FIPP model in Section and usage for inference in Section 4
3
We present experimental results in Section [ and further We conclude the paper in Section 7
discuss findings in Section 6
2 RELATED WORK
2.1 DISTRIBUTIONAL METHODS FOR QUANTIFYING SEMANTIC SHIFTS
Prior work has shown that monolingual text corpora from different communities time periods exhibit variations in semantics and syntax (Hamilton et al.) 2016a b). Word embeddings (Mikolov)
al. 2013b; Pennington al., 2014) [Bojanowski al.
2017) map words representations in
continuous space with the objective that the inner product between any two words representations is approximately proportional to their probability of co-occurrence. By comparing pairwise distances in monolingual embeddings trained on separate corpora, one can quantify semantic shifts associated with biases, cultural norms, and temporal differences (Gulordava & Baroni| 201T:: Sagi al.) [ Kim et al. 2014). Recently proposed metrics on embeddings compare all pairwise inner products of two embeddings, E and F, of the form || EE F FF || YYin et al. 2018). While these metrics have been applied in quantifying monolingual semantic variation, they have not been explored in
context of mapping embeddings to a common representation space or in multilingual settings.
2.2 CROSSLINGUAL EMBEDDING ALIGNMENT
The first work on this topic is by Mikolov et al. 20) who proposed using Stochastic Gradient Descent (SGD) to learn mapping, 52, to minimize the sum of squared distances between pairs of words in the seed dictionary Xx - xell? which achieves high word translation accuracy for similar languages. |mmith et al. (017) and Artetxe et al. (2017) independently showed that mapping with an additional orthogonality constraint, to preserve the geometry of the original spaces, can be solved with the closed form solution to the Orthogonal Procrustes problem, Ω arg mingeo(d) ||x X.Ilr. Dinu & Baroni 2015) worked on corrections to the "hubness" " problem in embedding alignment, where certain word vectors may be close to many other word vectors, arising due nonuniform density of vectors in the Rd space. Smith et al. (017)) proposed the inverted softmax metric for inducing matchings between words in embeddings of different languages. [Artetxe et al. (2016) studied the impact of normalization, centering and orthogonality constraints in linear alignment functions. Jawanpuria et al. (2019) presented a composition of orthogonal operators and a Mahalanobis metric of the form U BV BBV, U VT E O(d), B > 0 to account for observed correlations and moment differences between dimensions (Segaard et al. 2018). Joulin et al. (2018a) proposed an alignment based on neighborhood information to account for differences in density and shape of embeddings in their respective Rd spaces. AAttttx et al. (2018c) outlined a framework which unifies many existing alignment approaches as compositions of matrix operations such Orthogonal mappings, Whitening, and Dimensionality Reduction, Nakashole & Flauger (2018) 8) found that locally linear maps vary between different neighborhoods in bilingual embedding spaces which suggests that nonlinearity is beneficial in global alignments. Nakashole (2018) proposed an alignment method using neighborhood sensitive maps which shows strong performance on dissimilar language pairs. Patra et al. (2019) proposed novel hub filtering method and a semi-supervised alignment approach based on distributional matching. Mohiuddin et al. (2020) learned a non-linear mapping in the latent space of two independently pre-trained autoencoders which provide strong performance on well-studied BLI tasks. A recent method, most similar to ours, Glavaš & Vulić (2020) utilizes non-linear mappings to find translation vector for each source and target embedding using the cosine similarity and euclidean distances between nearest neighbors and corresponding translations. In the unsupervised setting, where bilingual seed dictionary is not provided, approaches using adversarial learning, distributional matching, and noisy self-supervision have been used to concurrently learn matching and alignment between embeddings (Cao et al. 2016 Zhang et al. 2017 Hoshen
3

---

Published as a conference paper at ICLR 2021
& Wolfi 2018 Grave al. 2019 Artetxe et al., 2017; 2018b Alvarez-Melis & Jaakkola 2018).
Discussion on unsupervised approaches included Appendix Section
FILTERED INNER PRODUCT PROJECTION (FIPP)
3.1 FILTERED INNER PRODUCT PROJECTION OBJECTIVE
In this section, we introduce Filtered Inner Product Projection (FIPP), method for aligning embed- dings shared representation space. FIPP aligns source embedding I, E lI "x target em- bedding I, E mg and projects vectors in X, to X, RRxdd Let X, E Reed, and X; E Rexda be the source and target embeddings for pairs in the seed dictionary D, DD = & min m). FIPP's objective is to minimize a linear combination of reconstruction loss, which regularizes changes in the pairwise inner products of the source embedding, and a transfer loss, which aligns the source and target embeddings on common portions of their geometries.
Reconstruction Loss
min X,eRxx22
XX,X X,X
+ (X,X7 XXXx)
Transfer Loss
where X, E RR. are tunable scalar hyperparameters whose effects are discussed in Section the Hadamard product, and A is binary matrix discussed in 3.1.2
E
is
3.1.1 RECONSTRUCTION LOss
Due to the limited, noisy supervision in our problem setting, alignment should be regularized against overfitting. Specifically, the aligned space needs to retain similar geometric structure to the original source embeddings; this has been enforced in previous works by ensuring that aligmments close to orthogonal mappings (Mikolov et al. 201 3a Joulin et al. 2018a Jawanpuria et al. (019). As X, and X, can be of differing dimensionality. we check structural similarity by comparing pairwise inner products, captured by reconstruction loss known the PIP distance or Global Anchor Metrie: |X,XX? XXX I| Yin & Shen 2018 Yin et al. 2018).
Theorem 1. Suppose E E B F E Rn are two matrices with orthonormal columns and Q* arg min
EN - F. It follows that (Yin et al. 2018):
E2 - Fl| < EE FF S / EQ Fl.
This metric has been used in quantifying semantic shifts and has been shown Yin et al. (2018) to be equivalent to the residual of the Orthogonal Procrustes problem up to small constant faccor, seen Theorem 1. Note that the PIP distance invariant to orthogonal operations such as rotations which known to be present in unaligned embeddings.
3.1.2 TRANSFER LOSS
In aligning X, to Xt, we should seek to only utilize common geometric information between the two embedding spaces. We propose simple approach, although FIPP admit other forms of filtering mechanisms, denoted as inner product filtering where we only utilize pairwise distances similar in both embedding spaces defined by threshold e. Specifically, compute a matrix A E {0, 1 where A is an indicator on whether X XT, | < c. In this form, is hyperparameter which determines how close pairwise distances must be in the source and target embeddings to be deemed similar. We then define transfer loss being the squared difference between the converted source embedding X, and target embedding X,, but only on pairs of words in K: || XX x,x?) ||? where is the Hadamard product. The FIPP objective is linear combination of the reconstruction and transfer losses.
3.2
APPROXIMATE SOLUTIONS TO THE FIPP OBJECTIVE
3.2.1
SOLUTIONS USING LOW-RANK SEMIDEFNITE APPROXIMATIONS
Denote the Gram matrices G & X,X,, G A x.XX and 6. & X,X!.
4

---

Published as a conference paper at ICLR 2021
Lemma 2. The matrix G which minimizes the FIPP objective for a fixed X and has entries:
(x.x7))+
if. E K
Gij- (X.X?) )o:
14A
otherwise
Proof. For a fixed X and e, L (X.essrexxxx) (X.X?) can be decomposed as follows:
AIA o (X,X7 x,x7))1 SFIPPAc(XxX3) =x1x))=11xx> =1 XX
Σ ((C)) - + MC G.y) E(G - (4)
ijEK
ijeK
By taking derivatives with respect to Gijy the matrix G which minimizes LFIPP,,, is:
(X -XXx)=--xXxx)
1+A if (i,j) E K
G arg min (FIPPAc(XxX]) G2 (xxx)))
(5)
X,XTeRexe
otherwise
now have the matrix E Rexe which minimizes the FIPP objective. However, for G* to valid Gram matrix, it is required that G* E 84xe, the set of symmetric Positive Semidefinite matrices. Additionally, to recover an X, Rexd2 such that X, XT G", we must have Rank(G") S d2.
Note that G* is symmetric by construction since the set K is commutative and G", G are symmetric. However, G not necessarily positive semidefinite nor is it necessarily true that Rank(G") S d2- Therefore, recover an aligned embedding X Rexds, we perform rank-constrained semidefinite approximation to find min minx,ERR cR
G -
Theorem 3. Let G QAQ" be the Eigendecomposition of G A matrix &, E Rmxxd which 1,,20 A, qu where X and qu are the ith largest eigenvalue minimizes [| G || is given by ddg
and corresponding eigenvector.
Proof. Since G* E Sexo, its Figendecomposition is G QAQT where Q is orthonormal. Let À, be the d2 largest nonnegative eigenvalues in A and their corresponding eigenvectors: additionally, denote the complementary eigenvalues and associated eigenvectors as X A X. Q ğ. Using the Eckart- Yountt-oouuu---aass Theorem for the Frobenius norm (Kishore Kumar & Schneider G||F 2 l X' Exex A3 and that ||" G|| is minimized for G qAof. Using this result, we can recover X.:
2017), note that for G Sixe, RRnn(Gi) S d2; ||
arg min
| Gl -
= 8.87
GESeXe Ra)
(6)
Y
Using the above matrix approximation, we find our aligned embedding X. Exer X vig. minimizer of [| - G.
Due the rank constraint on G. we are only interested the d2 largest eigenvalues and corresponding eigenvectors which incurs complexity of O(2ec) using power iteration (Panju 201 1).
3.2.2 SOLUTIONS USING STOCHASTIC GRADIENT DESCENT
Alternatively, solutions to the FIPP objective can be obtained using Stochastic Gradient Descent (SGD). This requires defining single variable X, IRexda over which to optimize. find that solutions obtained after convergence of SGD are close, with respect the Frobenius norm, to those obtained with low rank PSD approximations up to a rotation. However, the complexity of solving FIPP using SGD is OTT)), where T is the number of training epochs. Empirically find T > d2 for SGD convergence and, as a result, this approach incurs a complexity greater than that of low-rank semidefinite approximations.
5

---

Published as a conference paper at ICLR 2021
3.3 ISOTROPIC PREPROCESSING
Common preprocessing steps used by previous approaches (Joulin et al. 2018a Artetxe et al. 8018a), involve normalizing the rows of &, e to have an &2 norm and demeaning columns. The transfer loss of the FIPP objective makes direct comparisons source and target embeddings for words the seed dictionary. To reduce the influence of dimensional biases between X, and X, and ensure words weighted equalty during alignment, it is desired that X, and X₁ be isotropic i.c. Z(c) = E X exp e" X2[i] is approximately constant for any unit vector с (Arora al. 2016). Mu & Viswanath (2018) find that first order approximation to enforce isotropic behavior is achieved by column demeaning while second order approximation is obtained by the removal of the top PCA components. In FIPP, we apply this simple pre-prccessing approach by removing the top PCA component of X, and X₁. Empirically, the distributions of inner products between a source and target embedding can differ substantially when not preprocessed, rendering a substandard alignment, which is discussed further in the Appendix Section
the Gram matrices, xx.² and X,X? of the
4 INFERENCE AND ALIGNMENT
4.1 INFERENCE WITH LEAST SQUARES PROJECTION
To infer aligned source embeddings for words outside of the supervision dictionary, we make the assumption that source words not used for alignment should preserve their distances to those in the seed dictionary in their respective spaces, i.e., X,X? Xx Using this assumption, we formulate a least squares projection (Boyd & Vandenherghe 2004 on overdetermined system of equations
recover '.: x! x.) -
4.2 WEIGHTED ORTHOGONAL PROCRUSTES
As X, E Rexx2 has been optimized only with concern for its inner products, X, must be rotated so that it's basis matches that of X,. We propose weighted variant of the Orthogonal Procrustes solution to account for differing levels of translation uncertainty among pairs in the seed dictionary, which may arise due to polysemy, semantic shifts and translation errors. In Weighted Least Squares problems, an inverse variance-covariance weighting W is used (Strutz) 2010 Brignell et al. 2015) to account for differing levels of measurement uncertainty among samples. We solve weighted Procrustes objective, where measurement error is approximated as the transfer loss for each translation pair, W = |X XXT Xx 1:
SVD((WX,) -UEVT 22 = arg min Baainn|WW(ee - - REO(ds)
(7)
where (d2) is the group of d2 d2 orthogonal matrices. The rotation ? is then applied X,.
5 EXPERIMENTATION
In this section, we report bilingual lexicon induction results from the XLING dataset and downstream experiments performed on the MNLI Natural Language Inference and TED CLDC tasks.
5.1 XLING BILINGUAL. LEXICON INDUCTION
The XLing BLI task dictionaries constructed by Glavaš et al. (2019) include all 28 pairs between 8 languages in different language families, Croatian (HR), English (EN), Finnish (FI), French (FR), German (DE), Italian (IT), Russian (RU), and Turkish (TR). The dictionaries use the same vocabulary across languages and are constructed based on word frequency, to reduce biases known to exist in other datasets (Kementchedjhievaa et al. 2019). We evaluate FIPP across the all language pairs using a supervision dictionary of size 5K and IK. On 5K dictionaries, FIPP outperforms other approaches on 22 of 28 language pairs. On IK dictionaries, FIPP outperforms other approaches 23 of 28

---

Published as a conference paper at ICLR 2021
language pairs when used along with Self-Learning Framework (denoted HIPP SL) discussed in Appendix Section C. Our code for FIPP is open-source and available on Github
5.1.1
BILINGUAL LEXICON INDUCTION: XLING IK
Method
VecMap
ICP
CCA
PROC*
PROC-B
DLV
RCSLS*
FIPP
FIPP SL
DE-FI
0.302
0.251
0,241
0.264
0.354
0.259
0.288
0.296
0.345
DE-FR
0.505
0.454
0,422
0.428
0.511
0.384
0,459
0.463
0.530
EN-DE
0.521
0.486
0.458
0.458
0.521
0.454
0.501
0.513
0.568
EN-HR
0,268
0.000
0.218
0.225
0.296
0.225
0.267
0.275
0.320
FI-HR
0.280
0.208
0.167
0.187
0,263
0.184
0.214
0.243
0.304
FI-IT
0.355
0.263
0.232
0.247
0.328
0.244
0.272
0,309
0.372
HR-IT
0.389
0.045
0.240
0.247
0.343
0.245
0,275
0.318
0.389
IT-FR
0.667
0.629
0.612
0.615
0.665
0.585
0.637
0.639
0.678
RU-IT
0.463
0.394
0.352
0.360
0.466
0.358
0,383
0.413
0.489
TR-RU
0,200
0.119
0,146
0.168
0,230
0.161
0.191
0.205
0.248
Avg. (All
Lang. Pairs)
0.375
0.253
0,289
0.299
0.379
0.289
0.331
0.344
0.406
Table 1: Mean Average Precision (MAP) of alignment methods subset of XLING BLI with 1K supervision dictionaries, retrieval method is nearest neighbors. Benchmark results obtained from Glavaš al. (2019) in which (+) PROC evaluated without preprocessing and RCSLS was evaluated with Centering + Length Normalization (C+L) preprocessing. Full results for XLING 1K can be found in Appendix Table
5.1.2
BILINGUAL LEXICON INDUCTION: XLING 5K
Method
VecMap
MUSE
ICP
CCA
PROC#
PROC-B*
DLV
RCSLS
FIPP
DE-FI
0.302
0.000
0.251
0.353
0.359
0.362
0.357
0.395
0,389
DE-FR
0.505
0.005
0.454
0.509
0.511
0.514
0.506
0.536
0.543
EN-DE
0.521
0.520
0.486
0.542
0.544
0.532
0.545
0.580
0.590
EN-HR
0.268
0.000
0.000
0.325
0.336
0.336
0.334
0.375
0.382
FI-HR
0,280
0.228
0.208
0.288
0.294
0.293
0.294
0.321
0,335
FI-IT
0.355
0.000
0.263
0.353
0.355
0.348
0.356
0.388
0,407
HR-IT
0.389
0.000
0.045
0.366
0.364
0.368
0.366
0.399
0,415
IT-RR
0.667
0.662
0.629
0.668
0.669
0.664
0.665
0.682
0.684
RU-IT
0.463
0.450
0.394
0.474
0.474
0.476
0.475
0.491
0.503
TR-RU
0.200
0.000
0.119
0.285
0.290
0.262
0.289
0.324
0.319
Avg. (All 0.375
Lang. Pairs)
0.183
0.253
0,400
0.405
0.398
0.403
0.437
0.442
Table 2: Mean Average Precision (MAP) of alignment methods subset of XLING BLI with 5K supervision dictionaries, retrieval method is nearest neighbors. Benchmark results obtained from Glavaš al. (2019) which (*) Proc-B was reported using 3K seed dictionary, (?) PROC was evaluated without preprocessing and RCSLS was evaluated with (C+L) preprocessing. Full results for XLING 5K can be found in Appendix Table
5.2
DOWNSTREAM EVALUATIONS
5.2.1 TED CLDC- DOCUMENT CLASSIRICATION
The TED-CLDC corpus (Hermann & Blunsom) 2014) contains cross-lingual documents across 15 topics and 12 language pairs. Following the evaluation of Glavaš et al. (2019), simple CNN based classifier is trained and evaluated over each topic for language pairs included in our BLI evaluations (EN-DE, EN-FR, EN-IT, EN-RU, and EN-TR). RCSLS outperforms other methods overall and on DE, FR, and RU while FIPP performs best on IT and TR.

---

Published as a conference paper at ICLR 2021
Method
VecMap
MUSE
ICP
GWA
PROC
PROC-B
DLV
RCSLS
FIPP
DE
0,433
0.288
0.492
0.180
0.345
0.352
0.299
0.588
0.520
FR
0.316
0.223
0.254
0.209
0.239
0.210
0.175
0.540
0.433
IT
0,333
0.198
0.457*
0.206
0.310
0.218
0.234
0.451
0.459
RU
0,504
0.226*
0,362
0,151
0.251
0.186
0.375
0,527
0,481
TR 0.439
0.264*
0.175
0.173
0.190
0.310
0.208
0.447
0.491
Avg. 0,405
0.240
0.348
0.184
0.267
0.255
0.258
0,510
0.477
Table 3: TED-CLDC micro-averaged Fi scores using a CNN model with embeddings from different alignment methods. Evaluation follows Glavaš et al. (2019), (*) signifies language pairs for which unsupervised methods were unable yield successful runs.
5.2.2 MNLI NATURAL LANGUAGE INFERENCE
The multilingual XNLI corpus introduced by Conneau et (0118), based off the English only MultiNLI (Williams et al. 2018). includes 5 of the 8 languages used in BLI: EN, DE, FR, RU, and TR. We perform the same evaluation as Glavaš al. (2019) by training an ESIM model (Chen et 2017) using EN word embeddings from a shared EN-L2 embedding space for L.2 {DE, FR, RU,
TRJ. The trained model is then evaluated without further training on the L2 XNLI test set using L2 embeddings from the shared space. The bootstrap Procrustes approach (Glavaš et al. 2019) outperforms other methods narrowly while RCSLS performs worst despite having high BLI accuracy.
Method
VecMap
MUSE
ICP
GWA
PROC
PROC-B
DLV
RCSLS
FIPP
EN-DE
0.604
0.611
0.580
0.427
0.607
0.615
0.614
0.390
0.603
EN-FR
0.613
0.536
0.510
0.383*
0.534
0.532
0.556
0.363
0,509
EN-RU
0.574
0.363*
0.572
0.376*
0.585
0.599
0.571
0.399
0.577
EN-TR
0.534
0.359*
0.400*
0.359
0.568
0.573
0.579
0.387
0.547
Avg. 0.581
0.467
0.516
0.386
0.574
0.580
0.571
0.385
0.559
Table 4: MNLI test accuracy using an ESIM model with embeddings from different alignment methods. Evaluation follows Glavas et al. (2019), (+) signifies language pairs for which unsupervised methods were unable to yield successful runs.
6 DISCUSSION
6.1
RUNTIME COMPARISON
An advantage of FIPP compared to existing methods is it's computational efficiency. We provide a runtime comparison of FIPP and the best per- forming unsupervised (Artetxe et al. 2018ch and supervised (Joulin et al. 2018b) alignment methods on the
Method
FIPP
FIPP+SL
VecMap
RCSLS
Proc-B
CPU GPU
23s
15,896s
387s
885s
176s
612s
Table 5: Average alignment time; sup. approaches 5K dictionary. FIPP+SI. augments with additional 10K samples.
XLING 5K BLI tasks along with Proc- B (Glavaš et al. 2019) supervised method which performs best out of the compared approaches on IK seed dictionaries. The average execution time of alignment on 3 runs of the 'EN-DE' XLING 5K dictionary is provided in Table 4. The implementation used for RCSLS from Facebook's fastText repo with default parameters and the VecMap implementation is from the author's r with and without the 'cuda' flag. Proc-B is implemented from the XLING-Eval repo with default parameters. Hardware specifications are Nvidia GeForce GTX 1080 Ti GPUs, 12 Intel Core 17-6800K processors, and 1 112GB RAM.
'htpps//giiuub..oo/aatttttt http:///hhhh..mmm//aatteeee//aao http:://ggthhh.cmmm//oooggggggg
8

---

Published as a conference paper at ICLR 2021
6.2 ALIGNMENT OF EMBEDDINGS OF DIFFERING DIMENSIONALITY
6.2.1 COMPARISON OF GRAM MATRIX SPECTRUM
In this section, we compute alignments with previous methods, modification of Procrustes and RCSLS (Joulin et al., 20 8b), and FIPP
Spectrum of Aligned Gram Matrices: (EN DE)
on an English Embedding E 200 to a German
Cumulaaivve Sumn of Eigenvalues
Embedding R2000 In Figure 2, we plot the spectrum of the Gram Matrices for the aligned embeddings from each method and the target German Embedding. While the FIPP aligned embedding is isomorphic the target German Embedding E IR 3000 other methods produce rank deficient aligned embedding whose spec- trum deviates from the target embedding. The null space of aligned source embeddings, for methods other than FIPP, is of dimension d2 dd2
AR Number Eigenvalues
We note that issues can arise when learning and transferring models on embeddings from differ- ent rank vector spaces. For regularized models transferred from the source to the target space, at least d2 - d1 column features the target em- bedding will not be utilized. Meanwhile, models transferred from the target space to the source space will exhibit bias associated with model parameters corresponding to the source embedding's 's null space.
Figure 2: Spectrum of Aligned EN Embeddings 2000 and DE Embeddings E R-300 Spectrums of Original, RCSLS and Linear EN embeddings are all approx. equivalent.
6.2.2
BLI PERFORMANCE
FIPP is able to align embeddings of different dimensionalities to isomorphic vector spaces unlike competing approaches (Joulin al. 2018b Artetxe et al. (018c). We evaluate BLI performance on embeddings of different dimensionalities for the EN-DE, (En- glish, German), language pair. We assume that di < d2 and align EN embeddings with dimensions E {200, 250, 300) to DE em- bedding of dimension 300 and compare the performance of FIPP with CCA, implemented in scikit-learm as an iterative estima- tion of partial least squares, and the best linear transform with orthonormal rows, Ω² arg minger x42 _TT==aa, XX, Xelle, equivalent the Orthogonal Procrustes solution when dj = d2. Both the Linear and FIPP methods map X, and X, to dimension d2 while the CCA method maps both embeddings to min(d1, d2) which may be undesirable. While the performance of all methods decreases as d2 d2 increases, the relative performance gap between FIPP and other approaches is maintained.
d1
Linear
CCA
FIPP
300
0.544
0.542
0.590
250
0.524
0.529
0.574
200
0,486
0.501
0.543
Table 6: XLING 5K (EN-DE) MAP for Linear, RCSLS, and FIPP alignment methods on em- beddings of differing dimension- ality.
7 CONCLUSION
In this paper, we introduced Filtered Inner Product Projection (FIPP), method for aligning multiple embeddings to a common representation space using pairwise inner product information. FIPP accounts for semantic shifts and aligns embeddings only on common portions of their geometries. Unlike previous approaches, FIPP aligns embeddings to equivalent rank vector spaces regardless of their dimensions. We provide two methods for finding approximate solutions the FIPP objective and show that can be efficiently solved even in the case of large seed dictionaries. We evaluate FIPP on the task of bilingual lexicon induction using the XLING (5K) dataset and the XLING (1K) dataset, on which it achieves stateoffthheaat performance on most language pairs. Our method provides a novel efficient approach the problem of shared representation learning.
hhtps://seikiiteem..ooo/eeeeeeee http:////siiiiaaaa////aaaaaaaaan decoompssooooooiiiiiiiiio.o. CCt.:ssiiiiiio..hc..... CCA.html

---

Published as a conference paper at ICLR 2021
REFERENCES
Hanan Aldarmaki, Mahesh Mohan, and Mona Diab. Unsupervised word mapping using structural similarities in monolingual embeddings. Transactions of the Association for Computational Linguistics, 6: 185-196, 2018. doi: 211162/tac_a_0000 URL https: /www ac aclweb
Dxc/Annthology//11-11011
David Alvarez-Melis and Tommi Jaakkola. Gromov-Wasserster alignment of word embedding spaces. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Process- ing, pp. 1881-1890, Brussels, Belgium, Dctober-Novemble 2018, Association for Computational Linguistics. doi: 1/D1////////II1/// D18-1214
URL https: www. aclweb. org/ anthology
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016. doi: (e66:/aal_a_0011111 URL /tt::://wwwwww.w acwweb..leb... org/
anthologyy011-1029
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2289-2294, 2016.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 451-451144 462.. Association for Computational Linguistics,
2017. doi: 10. 10.18653/11//11--044 URL URL http::/ http:::///wwwbb..... www.wewww..rg..............o.... anthology hatployyyy/11111400
Mikel Artetxe, Gorka Labaka, and Encko Agirre. Generalizing and improving bilingual word embedding mappings with a multi-step framework of lincar transformations. In AAAI, pp. 5012-5019, 2018a. URL htttt::t/////tt
aaai, wrtp://o////////////// php/ AAAI AAAII8/
paper/view/1692
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 789-798, Melbourne, Australia, July 2018b. Association for Computational Linguistics. doi: 10. 10.18653/11/P18-11
URL https: wwww ac www.wwwwbbb.....c wtt..//ww....ttttoooooo////////e
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self- method for fully unsupervised
cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 789-798, Melbourne, Australia, July 2018c. Association for Computational Linguistics. doi: 10. 0...8653/v1/P18--
URL https: wwww aclweb. www.we...o.r.... orq/ wwtp:///tttoooooye/---110000
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolin- gual representations. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational Linguis- tics. doi: 102018653/112002.achhhhho a21 URL https: http://wwwww.. acwweb.www........... org/ anthology 2020. acl maa.mmaan...m 421.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146. 2017. ISSN 2307-387X.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004. ISBN 0521833787.
Christopher J. Brignell, II. Dryden, and William Browne. Covariance Weighted Procrustes Analysis, pp. 189-209. Springer, 2015,
Hailong Cao, Tiejun Zhao, Shu Zhang, and Yao Meng. A distribution-based model to learn bilingual word embeddings. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguisties: Technical Papers, pp. 1818-1827, Osaka, Japan, December 2016. The
10

---

Published as a conference paper at ICLR 2021
COLING 2016 Organizing Committee. URL htt httppss/ww...cceeb..rr./../ta/t C16-1171
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACI. 2017), Vancouver, July 2017. ACI..
Alexis Conneau, Guillaume Lample. Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. CoRR. abs/1809.0555 809.05053, 2018. URL http::httt:ttttteee org/ / abs/ 809. 05053
Georgiana Dinu and Marco Baroni. Improving zero-shot learning by mitigating the hubness problem. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http://arxix org/ 0tts/1112.6550
Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 462-4422444 471,, Gothenburg, Sweden, April 2014. Association for
Computational Linguistics. doi: 10.3115/v1/E14-1000 URL https:::: / www. awwwwb........ org/
antho hooplllyy///// E14 1049
Goran Glavaš and Ivan Vulié. Non-linear instance- based cross-lingual mapping for non-isomorphic embedding spaces. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7548-7555, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653//1/2222.aael/ awl.:000..mmmmi....... URL https: www. ac web. org httb.//r/aaaoooooo//2222
ac 10m-mmain.c. 675
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. How to (properly) evaluate cross- lingual word embeddings: On strong basclines, comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 710-721, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1070. URL URL htthttts::// /www. awwweb. http://ww....g.ttoooo/o///g///// 1070
Steven Gold and Anand Rangarajan. A graduated assignment algorithm for graph matching. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 18:377 -0777-388, 388, 05 1996. doi: 10.1 34.491619.
Edouard Grave, Armand Joulin, and Quentin Berthet. Unsupervised alignment of embeddings with wasserstein procrustes. In Kamalika Chaudhuri and Masashi Sugiyama (eds.). The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16- April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 1880-1890. PMLR, 2019. URL
mir wwt..mii.......eeemeem..e.....s. wttpee///////grveaaaaa..... html
Kristina Gulordava and Marco Baroni. A distributional similarity approach to the detection of semantic change in the google books ngram corpus. In Proceedings of the GEMS 201 Workshop on GÉometrical Models of Natural Language Semantics, pp. 67-71, 2011.
William L Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. Inducing domain-specific sentiment lexicons from unlabeled corpora. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 395-605, 2016a
William L Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pp. 1489-1501, 2016b.
Karl Moritz Hermann and Phil Blunsom. Multilingual models for compositional distributed semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pp. 58 68, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/Pi4/ 1006. URL hctps: www. . ac lweb. org/ hothol
P P14-1006
11

---

Published as a conference paper at ICLR 2021
Yedid Hoshen and Lior Wolf. An iterative closest point method for unsupervised word translation. CoRR, 180100661,,,, 2018. URl/attp::/azxii.orr wwtp:/ivvvgggaa/a//////ea/e 06126
Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukuttan, and Bamdev Mishra. Learning mul- tilingual word embeddings in latent metric space: A geometric approach. Transactions of the Association for Computational Linguistics, 7:107-120, 2019. doi: 10.116//acllaa URL
htttps://di..oo orgg 1et.0..1122tttt
00257
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, and Edouard Grave. Improving supervised bilingual mapping of word embeddings. CoRR, obs//8804.7775, 2018a. URL http:h//taatt arxiv
org/abs 1804.0000 07745
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in
translation: Learning bilingual word mapping with retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2979-2984, Brussels, Belgium, October-November 2018b. Association for Computational Linguisties. doi: 10.18653/ WVDI8-1130. URL www ac awwebb. org/ anthology//yaat DD1-/13300010000
Yova Kementchedjhieva, Mareike Hartmann, and Anders Søgaard. Lost in evaluation: Misleading benchmarks for bilingual dictionary induction. In Proceedings of the 2019 Conference Empirical
Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIIP-IJNNIPP), pp. 3336-3341, Hong Kong, China, November 2019.
Association for Computational Linguistics, doi: 18.18653/v1/ 1/15///1/D19.... 1328. URL http:://twww:..
aclweb. wateee/.rr/aaathoooo////// 1328
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. Temporal analysis of language through neural language models. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL.), pp. 61, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
//aaxiv.org/aaaaaa..e..a abaaaaaaaa 1112. 6980
N Kishore Kumar and Jan Schneider. Literature survey on low rank approximation of matrices. Linear and Multilinear Algebra,
22444 2017.
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference Learning Representations, 2018. /ttl:tttts////pereeer ew ew.
d= 96sainb
Anne Lauscher and Goran Glavaš. Are we consistently biased? multidimensional analysis of biases in distributional word vectors. In Proceedings of the Eighth Joint Conference Lexical and Computational Semantics (*SEM 2019), pp. 85-91, Minneapolis, Minnesota. June 2019. Association for Computational Linguisties. doi: 10.18653/w1/S19- 1010. URL https://w//w
ac seclweb.orgg http://anhholllyy119111111
Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. Cheap translation for cross-lingual named entity recognition. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2536-2545, Copenhagen, Denmark, September 2017. Association for Computational Linguistee.. doi: 10. 10.18653/w//D17-110 URL www aclweb.org
anthology/D17-1222
Tomas Mikolov, Quoe V. Le,, and Ilya Sutskever. Exploiting similarities among languages for machine translation. CoRR, 1444339,,468, 2013a.
wwt.///gg/bbbboooo aa0.aa/s/11.... 4168
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality In Advances in Neural Information Processing Systems (NIPS), pp. 3111-3119, 2013b.
Aditya Mogadala and Achim Rettinger. Bilingual word embeddings from parallel and non non-parallel corpora for cross-language text classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, pp. 692-702, 2016.
12

---

Published as a conference paper at ICLR 2021
Tasnim Mohiuddin, M Saiful Bari, and Shafiq Joty. LNMap: Departures from isomorphic assumption in bilingual lexicon induction through non-linear mapping in latent space. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2712 2712-2723, Online, November 2020. Association for Computational Linguistics. URL https
www. ac web. www.:b...g..aa.e.. htc.hooooyyy222222 www.::000emmmmmmmmmmmmmm sommlp-mminn... .2
Jiaqi Mu and Pramod Viswanath. All-but-the-op:: Simple and effective postprocessing for word
representations. In International Conference on Learning Representations, 2018. URLhttps
Ndapa Nakashole. NORMA: Neighborhood sensitive maps for multilingual word embeddings. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp- 512 112-522, Brussels, Belgium, October-November 2018. Association for Computational Linguisties. doi: 10.18653/v1/D1s 1047. URL (Rtt:s:: http://wwwwwaa.m www.:w.wwww................ org/ wwwpg//gnntoag//naaoooooo 1044
Ndapa Nakashole and Raphael Flauger. Characterizing departures from linearity in word translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 221-227, Melbourne, Australia, July 2018. Association for Com- putational Linguistics. doi: 10. 10.18653/11//118-0003
URL https: www.ww.owwwwbrgog.
anthology/P18 2036
Maysum Panju. Iterative methods for computing eigenvalues and eigenvectors. arXiv preprint 1005 1855, 2011.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig.
Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 184-193, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/119 1018. URL https: www a C wwggb.org//attee httpe//orrrrtttooo////99/--1
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Inexpensive domain adaptation of pretrained language models: Case studies on biomedical NER and covid- 19 QA. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pp. 1482-1490, Online, November 2020. htw.:///....iiinneneeeeeee https: www. ac wwwwwwww..oooomo org wttp:////aaat.o//////ooe///// enn..oo.............nnnnneeeee.. 134
Association for Computational Linguistics. doi: 10. 106118663/111
URL
Sebastian Ruder, Ryan Cotterell, Yova Kementchedjhiiva,, and Anders Søgaard. A discriminative
latent-variable model for bilingual lexicon induction. In Proceedings the 2018 Conference on
Empirical Methods in Natural Language Processing, pp. 458-468, Brussels, Belgium, October- November 2018. Association for Computational Linguisties. doi: (0.18653/v//D18-110 URL
https: www ac www.wwww.b.......c..... wwt.w/ww...../aaaoooooo///eeeeee
Eyal Sagi, Stefan Kaufmann, and Brady Clark. Tracing semantic change with latent semantic analysis. Current methods in historical semantics, pp. 161-183, 2011.
Dominik Schlechtweg, Anna Hätty, Marco Del Tredici. and Sabine Schulte im Walde. A wind of change: Detecting and evaluating lexical semantic change across times and domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 732-746, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ 107- URL
www. www.eb.e....c...coo.........mbbb org/
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In 5th International Confer- on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https::///........tt...tt /ttps///ee/eeeeiee...ooo net wateeeeforiieee forum?id
r (1Abb85gg
13

---

Published as a conference paper at ICLR 2021
Anders Søgaard, Sebastian Ruder, and Ivan Vulié. On the limitations of unsupervised bilin- gual dictionary induction. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 778-788, Melbourne. Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.1865311//118-1072 URL
https: www. ac web www.b.org/aattee htt../orgg//anoooooo///-11111100 ogy/P18-1072
Tilo Strutz. Data Firting and Uncertainty: A Practical Introduction to Weighted Least Squares and Beyond. Vieweg and Teubner, Wiesbaden, DEU, 2010. ISBN 3834810223.
Ivan Vulić and Marie-Francine Moens. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In Proceedings of the 38th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR 15. pp. 363-372, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336215 doi: 10.1 10.1155/766466227777722 URI.tttps:: / http///oooooo///.///// 100.0..1115226666.11 2267775
Rui Wang, Hai Zhao, Sabine Ploux, Bao-Liang Lu, Masao Utiyama, and Eiichiro Sumita. Graph- based bilingual word embedding for statistical machine translation. ACM Trans. Asian Low- Resour: Lang. Inf. Process., 17(4), July 2018. ISSN 2375-4699, doi: 10.1145/3203078. URL
hittps://do. /ww. www.rrgg............. /10. 145/14555 3203078
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, Volume / (Long Papers), pp. 1112-1122. Association for Computational Linguisties, 2018. URL httthttt://eee web. org/ www.annthohayaaaeaa 11001
Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. S. Bengio, H. Wallach. and R. Gareet (eds.), Advances in Neural Informa- tion Processing Systems 31, pp. 887-898. Curran Associates, Inc., 2018. htttttt:::////rr /aapers n nips. wwtpcccppppp//cee 68 on t he oo
H. Larochelle, K. Grauman, N.
Zi Yin, Vin Sachidananda, and Balaji Prabhakar. The global anchor method for quantifying linguistic shifts and domain adaptation. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. Curran Associates, Inc., 2018.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pp. 1959-1970. Association for Computational Linguistics, 2017. doi: 10. 11.18633/11P11-11199 URL httph:ttp//cleeeeeee org/ anthology/ P P17-1179
Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. Ten pairs to tag multilingual POS tagging via coarse mapping between embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1307-1317, San Diego, California, June 2016. Association for Computational Linguistics. doi: 11118553/1/NN1--1155 URL https:///////... wwwp://wwww..... aclweb. org/ anthology N16 N16-1156
Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1393- 1398, 2013.
14

---

Published as a conference paper at ICLR 2021
A FULL BLI EXPERIMENTATION XLING (IK) AND XLING (5K)
In Table 6 below, we provide experimental results for FIPP using 1K seed dictionaries. Unlike in case of a 5K supervision set, FIPP is outperformed by the bootstrapped Procrustes method (Glavaš et al. 2019) and the unsupervised VecMap approach (Artetxe et al.) 2018c). However, the addition of self learning framework, detailed in Section C, FIPP + SL) results performance greater than compared methods albeit at the cost of close to a 8x increase in computation time, from 23s to 176s, and the requirement a GPU. Other well performing methods for XLING 1K, Proc-B (Glavaš et al. 2019) and VecMap (Artetxe et al., 20T 8c), also use self-learning frameworks; further analysis required to understand the importance self-learning frameworks the case of small (or no) seed dictionary.
The methods compared against were originally proposed in: VecMap (Artetxe et al. 201 MUSE
(Lapple et al., 2018), ICP (Hoshen & Wolf, 2018), CCA (Faruqui & Dyer, 2014), GWA (Alvree
Melis & laakkola 2018). PROC (Mikolov et al. 2013a). PROC-B (Glavaš et al. 2019), DLV (Ruder
al. 2018 and RCSLS (Joulin al. 2018b).
Method
VecMap
ICP
CCA
PROC*
PROC-B
DLV
RCSLS
FIPP
FIPP + SL
DE-FI
0.302
0.251
0.241
0.264
0.354
0.259
0.288
0.296
0.345
DE-FR
0.505
0.454
0,422
0.428
0.511
0.384
0.459
0.463
0.530
DE-HR
0,300
0.240
0.206
0.225
0.306
0.222
0,262
0.268
0.312
DE-IT
0,493
0,447
0.414
0,421
0.507
0.420
0.453
0,482
0.526
DE-RU
0.322
0.245
0.308
0.323
0.392
0.325
0.361
0.359
0.368
DE-TR
0.253
0.215
0.153
0.169
0.250
0.167
0.201
0.215
0.275
EN-DE 0.521
0.486
0.458
0.458
0.521
0.454
0.501
0.513
0.568
EN-FI
0.292
0.262
0,259
0.271
0,360
0.271
0.306
0.314
0.397
EN-FR
0.626
0.613
0.582
0.579
0.633
0.546
0.612
0.601
0.666
EN-HR
0,268
0.000
0.218
0.225
0.296
0.225
0.267
0.275
0.320
EN-IT 0.600
0.577
0.538
0.535
0.605
0.537
0.565
0.591
0.638
EN-RU
0.323
0.259
0.336
0.352
0.419
0.353
0.401
0.399
0.439
EN-TR
0.288
0,000
0,218
0.225
0.301
0,221
0.275
0,292
0.360
FLERR
0.368
0.000
0.230
0.239
0.329
0.209
0.269
0.274
0.366
FLHR
0,280
0.208
0.167
0.187
0,263
0.184
0.214
0.243
0.304
FLITT
0.355
0.263
0.232
0.247
0.328
0.244
0.272
0,309
0.372
FI-RU
0.312
0.231
0.214
0.233
0.315
0.225
0.257
0.285
0.346
HR-FR
0.402
0.282
0.238
0.248
0.335
0.214
0.281
0,283
0.380
HR-IT
0.389
0.045
0.240
0.247
0.343
0.245
0.275
0.318
0.389
HR-RU
0.376
0.309
0.256
0.269
0.348
0.264
0.291
0.318
0.380
IT-FR
0.667
0.629
0.612
0.615
0.665
0.585
0.637
0.639
0.678
RU-FR
0,463
0.000
0.344
0.352
0.467
0.320
0.381
0.383
0.486
RU-IT
0.463
0.394
0.352
0.360
0.466
0.358
0.383
0.413
0.489
TR-FI 0.246
0.173
0.151
0.169
0.247
0.161
0.194
0.200
0.280
TR-FR
0.341
0.000
0,213
0.215
0.305
0.194
0.247
0.251
0.342
TR-HR
0.223
0.138
0.134
0.148
0.210
0.144
0.170
0.184
0.241
TR-IT 0.332
0.243
0.202
0.211
0.298
0,209
0.246
0.263
0.335
TR-RU
0.200
0.119
0.146
0.168
0.230
0.161
0.191
0.205
0.248
AVG
0.375
0.253
0.289
0.299
0.379
0.289
0.331
0.344
0.406
Table 7: Mean Average Precision (MAP) of alignment methods XLING with 1K supervision dictionaries, retrieval method is nearest neighbors. Benchmark results obtained from Glavaš et al. (2019) in which ( PROC was evaluated without preprocessing and RCSLS was evaluated with Centering + Length Normalization (C+L.) preprocessing
15

---

Published as a conference paper at ICLR 2021
Method
VecMap
MUSE
ICP
CCA
PROC
PROC-B
DLV
RCSLS'
FIPP
DE-FI
0.302
0.000
0.251
0.353
0.359
0.362
0.357
0.395
0.389
DE-FR
0,505
0.005
0.454
0.509
0.511
0.514
0.506
0.536
0.543
DE-HI
0.300
0.245
0.240
0.318
0.329
0.324
0.328
0.359
0.360
DE-IT
0.493
0.496
0.447
0.506
0.510
0.508
0.510
0.529
0.533
DE-RU
0.322
0.272
0.245
0.411
0.425
0.413
0.423
0.458
0,449
DE-TR
0,253
0.237
0.215
0.280
0.284
0,278
0.284
0.324
0.321
EN-DE 0.521
0.520
0.486
0,542
0.544
0.532
0.545
0,580
0.590
EN-FI
0.292
0.000
0.262
0.383
0.396
0,380
0,396
0.438
0.439
EN-FR
0.626
0.632
0.613
0.652
0.654
0,642
0,649
0.675
0.679
EN-HR
0.268
0.000
0.000
0.325
0.336
0.336
0.334
0.375
0.382
EN-IT 0.600
0.608
0.577
0.624
0.625
0.612
0.625
0.652
0.649
EN-RU 0.323
0.000
0.259
0.454
0.464
0.449
0.467
0.510
0.502
EN-TR
0,288
0.294
0.000
0.327
0.335
0.328
0.335
0.386
0.407
FL-FR
0.368
0.348
0.000
0.362
0.362
0.350
0.351
0.395
0.407
FI-HR
0,280
0.228
0.208
0.288
0.294
0.293
0.294
0.321
0.335
FI-IT
0.355
0.000
0.263
0.353
0.355
0.348
0.356
0.388
0.407
FI-RU
0.312
0.001
0.231
0.340
0.342
0.327
0.342
0.376
0.379
HR-FR
0,402
0.000
0.282
0.372
0.374
0.365
0.364
0.412
0.426
HR-IT
0.389
0.000
0.045
0.366
0.364
0.368
0.366
0.399
0.415
HR-RU
0.376
0.000
0.309
0.367
0.372
0.365
0.374
0.404
0.408
IT-RR
0.667
0.662
0.629
0.668
0.669
0.664
0,665
0.682
0.684
RU-FR 0.463
0.005
0.000
0.469
0.470
0.478
0.466
0.494
0.497
RU-IT
0.463
0.450
0.394
0.474
0.474
0.476
0.475
0.491
0,503
TR-FI
0.246
0.000
0.173
0.260
0.269
0.270
0.268
0.300
0.306
TR-FR
0.341
0.000
0.000
0.337
0.338
0.333
0.333
0.375
0,380
TR-HR
0,223
0.133
0.138
0.250
0.259
0,244
0,255
0,285
0.288
TR-IT 0.332
0.000
0.243
0.331
0.335
0.330
0,336
0.368
0.372
TR-RU
0.200
0.000
0.119
0.285
0.290
0.262
0.289
0.324
0.319
Avg.
0.375
0.183
0.253
0.400
0.405
0.398
0.403
0.437
0.442
Table 8: Mean Average Precision (MAP) of alignment methods on XLING with 5K supervision dictionaries, retrieval method is nearest neighbors. Benchmark results obtained from Glava' et al. (2019) in which (*) Proc-B was reported using 3K seed dictionary and (t) PROC was evaluated without preprocessing and RCSLS was evaluated with Centering Length Normalization (C+L) preprocessing.
B EFFECTS OF RUNNING MULTIPLE ITERATIONS OF FIPP OPTIMIZATION
Although other aligmment approaches (Joulin et al. 2018b Artetxe et al. 2018c) multiple itera- tions of their alignment objective, we find that running multiple iterations of the FIPP Optimization does improve performance. We between 1 and 5 iterations of the FIPP objective. For 1K seed dictionaries, 26/28 language pairs perform best with iteration while 26/28 language pairs perform best with 1 iteration for 5K seed dictionaries. the case of 1K seed dictionaries. (EN, FI) with iterations and (RU, FR) with 2 iterations resulted in MAP performance increases of 0.002 and 0.001. For 5K seed dictionaries. (EN, FI) with iterations and (EN, FR) with iterations resulted in MAP performance increases of 0.002 and 0.002. Due to limited set of language pairs on which performance improvements were achieved, these results have not been included in Tables and 6.
C SELF-LEARNING FRAMEWORK
A self-learning framework, used for augmenting the number of available training pairs without direct supervision, has been found be effective (Glavaš al., 2019 Artetxe al,, 2018c)) both in the case of small seed dictionaries or in unsupervised setting. We detail a self-learning framework which improves the BLI performance of FIPP in the XLING IK setting not case of 5K seed pairs.
16

---

Published as a conference paper at ICLR 2021
Let X, Rexd and X, Rexda be the source and target embeddings for pairs in the seed dictionary D. Ad- ditionally, X, E TR and X, E Rmxda the source and target beddings for the entire vocabulary. Assume all vectors have been normal- ized to have an E2 norm of 1. Each source vector X,,, and target vector It,j can be rewritten d dimensional row vectors of inner products with their corresponding seed dictionaries: A.,i X,, E R x and Aa = Xe, E ERIc.
Cos. Sim. Rank
English word
French word
oversee renegotiation inform buy optimize participate
superviser renégociation informer
4
acheter optimiser
participer
conceptualization
conceptualisation
8
internationalization renégocier interactivité
internationalisation
renegotiate
10
interactivity
Table 9: Top 10 pairings by cosine similarity when using Self-Learning framework the English- French language pair.
For each source word we compute the target word j with the greatest co- sine similarity, equivalent to the inner product for vectors with norm of 1, in this d dimensional space as (i. = arg max, As, For XLING BLI IK experiments, we find the 14K (i.j) pairs with the largest cosine similarity and augment seed dictionaries with these pairs. In Table 7, the top 10 translation pairings, sorted by cosine similarity, obtained using this self-learning framework are shown for the English to French (EN-FR) language pair.
D COMPARISON OF FIPP SOLUTION TO ORTHOGONAL ALIGNMENTS
In this section, we conduct experimentation to quantify the degree to which the FIPP alignment differs from an orthogonal solution and compare performance monolingual tasks before and after FIPP alignment.
D.1 DEVIATION OF FIPP FROM THE CLOSEST ORTHOGONAL SOLUTION
For each language pair, we quantify the deviation of FIPP from an Orthogonal solution by first calculating the FIPP alignment before rotation, X,, on the seed dictionary. We then compute the relative deviation of the FIPP alignment with the closest orthogonal alignment on the original embedding, X., This is equal to D ||,,-2'x.116 where Ω = arg mingeo(d,) |X,Ω X.llr. ||,
The average of these deviations for 1K seed dictionaries is 0.292 and for 5K seed dictionaries is 0.115. Additionally, the language pairs with the largest and smallest deviations from orthogonal solutions presented in the Table below. We find that in most cases, small deviations from orthogonal solutions are observed between languages in the same language family (i.e. Indo-Furopean -> Indo- European) while those in different Language families tend to have larger deviations (i.e. Turkic Indo-European). A notable exception to this observation is English and Finnish which belong to different language families, Indo- aado-European and Uralic respectively, yet have small deviations their FIPP solution compared to an orthogonal alignment.
Lang. Pair (IK)
fg Deviation
Lang. Pair (5K)
Deviation
German-Italen (DE-IT)
0.025
English-Finnn Finnish (EN-FI)
0.008
English-Finniss (EN-FI)
0.090
Croatian- Italian (HR-IT)
0.012
Italian-French (TT-FR)
0.100
Poaatinn-Russian (HR-RU)
0.014
Turkish-French (TR-FR)
0.40S
Finnish-Frenn (FI-FR)
0.343
Turkish-Russinn (TR-RU)
0.408
(FI-HR)
0.351
Turkish--innish (TR-FI)
0.494
Turkish-Finnin (TR-FI)
0.384
Table 10: Smallest and Largest Deviations FIPP from Orthogonal Solution, XLING BLI IK and 5K
17

---

Published as a conference paper at ICLR 2021
D.2 EFFECT OF INNER PRODUCT FILTERING ON WORD-LEVEL ALIGNMENT
In FIPP, Inner Product Filtering is used to find common geometric information by comparing pairwise distances between source and target language. To illustrate this step with translation word pairs, in the Table below we show the 5 words with the largest and smallest fraction of zeros, i.e. the "least and most filtered", in the binary filtering matrix A during alignment between English (En) and Italian (It). The words which are least filtered tend to have an individual word sense, i.c. proper nouns, while those which are most filtered are somewhat ambiguous translations. For instance, while the English word "securing" can be translated to the Italian word "fissagio" depending on the context the Italian words "garantire", "assicurare" or "fissare" may be more appropriate.
Least Filtered
Most Filtered
English Word
Italian Word
English Word
Italian Word
japanese los china
giapponese
securing
fissagio astuccio servi
los cina
serves
piano
piamoforte
joining
accoppiamento
film
film
fraction
frazione
Table 11: Most and Least Filtered word pairs during FIPP's Inner Product Filtering for English-Itan Italian
alignment
D.3 MONOLINGUAL TASK PERFORMANCE ALIGNED EMBEDDINGS
As FIPP does not perform an orthogonal trans- form, it modifies the inner products of word vec- tors in the source embedding which can impact performance on monolingual task accuracy. We evaluate the aligned embedding learned using FIPP, X,, monolingual word analogy tasks and compare these results to the original fastText embeddings x.. In Table 3, compare mono- lingual English word analogy results for English embeddings H. which have been aligned to Turkish target embedding using FIPP. Evalua- tion of the aligned and original source embed- dings on multiple English word analogy experi- ments show that aligned FIPP embeddings retain performance on monolingual analogy tasks.
Table 12: Monolingual Analogy Task Performance for English embedding before/after alignment to Turkish embedding.
English Word Analogy Similarity
Dataset
Original
FIPP
WS-353
0.739
0.736
M'Turk-771
0.669
0.670
SEMEVAL-17
0.722
0.722
SIMLEX-999
0.382
0.381
E EFFECT OF HYPERPARAMETER, ON BLI PERFORMANCE
In our experiments, we tune the hyperparameters and À which signify the level of discrimination in the inner product filtering step and the weight of the transfer loss respectively. When tuning, we account for the sparsity of A by scaling À in the transfer loss by 7 NNZIA.J where NN the number of nonzeroes in A Values of c used in tuning were [0.01, 0.025. 0.05, 0.10, 0.15) and scaled values of À used in tuning were (0.25, 0.5, 0.75, 1.0, 1.25]. Values of (c, which are close to one another result in similar performance and results are deterministic across reruns of the same experiment. As no validation set provided, hyperparameters are tuned by holding 20% of the training set.
22
18

---

Published as a conference paper at ICLR 2021
F DIFFERENCE IN SOLUTIONS OBTAINED USING Low RANK APPROXIMATIONS AND SGD
As detailed in Section 3, solutions to FIPP can be calculated either using Low Rank Approxi- mations or Stochastic Gradient Descent (SGD). In this section, we show the error on the FIPP objective for SGD trained over 10,000 epochs on alignment of a Finnish (FT) embedding to a French (FR) embedding. The Adam (Kingma & Bal 2015) optimizer is used with a learning rate of Ie and the variable being optimized XSGD is initialized to the original Finnish em- bedding X.. We find that the SGD solution XSGD approaches the error of the Low Rank Approximation XLRA, which is the global min- ima of the FIPP objective, as shown in the Figure below but is not equivalent. While the deviation between SGD and the Low Rank Approxima- tion, DSGD, || -
FIPP Objective Loss (FI-FR) for SGD Low Rank Approximation
URA
180
$60
FiPp Objective Loss
160
140
100
2000
6000
6.00
5900
10000
Timestap
Figure 3: Comparison of FIPP Objective Loss for (FI-FR) for solutions obtained using SGD vs LRA
-XLRA Ir 0.041, |X smaller than the deviation between the original embedding and the Low Rank Approximation,
Dorig,LRA [|x -X.-XLRAJJ ||r 0.343 we note that the solutions are close but not equivalent.
1X
G EFFECTS OF PREPROCESSING ON INNER PRODUCT DISTRIBUTIONS
We plot the distributions of inner products, entries of X,X7 2 and X,X?', for English (En) and Turkish (Tr) words in the XLING 5K training dictionary before and after preprocessing in Figure below. All embeddings used in experimentation are fastText word vectors trained on Wikipedia (Bojanowski et al. 2017). Since inner products between X, and X, compared directly, the isotropic preprocessing utilized in FIPP is necessary for removing biases caused by variations in scaling, shifts, and point density across embedding spaces.
Inner Product Distributions en and Unprocessed
Inner Product Distributions of an and tr Isotropic Preprocessing
20002
90003
17203
Frequency
7502
20003
2933
3.0
Inner Product Product Magritude
Pnner Product Magnitude
Figure 4: Gram matrix entries unprocessed fast- Text embeddings
Figure 5: Gram matrix entries fastText embed- dings with preprocessing
H COMPLEXITY ANALYSIS
In this section, we provide the computational and space complexity of the FIPP method as described in the paper for computing X_. We split the complexity for each step in the method. We leave out steps (i.e. preprocessing) which do not contribute significantly to the runtime or memory footprint. The computational complexity of matrix multiplication between two matrices A, A2, where A Rmxx A2 Roxp, is denoted M (m, n. which is upper bounded by 2mmp operations.
19

---

Published as a conference paper at ICLR 2021
H.1 FIPP OPTIMIZATION
The complexity for solving the FIPP objective is detailed below:
Space Complexity = 4
+
3ed
,", ,0
Xo.
X..
Time Complexity = (ld2c²)
36
+
5e2
= (d2c?) (8)
Compute X.X! w/ Power Iteration
1xx,, x.x
AIll o (8.8 xxxD)le
H.2 SVD ALIGNMENT AND LEAST SQUARE PROJECTION
The complexity of the alignment using the Orthogonal Procrustes solution and the Least Squares Projection follows:
Space Complexity
3c
cn
+ nd2
USVT
Time Complexity = M MM (c, d2, d2) + MM (M (2, d2)
&
8.VU
SVD(X;
(9)
MM + MM d2, + (M2,,
Least Squares
<2(cd z00 edinn &d cdgn) +
3
H.3 DISCUSSION
In performing our analysis, note that the majority of operations performed are quadratic in the training set size c. While we incur a time complexity of O(e)) during our Least Squares Projection due the matrix inversion in the normal equation, this inversion is time cost. The space complexity of FIPP is (e²2) which is tractable as c is at most 5K. Empirically, FIPP is fast to compute taking less than 30 seconds for seed dictionary of size 5K which more efficient than competing methods.
I RELATED WORKS: UNSUPERVISED ALIGNMENT METHODS
Cao et al. (2016) studied aligning the first two moments of sets of embeddings under Gaussian distribution assumptions.
Zhang al. (2017), an alignment between embeddings of different languages is found by matching distributions using an adversarial autoencoder with an orthogonal regularizer. Artetxe et al. (2017) proposes an alignment approach which jointly bootstraps a small seed dictionary and learns an alignment in self-learning framework. Hoshen & Wolf (2018) first projects embeddings to subspace spanned by the top principle components and then learns an alignment by matching the distributions of the projected embeddings. Grave al. (2019) proposes an unsupervised variant to the Orthogonal Procrustes alignment which jointly learns an orthogonal transform Ω ddd and assignment matrix P {0, 1 jaxn to minimize the Wasserstein distance between the embeddings subject to an unknown rotation. Three approaches utilize the Gram matrices of embeddings computing alignment initializations and matchings. Artetxe al. (201 8b) studied the alignment of dissimilar language pairs using Gram matrix based initialization and robust self- learning. Alvarez-Melis & Jaakkola (2018) proposed an Optimal Transport based approach using the Gromov- Wasserstein distance, GW (C, C", p. = minrel((p.g) A. L(C's, C² 1 Iw where
C, C are Gram matrices for normalized embeddings and I' is assignment, Aldarmaki et al. (2018) learns a unsupervised linear mapping between a source and target language with a loss at each iteration equal to the of squares of proposed source and target Gram matrices.
20

---

Published as a conference paper at ICLR 2021
J ABLATION STUDIES
J.1 AGGREGATE MAP PERFORMANCE FOR BLI TASKS
We provide ablation study quan- tify improvements associated with
Dict
1K
% Imprv.
5K
% Imprv.
the three modifications of our align- ment approach compared to standard
Procrustes
0.299
0.405
Procrustes alignment: (i) isotropic rre-processing (IP), (īi) inner prod- uct filtering (IPF) and (ii) weighted procrustes objectives (WP), on the XLING 1K and 5K BLI tasks. For seed dictionaries of size IK, improve- ments associated with each portion of FIPP are approximately the same while for seed dictionaries of size 5K. larger improvement is obtained due to isotropic ree-procsssing than inner product filtering or weighted procrustes alignment.
FIPP w/o IP
0.333
11.4%
0.430
6.2%
FIPP w/o IPF
0.335
12.0%
0,439
8.4%
FIPP w/o WP FIPP
0.336
12.4%
0.440
8.6%
0.344
15.1%
0.442
9.1%
FIPP SL (+14K)
0.406
35.8%
0.441
8.9%
Table 13: FIPP Ablation study of Mean Average Precision (MAP) on XLING IK and 5K BLI task.
J.2 PREPROCESSING ABLATION
We perform an ablation study to understand the impact of preprocessing on XLING 1K and 5K BLI performance both Procrustes and FIPP. Additionally, compare the isotropic preprocessing used in our previous experimentation with iterative normalization, a well performing preprocessing method proposed by /Zhang al. (2017).
For training dictionaries of size 1K, iterative normalization and isotropic preprocessing before running procrustes result in equivalent aggregate MAP performance of 0.3 We find that iterative normalization and isotropic preprocessing each achieve better performance on 11 of 28 and 12 of 28 language pairs respectively.
Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for IK training dictionaries (0.344) when compared iterative normalization (0.331). In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.
Method
Proc. + IN
Proc. + IP
FIPP + IN
FIPP IP
DE-FI
0.278
0.264
0.291
0.296
DE-FR
0.421
0.422
0.438
0.463
DE-HR
0.240
0.239
0.264
0.268
DE-IT
0.452
0.458
0.468
0.482
DE-RU
0.336
0.331
0.304
0.359
DE-TR
0.191
0.182
0.212
0.215
EN-DE
0.484
0.490
0.490
0.513
EN-FI
0.301
0.299
0.304
0.314
EN-FR
0.578
0.585
0.582
0.601
EN-HR
0,250
0.258
0.261
0.275
EN-IT
0.562
0.563
0.575
0.591
EN-RU
0.372
0.375
0.381
0.399
EN-TR
0.262
0.267
0.275
0.292
FI-FR
0.245
0.245
0.256
0.274
FI-HR
0.214
0.214
0.239
0,243
FI-IT
0.270
0.272
0.287
0.309
FI-RU
0.249
0.251
0.274
0.285
HR-FR
0.255
0.251
0.270
0.283
HR-IT
0,273
0.273
0.296
0.318
HR-RU
0.285
0.279
0.310
0.318
21

---

Published as a conference paper at ICLR 2021
IT-FR
0.612
0.612
0.622
0.639
RU-FR
0.362
0.359
0.374
0.383
RU-IT
0.384
0.386
0.400
0.413
TR-FI
0.181
0.177
0.193
0.200
TR-FR
0.226
0.226
0.235
0.251
TR-HR
0.155
0.154
0.169
0.184
TR-IT 0.232
0.230
0.247
0.263
TR-RU 0.173
0.177
0.191
0.205
AVG 0.316
0.316
0.331
0.344
Table 14: Mean Average Precision (MAP) of alignment methods on XLING with IK supervision dictionaries using either Iterative Normalization (IN) (Zhang et (017) or Isotropic Preprocessing (IP).
For training dictionaries of size 5K, iterative normalization and isotropic preprocessing before running procrustes result in comparable aggregate MAP performances of 0.422 and 0.424 respectively. We find that iterative normalization and isotropic preprocessing each achieve better performance on 8 of 28 and 16 of 28 language pairs respectively.
Utilizing isotropic preprocessing before running FIPP results higher aggregate MAP performance for 5K training dictionaries (0.442) when compared to iterative normalization (0.425). In this setting, isotropic preprocessing achieves better performance on all 28 language pairs.
Method
Proc. + IN
Proc. + IP
FIPP + IN
FIPP + IP
DE-FI
0.378
0.370
0.383
0.389
DE-FR
0,519
0.526
0,521
0.543
DE-HR
0.345
0.344
0.349
0.360
DE-IT
0,523
0.523
0.522
0,533
DE-RU
0.432
0.428
0.435
0.449
DE-TR
0.309
0.300
0.314
0.321
EN-DE
0.562
0.574
0.562
0,590
EN-FI
0.420
0.424
0.425
0.439
EN-FR
0.660
0.667
0.660
0.679
EN-HR
0.361
0.370
0.360
0.382
EN-IT
0,641
0.642
0.643
0.649
EN-RU
0.483
0.489
0.482
0,502
EN-TR
0.370
0.370
0.368
0.407
FI-FR
0.377
0.383
0.386
0.407
FI-HR
0.315
0.316
0.318
0.335
FI-IT
0.382
0.382
0.386
0.407
FI-RU
0.362
0.361
0.366
0.379
HR-FR
0.397
0.400
0.402
0.426
HR-IT
0,396
0.397
0.396
0.415
HR-RU
0.394
0.390
0.397
0.408
IT-FR
0.671
0.674
0.671
0.684
RU-FR
0.481
0.482
0.485
0.497
RU-IT
0.488
0.489
0.490
0.503
TR-FI
0.288
0.286
0.292
0.306
TR-FR
0.352
0.355
0.354
0.380
TR-HR
0.267
0.272
0.274
0,288
TR-IT
0.353
0.353
0.354
0.372
TR-RU
0.304
0.303
0.304
0.319
AVG
0.422
0.424
0.425
0.442
22

---

Published as a conference paper at ICLR 2021
Table 15: Mean Average Precision (MAP) of alignment methods on XLING with 5K supervision dictionaries using either Iterative Normalization (IN) (Zhang et al. 2017 or Isotropic Preprocessing (IP).
J.3 WEIGHTED PROCRUSTES ABLATION
In order to measure the effect of a Weighted Procrustes rotation on BLI performance, we perform an ablation both the IK and 5K XLING BLI datasets against the standard Procrustes formulation.
For training dictionaries of size IK, weighted procrustes achieves a marginally better aggregate MAP performance when compared to standard procrustes 0.344 vs 0.336 respectively. In 27 of 28 language pairs, weighted procrustes provides improved MAP performance over standard procrustes.
With training dictionaries of size 5K, weighted procrustes achieves marginally better aggregate MAP performance when compared to standard procrustes 0.442 vs 0.440 respectively, In 21 of 28 language pairs, weighted procrustes provides improved MAP performance over standard procrustes.
Method
FIPP (IK)
FIPP + WP (1K)
FIPP + (5K)
FIPP WP (5K)
DE-FI
0.286
0.296
0.386
0.389
DE-FR
0.446
0.463
0.541
0.543
DE-HR
0.260
0.268
0.358
0.360
DE-IT
0.471
0.482
0.534
0.533
DE-RU
0.350
0.359
0.444
0.449
DE-TR
0.207
0.215
0.321
0.321
EN-DE
0.508
0.513
0.589
0,590
EN-FI
0.314
0.314
0,440
0.439
EN-FR
0.594
0.601
0.678
0.679
EN-HR
0.271
0.275
0.382
0.382
EN-IT
0.581
0.591
0.648
0.649
EN-RU
0.393
0.399
0.503
0.502
EN-TR
0.285
0.292
0,405
0.407
FI-FR
0.268
0.274
0,403
0.407
FL-HR
0.233
0.243
0.330
0.335
FI-IT
0.294
0.309
0,402
0.407
FI-RU
0.274
0.285
0.376
0.379
HR-FR
0.276
0.283
0.424
0.426
HR-IT
0.305
0.318
0.413
0.415
HR-RU
0.307
0.318
0.405
0.408
IT-FR
0.628
0.639
0.683
0.684
RU-FR
0.381
0.383
0.498
0.497
RU-IT
0,400
0.413
0.500
0.503
TR-FI
0.194
0.200
0.303
0.306
TR-FR
0.249
0.251
0.379
0.380
TR-HR
0.169
0.184
0.284
0.288
TR-IT
0.253
0.263
0.372
0.372
TR-RU 0.195
0.205
0.317
0.319
AVG 0.336
0.344
0.440
0.442
Table Mean Average Precision (MAP) of FIPP on XLING 5K and 1K supervision dictionaries using with either weighted procrustes (WP) procrustes (P) rotation.
23

---

Published as a conference paper at ICLR 2021
K BLI PERFORMANCE WITH ALTERNATIVE RETRIEVAL CRITERIA
In order to measure the effect on BLI performance of different retrieval criteria, we perform experi- mentation using CSLS and Nearest Neighbors both the IK and 5K XLING BLI datasets.
We find that the CSLS retrieval criterion provides significant performance improvements, compared Nearest Neighbors search, both for Procrustes and FIPP both 1K and 5K training dictionaries. For IK training dictionaries, CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.051 and 0.043 for Procrustes and FIPP respectively. In the case of 5K training dictionaries, CSLS improves aggregate MAP, compared to Nearest Neighbors search, by 0.049 and 0.031 for Procrustes and FIPP respectively.
Method
Proc. + NN (1K)
Proc. CSLS (1K)
FIPP NN (IK)
FIPP CSLS (1K)
DE-FI
0.264
0.329
0.296
0.358
DE-FR
0.428
0.474
0.463
0.509
DE-HR
0.225
0.278
0.268
0.317
DE-IT
0.421
0.499
0.482
0.526
DE-RU
0.323
0.379
0.359
0.407
DE-TR
0.169
0.220
0.215
0.260
EN-DE
0.458
0.525
0.513
0.555
EN-FI
0,271
0.336
0.314
0.365
EN-FR
0,579
0.623
0.601
0.641
EN-HR
0.225
0.280
0.275
0.314
EN-IT
0.535
0,599
0.591
0.637
EN-RU
0.352
0.398
0.399
0.427
EN-TR
0.225
0.278
0.292
0.328
FI-FR
0,239
0.285
0.274
0.332
FL-HR
0.187
0.233
0.243
0.279
FL-IT
0.247
0.310
0.309
0.358
FI-RU
0,233
0.271
0.285
0.317
HR-FR
0,248
0.279
0.283
0.331
HR-IT
0.247
0.317
0.318
0.364
HR-RU
0.269
0.313
0.318
0.354
IT-FR
0.615
0.647
0.639
0.675
RU-ER
0.352
0.393
0.383
0.429
RU-IT
0.360
0.418
0.413
0.462
TR-FI
0.169
0.206
0.200
0.242
TR-FR
0.215
0.254
0.251
0.303
TR-HR
0.148
0.179
0.184
0.214
TR-IT
0.211
0.269
0.263
0.306
TR-RU 0.168
0.201
0.205
0.237
AVG
0.299
0.350
0.344
0.387
Table 17: Mean Average Precision (MAP) of FIPP and Procrustes on XLING with 1K supervision dictionaries using with either (CSLS) Nearest Neighbors (P) retrieval criteria.
Method
Proc. + NN (5K)
Proc. CSLS (5K)
FIPP NN (5K)
FIPP CSLS (5K)
DE-FI
0.359
0.433
0.389
0.440
DE-FR
0.511
0.560
0.543
0.576
DE-HR
0.329
0.386
0.360
0.400
DE-IT
0.510
0.555
0.533
0.568
DE-RU
0.425
0,456
0.449
0.461
DE-TR
0.284
0.334
0.321
0.357
EN-DE
0.544
0.584
0.590
0.608
24

---

Published as a conference paper at ICLR 2021
EN-FI 0.396
0.453
0.439
0.479
EN-FR
0.654
0.686
0.679
0.696
EN-HR
0.336
0.394
0.382
0.422
EN-IT
0.625
0.665
0.649
0.674
EN-RU
0.464
0.499
0.502
0.517
EN-TR
0.335
0.390
0,407
0.429
FI-FR
0.362
0,417
0.407
0.447
FI-HR
0.294
0.348
0.335
0.374
FIIT
0.355
0,427
0.407
0.442
FI-RU
0.342
0.392
0.379
0.405
HR-FR
0.374
0,434
0.426
0.461
HR-IT
0.364
0.427
0.415
0.449
HR-RU
0.372
0.430
0.408
0.438
IT-FR
0.669
0.698
0.684
0.703
RU-FR
0.470
0.522
0.497
0.537
RU-IT
0.474
0.517
0.503
0.529
TR-FI
0.269
0.326
0.306
0.346
TR-FR
0.338
0.391
0.380
0.413
TR-HR
0.259
0.310
0.288
0.329
TR-IT
0.335
0.389
0.372
0.408
TR-RU 0.290
0.323
0.319
0.338
AVG
0.405
0.454
0.442
0.473
Table 18: Mean Average Precision (MAP) of FIPP and Procrustes on XLING with 5K supervision dictionaries using with either (CSLS) Nearest Neighbors (P) retrieval criteria.
25


Reviews:
[{"review_id": "gts5QmcbCix", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== After the revision and the author response ===\nI would like to thank the authors for their very elaborate response. I acknowledge that I might have asked for too many experiments in my review, but this was mostly because I really wanted to understand the various aspects of the method (as one other reviewer mentions - it has too many 'moving parts') and ensure that all the comparisons are valid by comparing to the latest work. While it is not possible (due to time and computational restrictions) to run all the required experiments, I am still happy to raise my score. While the paper might not be so impactful, it might still be a pretty nice and quick-to-compute BLI baseline for any future developments in this area.\n=== ===\n\nThis paper presents a new method to learn projection/mapping-based cross-lingual word embeddings, termed FIPP (Filtered Inner Product Projection), which shows some promising results in supervised settings with 5K dictionaries. The paper is clearly written and easy to follow, and the main results across three tasks (bilingual lexicon induction + 2 downstream tasks) following a standard evaluation setup demonstrate that the method can often outperform its competitors. Another advantage of the method is the efficiency of learning the alignment; the method is several orders of magnitude quicker than standard choices such as VecMap or RCSLS. Although the paper does provide a valuable contribution to the field of learning (static) cross-lingual word embeddings, I still have quite some concerns and additional questions related to this work, and I am not ready to accept this work in its current format.\n\n*The authors mention (even in the abstract) that FIPP is applicable \"even when the source and target embeddings are of differing dimensionalities.\" However, I don't see how previous approaches are not applicable in these setups - the linear or non-linear mapping can be formulated between embedding spaces of different dimensionalities in a general case. Therefore, I don't see why this is important to stress with FIPP. This is some inaccurate argumentation imo.\n\n*The reconstruction loss obtained from the Gram matrices has not been used in bilingual settings, but this idea is directly borrowed from prior work on monolingual embeddings, so I do not see it as a methodological contribution. The second term of FIPP, however, reminds me of a recent work on instance-based mapping and learning local mapping functions, as proposed and validated by Glavas and Vulic (ACL 2020) and Nakashole et al. (several papers). However, the authors do not discuss this very relevant recent work nor provide any comparisons to these approaches, which seem closely related.\n-- I would also suggest the authors to include some other very relevant recent models in their comparison. For instance, what about the following papers:\n- https://arxiv.org/abs/2004.13889 (Mohiuddin et al.)\n- https://doi.org/10.1162/coli_a_00374 (Mohiuddin and Joty, Computational Linguistics 2020)\n- https://doi.org/10.18653/v1/p19-1018 (Patra et al., ACL 2019)\n- https://www.aclweb.org/anthology/2020.acl-main.766/ (Anastasopoulos et al., ACL 2020)\n- https://openreview.net/forum?id=S1l-C0NtwS (Wang et al., ICLR 2020)\n- There are other recent papers...\n\nThis field is a rapidly evolving field, and comparing to the models which were SotA in 2019 is definitely not sufficient, especially if the good results are supposed to be the main contribution of the paper.\n\n*Along the same line as the previous point, preprocessing of word embeddings can be quite crucial to the final performance, I wonder if the authors have also tried a well-performing preprocessing strategy from Zhang et al. (ACL 2019), also with the baselines. Would this change the ranking of the models in the comparison?\n\n*Concerning the previous point, the paper lacks a comprehensive ablation study. Given that it does blend several different components (e.g., multi-loss, preprocessing regime, dictionary size), it is quite underwhelming to see that the authors did not conduct any experiments to further understand what components of the model are more crucial to the final performance, and how different choices of hparams affect the performance. Further, there are no experiments with different dictionary sizes and we're left wondering if the method would work with a smaller number of training examples (e.g., 500 examples or 1K examples as evaluated by Vulic et al., EMNLP 2019). What about typologically more dissimilar languages, etc.? What about self-learning strategies? Is it possible to improve the results with FIPP by running several iterations of the method? Have the authors tried that? How does preprocessing affect the final performance? What about the hyper-parameter epsilon? How is that hyperparameter related to language (dis)similarity?\n\n-- In sum, a complete piece of work should provide answers to all these questions, and it is still quite difficult to judge whether this particular model will really advance the induction of cross-lingual word embeddings. Does it also rely on the approximate isomorphism assumption (Soogard et al., ACL 2018) and how does language distance relate to the transfer loss? \n\n*Notes on motivation: While working on static cross-lingual word embeddings is still an important research thread, the paper should provide a thorough discussion on why these models are still important in the context of the current mainstay of multilingual/cross-lingual NLP based on massively multilingual models such as multilingual BERT, XLM-R, multilingual T5, etc. An open question is what applications (besides bilingual dictionary induction) could still benefit from static cross-lingual word embeddings.\n\nMinor comments:\n- Mikolov et al. did not propose using the Orthogonal Procrustes solution per se - they solved the problem via SGD and did not impose any orthogonality constraint (which is needed to solve it analitically via the Procrustes method - see later work of Xing et al., and the survey paper of Ruder et al.)\n\n- The title is not exactly accurate: 'multilingual embedding alignment' would suppose aligning more than 2 languages (3 and more), while the authors, as most of prior work, learn only bilingual word embeddings. Another question concerns the possibility to extend this idea to shared spaces with more than 2 languages. How feasible is this?", "extra_scores": {}, "extra_text": {}}, {"review_id": "QYoY6a3m3Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to align word vector spaces based on a dictionary of pairs (e.g. translations) that retains \"common geometric structure\" (inner products within each vector space for a subset of word pairs). In essence the approach applies preprocessing to each vector space, finds a projection of one vector spaces to the (lower-dimensional) vector space minimising a combination of a reconstruction and transfer loss, and then aligns the two using weighted procrustes to resolve rotational symmetries.\n\nThe topic is of interest to the community and related papers on vector space alignment have been published at previous ICLRs. Essentially, this paper combines insights from a number of different works, with good results. However, overall the solution is relatively complex with a lot of moving parts.\n\nStrengths:\n* The paper is largely well written and easy to follow.\n* The proposed method can be applied if vector spaces have differing dimensionality\n* Results compare well to existing methods\n\nWeaknesses:\n* As the initial projection is not constrained to be orthogonal, the geometric structure within the projected vector space is altered (only relevant if dimensionality is the same). This is usually seen as a big advantage of procrustes-based solutions and not really discussed by the authors. How much does the solution found using this method deviate from an orthogonal solution?\n* Introducing the isotropic preprocessing makes it difficult to compare the alignment procedures, as both word vector spaces are altered significantly (As shown in Arora where it significantly impacts performance on semantic similarity). The authors should explore the impact of this preprocessing also for other methods (e.g. procrustes).\n* No comparison between weighted procrustes and regular procrustes to address the rotation symmetry for the inner product approach. Is it crucial to use the weighted variant here?\n* Unclear how well this approach works in conjunction with retrieval strategies other than nearest neighbour - previous work found this to be a significant factor for performance.\n* Only anecdotal evidence for comparison of SGD and low-rank approximation\n* Related work: Mikolov 2013a did not propose a procrustes solution to the alignment problem. Instead, Artexte 2017 and Smith 2017 showed (independently) that procrustes provides an analytical solution to the optimisation problem proposed in Mikolov 2013 under the orthogonality constraint.\n\nSmaller issues / suggestions:\n* I found the \"preview\" of the technical approach in the introduction more confusing than clarifying, as some of the details mentioned later on where missing, e.g. that rows in $X_s$ and $X_t$ refer to the same word based on the dictionary.\n* It would be great to get a more intuitive \"feel\" how the alignment that respects the \"geometric structure\" would be different to a procrustes alignment, e.g. by showing a few example translations\n* The plural of corpus is corpora\n* You assume dimensionality $d_2 < d_1$ but don't really state it anywhere?\n\nOther (not considered in review):\n* The CCA results are quite strong - If the authors are using the CCA implementation in scikit learn I would like to point out that it actually uses an iterative estimation of partial least squares, which works much better than a more naive implementation of CCA (as e.g. demonstrated in https://arxiv.org/abs/1905.05547)", "extra_scores": {}, "extra_text": {}}, {"review_id": "75XtNVyUIc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\nThis paper proposes a new approach for Bilingual Lexicon Induction (BLI). To do this, the paper proposes the use of a FIPP objective, which consists of the linear combination of reconstruction and transfer loss terms. It derives the Gram matrix that minimizes this loss, followed by a rank-constrained semi-definite approximation to obtain an aligned embedding whose Gram matrix is close to the one minimizing the FIPP. The proposed approach is deterministic and has been shown to be efficient. Experiments also demonstrate the method works well, both on BLI and on downstream tasks.\n\n**Strengths:**\n* The proposed method is interesting and elegant, and novel in the context of BLI. FIPP also performs well empirically, demonstrated both by its superior performance on BLI (as shown on with an evaluation on 28 XLing pairs) and on its comparable performance in the downstream CLDC and XNLI tasks. While the proposed approach does not do the best on the presented downstream tasks, it seems to perform well (reasonably comparably to the best) consistently, which none of the other methods seem to do.\n* FIPP allows for BLI using embeddings of different dimensions, and performs well even when the embeddings are of different dimensions, as shown experimentally for en-de. To the best of my knowledge, no other method supports this.\n* The proposed method is quite efficient because it can be represented as a few matrix operations. This has been demonstrated by comparing its runtimes with two other SotA approaches (VecMap and RCSLS).\n\n**Comments/Questions:**\n* One drawback is that the proposed method does not perform as well for the 1k case (Appendix). Some analysis about why that might be the case would have been nice. That being said, the same is the case for the other supervised methods as well (PROC-B aside). I also appreciate the authors' transparency in this regard.\n* The authors present the downstream performance on 2 (CLDC, XNLI) of the 3 tasks (the other being CLIR) used in [5]. Is there a reason these 2 specific tasks were chosen, but not CLIR?\n* The proposed method uses a weighted Procrustes objective to rotate $\\widetilde{X}_s$, which is a minor detail, but whose use is, as far as I am aware, novel in the context of BLI. An ablation here to show how much the weighted Procrustes objective improves over the regular Procrustes transform (such as MUSE [4] does) would have been nice to see (Eg: a row \"FIPP w/o WP w/ P\" in Appendix Table 7).\n* GeoMM [3] is very efficiently optimizable as well. Do the authors have an estimate about how FIPP performs compared to GeoMM, with respect to the average time they each take on CPU?\n* The Appendix had some nice experimental details, additional experiments and insights. It would be good to add the Appendix to the end of the main paper (as opposed to separately) if possible.\n\n**Minor points:**\n* For XNLI, [1] should probably be cited along with [2], since XNLI was introduced by [1] based on the English-only MultiNLI corpus introduced by [2]\n* Section 6.2 didn't seem to flow very well with the rest of Section 6 (which follows Section 5 nicely). Consider moving it to before Section 5 if possible (I'm not entirely certain where it would fit best, possibly as a new Section 3.3).\n\n[1] Conneau, Alexis, et al. \"XNLI: Evaluating cross-lingual sentence representations.\" arXiv preprint arXiv:1809.05053 (2018).\n[2] Williams, Adina, Nikita Nangia, and Samuel R. Bowman. \"A broad-coverage challenge corpus for sentence understanding through inference.\" arXiv preprint arXiv:1704.05426 (2017).\n[3] Jawanpuria, Pratik, et al. \"Learning multilingual word embeddings in latent metric space: a geometric approach.\" Transactions of the Association for Computational Linguistics 7 (2019): 107-120.\n[4] Conneau, Alexis, et al. \"Word translation without parallel data.\" arXiv preprint arXiv:1710.04087 (2017).\n[5] Glavas, Goran, et al. \"How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions.\" arXiv preprint arXiv:1902.00508 (2019).\n\n\n\n============================================================\n\n**Update:**\n\nA few  suggestions on the latest version:\n\n* Adding pointers from Table 1 and 2 to their full equivalents in the Appendix would be helpful.\n* The Appendix ablations are now thorough and exhaustive, but parsing them and digesting what they represent is a little tricky. Adding a one-line summary (much like in Section 5.`1) would be very helpful here. For example, talking about in how many pairs FIPP+IP outperforms Proc.+IP,  FIPP+IN outperforms Proc.+IN, FIPP+IP outperforms FIPP+IN.\n\nand a few comments:\n\n* The sheer number of experiments that the authors have performed in general (and in particular in the relatively short time period of this rebuttal) is impressive, and is in my experience indicative of an extremely good, well-designed and easy to iterate upon framework/code-base. If my guess is correct, I urge the authors to release their code if possible, because I believe it would greatly help anyone working in this space (or even consuming BLI's output in a downstream task).\n* It is also very heartening to see how much R1's suggestion helps improve the 1k case! However, because the efficiency and the lack of need of a GPU are big selling points, the drawback of adding self-supervision (10x the time, need for a GPU) might be a good caveat to add in Section 6.1 as opposed to keeping it till Appendix A.\n\nOverall, I would like to thank the authors for their very detailed and thorough response,  and for taking into account so much of all the reviewers' feedback to make the paper clearer with much more comprehensive ablations. The paper, its techniques proposed and their performance and efficiency, and the detailed experiments it conducts will be helpful for both the field of BLI and other fields relying on it. In view of this, this paper is a clear accept in my opinion, and I raise my score from 7 to 8.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for bilingual lexicon induction.
The proposed method is efficient, it optimizes a reconstruction and transfer loss.
Extensive experiments are reported, and the methods provides improvements over prior work.
Overall, the paper brings together prior ideas in a useful way.
Decision: Accept (Poster)