Normalization of Language Embeddings for Cross-Lingual Alignment
Authors: Prince Osei Aboagye
Jeff Phillips
Yan Zheng
Chin-Chia Michael Yeh
Junpeng Wang
Wei Zhang
Liang Wang
Hao Yang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: cross-lingual word embeddings
natural language processing

Abstract: Learning a good transfer function to map the word vectors from two languages into a shared cross-lingual word vector space plays a crucial role in cross-lingual NLP. It is useful in translation tasks and important in allowing complex models built on a high-resource language like English to be directly applied on an aligned low resource language.  While previous approaches have identified Procrustes and other techniques to align language models with some success, it has recently been identified that structural differences (for instance, due to differing word frequency) create different profiles for various monolingual embedding. When these profiles differ across languages, it correlates with how well languages can align and their performance on cross-lingual downstream tasks.  In this work, we develop a very general language embedding normalization procedure, building and subsuming various previous approaches, which removes these structural profiles across languages without destroying their intrinsic meaning.  We demonstrate that meaning is retained, and alignment is improved on similarity, translation, and cross-language classification tasks.  Our proposed normalization clearly outperforms all prior approaches like centering and vector normalization on each task and with each alignment approach.

The best multilingual NLP approaches typically do not jointly learn single embedding, since words 19 of the same language tend cluster, and thus are not useful for translation and cross-lingual learning tasks. Rather, after learning individual embeddings, the standard approach is to map word vectors Normalization of Language Embeddings for Cross-Lingual Alignment
Anonymous Author(s)
Affiliation Address email
Abstract
Learning good transfer function to map the word vectors from two languages into a shared cross-lingual word vector space plays a crucial role in cross- lonss-lingual NLP. It is useful in translation tasks and important in allowing complex models built on high-resource language like English to be directly applied on an aligned low resource language. While Procrustes and other techniques can align language models with some success, has recently been identified that structural differences (for instance, due differing word frequency) create different profiles for various monolingual embedding. When these profiles differ across languages, it corre- lates with how well languages can align and their performance on cross-lingual downstream tasks. In this work, we develop a very general language embedding normalization procedure, building and subsuming various previous approaches, which removes these structural profiles across languages without destroying their intrinsic meaning. We demonstrate that meaning retained and alignment is improved on similarity, translation, and cross- cross-language classification tasks. Our proposed normalization clearly outperforms all prior approaches like centering and vector normalization on each task and with each alignment approach.
2
10
11
13
15
Introduction
17
21 from multiple languages into a shared cross- lingual word vector space [13. This shared space creates cross-lingual word embedding (CLWE) [22] 43). These serve valuable tool for transferring 23 data across different languages, understanding cross- linguiinn differences, and cross-lingus transfer for downstream tasks, such direct translation [16] 20 244), cross-lingual information retrieval 442
25
cross-lingual document classification 23 and cross-lingual dependency parsing 17 39
26 A common element of almost all CLWE methods is the use of rigid, orthogonal transformation mapping one embedding onto another so they inhabit a shared linguistic space. An orthogonal transformation is special class of transformations that can be interpreted the space of (in our 29 case, high-dimensional) rotations around the origin, and also allowing a mirror flip. This family of that
27
transformations preserves (a) linear and (b) angular properties. By linear properties, the straight-line Euclidean distance between elements preserved, as are more powerful properties like analogies (c.g.. Paris France + Italy Rome). Angular properties refer to measuring angles
31
39 between pairs of points (from the origin), and as a result, cosine distance is preserved. Given a embeddings, the classic Procrustes method,
correspondence between pairs of objects across 35 provides a closed-form solution which minimizes the sum of Euclidean distances. Moreover, the vectors are all first made unit vectors, then this also maximizes the sum of cosine similarities (10).
Submitted to Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37
Under this framework, there has been a flurry of work significantly improving CLWE model per- formance along two directions. supi-supee and unsupervised models make these approaches require less input, and more amenable to lower-resource languages. For example Bootstrap Procrustes (PROC-B) [S 41 is semi-supervised in that it starts with a small pairwise correspondence (of 500-1000 words), aligns those to infer larger correspondence, and repeats applying Procrustes alignment. Methods like MUSE [8] are unsupervised, and use a GAN to estimate a correspondence
38
39
40
41
42
43
before applying Procrustes procedure.
44
The second direction is preprocessing the embeddings before applying the Procrustes alignment. These involve methods like removing the mean, removing principal components, and normalization which we will discuss in depth later. In principle, these methods aim remove the geometry of data intrinsic to particular languages (but not shared across languages) while preserving similarity properties as assured by orthogonal alignment. The space of transformations allowed under orthogonal 48
45
46
47
49
alignments is quite large, and we make the point that unless this data geometry is "normalized" it inhibits the alignment from optimizing over the entirety of this large space.
50
51
Finally, we note that methods like Canonical Correlation Analysis (CCA) [3]. Discriminative Latent Variable (DLV) [36., and Ranking-based optimization (RCSLS) 21 have also been applied towards finding an orthogonal alignment (or pair of alignments) which minimizes a different optimization function since the objective function may not align with sums of squared Euclidean of cosine distance g [38. Unlike the others, the RCSLS method notably does not require rigid transformation.
52
53
54
55
The focus of this paper is embedding preprocessing, and is agnostic the method of alignment used afterward, whether it is Procrustes-based, or optimizing something else.
57
Our contribution. This work proposes new and general approach preprocessing word embed- dings, subsuming many previous approaches. The key is Spectral Normalization which regularizes the spectral properties of monolingual embeddings by setting all of the top singular vectors have the same singular value. However, it leaves alone the smaller singular value; these capture important information and cannot be zeroed out, but making them the same value the top singular vectors introduces too much noise. Spectral normalization already performs as well as the best previous approaches on alignment and translation tasks, and since it applies fairly uniform stretching the embeddings it does not distort monolingual similarity performance. Moreover, we show layering Spectral Normalization within an iterative sequence with also centering and vector length normaliza- tion improves results further. We first demonstrate this improvement on the standard translation task (BLI). We also show that this normalization preserves the core ontological structure of embeddings across languages, and that applying our normalization before aligning a low resource language to English improves performance on topic classification and on a natural language inference task.
59
61
82
69
64
85
86
87
68
69
70
71
2
Existing Methods for Orthogonal Vector Spaces Alignment
72 Given a language L, our starting point is an embedded representation of a set of n words. Indexing words from i = 1 n, each word is associated with vector z'_ E Rd. And let X2 = {zt...... ......... . 74 be the set of n words as their vector representation. These vector representations (derived by methods
75 like word2vec (28). Glo (33], FastText (4) are chosen words with similar pairwise cosine
76 similarity are found in the similar local context in large text corpora on which they are trained.
77
Higher-level linear structure is shown to emerge, such as concept subspaces and analogies 29
78 The focus of this paper is on aligning embeddings of two languages L1 and L2. Each embedding XL, and Xiz, only is designed to ensure pairwise relationships between its word vectors, but the actual coordinates of those vectors do not have any explicit meaning. Yet. previous work has clearly demonstrated that there exists significant overall structural similarity, and alignment seeks to make correspondences between those structures for translation and joint understanding.
80
81
82
83 Most methods start with known correspondence (or build one) between a set of K words in
84 two languages, wlog let these be the same first K indexed words in those languages, denoted
85 X = {=2,--
{xt ... _xK J. Then the Procrustes Problem solve for an 86 orthogonal matrix W arg minw | W- X X l1ž. There exists a simple solution 2 [8 [1)
=E. ...f.. and X La
87 as V = UVT where UEVT = svd(X)
X L, (XX)) Dev et. [10] also point out that if all vectors
normalized first. then this procedure also maximizes the sum of cosine similarities.
2

---

89 2.1 Pre-processing Embeddings before Orthogonal Alignment
90
turns out directly aligning embeddings from two languages (even using the "optimal" Procrustes solution) does not provide the best possible joint embedding for translation tasks. While word meaning appears to hold a similar structure, languages have other properties such as differing word frequency, and this for instance leads to more frequent words having longer vectors in embeddings. structure tends to interfere with alignment. As result, number
91
92
99
94 This extra language-specifg of techniques have been developed to "normalize" the embeddings before Procrustes (or other) aligmment. This in some sense allows the word meaning to dominate the optimization tasks without other confounding factors. We review the most common normalization approaches.
96
98
Mean Centering (C) subtracts the mean of all vectors in an embedding from each vector in that embedding. The result is that the mean of all vectors is O. This is rigid transformation, and does not change the Fuclidean distance between any pair of points in an embedding, and also preserves any linear property like analogies (e.g., Paris France + Italy se Rome). Dev et. al. [IO] points out that this is the first step (followed by the Procrustes orthogonal transformation) to minimize the sum
99
100
101
102
103
of squared Euclidean distances among paired words, under any rigid transformation. However, this
104
does change the cosine distance between pairs of points.
105
Lenghh Normalization (L.) makes each vector have a 2-norm equal to 1, but retains its direction from the origin 2 45 This preprocessing step does not change the cosine distance between any pair of points in an embedding. But, it does change the Euclidean distance between pairs of points.
106
107
108
Despite these contrasting goals, these two normalizations each turn out to be individually effective in regularizing the geometry of the embeddings, and allow for better CLWE. Wang et. al. [47. realized doing both was even more effective, and showed that iterating these two steps achieves the state- of-the-art way to preprocess, we denote as I-C+L. Iterative Normalization transforms monolingual word embeddings to have umit-length and zero-mean simultancously (in practice they terminate this iterative process after a few steps before it achieves these two goals exactly).
109
110
111
112
113
114
PCA Removal (PR) computes the principal component analysis (PCA) of an embedding, and then projects away from the direction of the top principal component, removing it (3.. Mu et. al. [I observed that the top singular values typically do not encode essential semantic relationships between words but rather align strongly with word frequency. Also, they notice that the top principal values much larger than the other values. After eliminating d/ 100 principal components, where d is the dimension of a word representation, they achieve better performance on both intrinsic and extrinsic tasks. Sachidananda et. al. (37) applied this simple pre- peeppocesssing approach to their Filtered Inner Product Projection (FIPP) alignment method, and they significantly improve on the BLI task.
115
116
117
119
120
121
122
2.2
Spectral Statistics of Embeddings
128
Dubossarsky ef. al. 11 recently documented how cross-lingual alignment is strongly affected by the spectral statistics of monolingual embeddings. We stack the embedded vectors x² E Rd as rows in a n x d matrix A E Raxd The SVD decomposes into UEV where U and V contain the left and right singular vectors, and the singular values σ 02 2 ... 2 aa 2 0 are on the diagonal of Σ. The effective rank of A is a smoother analog to rank (when there is noise in low rank components),
124
125
126
127
defined er(A) = e (E) where H(E) = - Ede ō. log ã; with ã; - Ei G;. The effective condition number Keo (A) = (r))) which replaces the numerator (of condition number, a.a) with the more robust singular value at the effective rank. This is desired to be small in stable data sets. The joint effective condition number measures the harmonic mean of the effective condition 2A)r(A)eenAA))
128
129
131
132
number across two matrices A, A' as ECOND-HM( A, A') Manl Aar(A)+ ( The singular value gap measures how similar the singular value sequences are between two matrices as svG(A,A')
133
Ei
(log a; - log 2 These should both be smaller, for more comparable data sets.
134
135
Dubossarsky et. al. (II] applied these to monolingual embeddings and demonstrated that the performance of several CLWE methods were closely tied to these spectral properties. Basically embeddings align better if they are better jointly conditioned, especially measured via joint effective condition number and the singular value gap. Motivated by this idea, we propose methods that spectrally normalize embeddings improving these statistics while retaining intra- cettaddnbedding meaning.
196
138
3

---

140
3
New Normalization Methods
We introduce more direct and more general techniques to normalize monolingual word embeddings 142 to more effectively prepare them for alignment. The goal is to remove language-specific geometry
141
143
while maintaining the intrinsic similarity and structure captured within them.
3.1
Geometric Median Normalization
145
Iterative Normalization enforces individual word embeddings to have unit length and each monolin- gual embedding to have zero mean through an iterative technique. Wang et. al. [47] showed that iterating solutions for these distinct goals will eventually converge
146
147
solution which satisfies both.
148
In this paper, we observe that both goals can be done in one shot without iterating by solving the Fermat-Wehbe Weber problem 27 25 This dates to the 17th century, and corresponds with identifying the geometric median of a point set. Formally, the goal is point x² E F that minimizes the sum of
149
150
distances from n anchor points {ai..... an C I& which are not collinear: x" min, Σi= ||
151
152
a.ll Several methods [6] 12 32] have been proposed; the most popular is the Weiszfeld's is algorithm
(Weiszfeld, see Appendix Ab. It is folklore that the solution x* satisfies that 0 - >i 1
do not know of written proof, so prove this in Appendix A.T for completeness.
155
Using this characteristic of the geometric median, we can simultaneously enforce monolingual word embeddings to have unit-length and zero-mean in just one step. This can be done using the Geometric Median normalization (GeoMediaN) algorithm (as Algorithm ID. Given a monolingual word embedding A, we compute the geometric median x", and "center" the data on this point, and unit length normalizes the centered embedding.
157
159
Algorithm 1 Geometric Median Normalization: (eoMediaN(4)
1: x² + Weiszfeld (A)
2: for all E A do <- 04
0.-2"
3: return A
180 After these steps, all vectors are unit length, and because of the folklore property (Theorem A.Ip. the mean of those points is also O. As result, we can state the following property.
162
Theorem 3.1. The output of (eoMediaN(A) centered and length normalized.
Despite the Geometric Median Normalization algorithm's 's ability to enforce unit-length and zero- mean in just one step, we will observe that it does not perform especially well on the BLI task. Both GeoMediaN and I-C+L achieve one of many solutions which achieve these joint goals. We next investigate another one that works better: it preserves meaning and structure, and removes songuage-speciffe geometry allowing improved alignment.
164
186
188
3.2
Spectral Normalization
The predominant effect of unit-length and zero-mean normalization on monolingual word embeddings is that it makes embedding vectors from a language lic on a hypersphere with the center of the hypersphere centered at the origin. However, this does not take into account how the word embeddings vectors are spread out or clustered on the hypersphere. Approaches like PCA removal and mean centering have the effect of reducing the top principal component top singular vector. As result, the spectral properties are extreme, it can help regularize them. However, this approach can be a bit blunt. PCA removal makes the top singular value exactly 0, the condition number becomes infinite. Other quantities like the effective condition number, Keff (AA, do however, tend to decrease.
170
172
174
176
177 To this effect, we propose a new algorithm Spectral Normalization that more gently regularizes
178 the spectral properties of word embeddings; see Algorithm 2 We will then combine with other
approaches to again ensure the embedding vectors lie the unit sphere.

---

Algorithm 2 Spectral Normalization (SpecNorm(A,3))
1: Compute svd A) = LEVT; Let D E IRd be diagonal matrix.
2: Compute 7)- v [ where d is the dimension of the word embedding
3: for i = I,....d .d do
4: if (En 5n) then Du: +- En/(3n)
5: else Du: = 1.
6: return AVD-
180
Given a monolingual word embedding A it updates part of the spectral properties of A whole, using on parameter 8 2 Based on an average of singular values 7j = /|lAIl /d, if value is above 5 times that average, it adjusts it to Sn. Hence, all of the top directions are given the same singular value. Otherwise, it is below f1,, it is considered a minor effect (some are quite small, and fairly noisy), and it is left alone. If these small ones are completely zeroed out, the critical information within is destroyed. However, if these small ones are also given the same value (i.e., 30)) then components which do not contribute to the most prevalent aspects of vectors similarity is given more importance, and we observed (see Section 4.1 that the usefulness of the embedding decreased.
181
182
183
184
185
187
188
Iterative Spectral Normalization. Spectral normalization makes the most sense (see Appendix G in setting where the vectors already centered, and also unit length. While SpecNorm does not change the center of the data, it does not maintain the length of individual vectors. As such, we advocate combining these methods into a single iterative algorithm: I-C+SN+L in Algorithm 3
190
191
Algorithm 3 Iterative Spectral Normalization with C+L normalization (I-C+SN+L(A, #lter)
1: for #lter steps do
2: A < Center A
3: < SpecNorm(A)
4: A < Unit length normalization of A
5: return
192
We observe in Figure Ithat this process significantly improves the spectral properties, compare to any other approach. Without preprocessing (None), the languages (EN: English, DE: German, HI: Hindi, JA: Japanese shown) have large effective condition numbers indicating that there is large disparity between meaningful singular values. Note the y-axis is in log scale. Hence, aligning these languages without normalization would likely restrict alignment among top singular vectors, not allowing enough degree of freedom align corresponding words.
183
194
195
196
197
198
In contrast, after preprocessing when these values are more uniform, rotations among the dimensions containing the top principal components will not have influence the data distribution, and can fully optimize the alignment between words. Moreover, Figure [ shows that I-C+SN+L most decreases the effective condition number, joint effective condition number, and singular value gap.
200
202
Further, these values are fairly uniform across languages, despite great variation beforehand (as
203
shown with None). In fact, I-C+SN+L. is much more effective than other methods.
Effective Pank
Effective Concition Number
Joint Effective Condition Number
Singular Value Gap
Nose
LCISNIL
ECONB (10)
000001100 (1a2)
Gookiiduhh
674
11
EN
16
2N.00
17648
076
11M
Langaage Pair
Language Paire
Figure 1: Spectral Measures of four (4) monolingual word embeddings, before (None) and after applying various normalization methods.
5

---

4
Experimental Analysis
204
205
We provide an evaluation of our proposed preprocessing methods using 8 language embeddings pretrained on Wikipedia [4] of each language: Croatian (HR), English (EN), Finnish (FI), French
206
207
(FR), German (DE), Italian (IT), Russian (RU), and Turkish (TR). We use the 300-dimensional TastText 4 embeddings, and all vocabularies are trimmed to the 200K most frequent words.
208
209
A Alggmment evaluation tasks: BLI We evaluate and compare our proposed preprocessing methods mostly on the Bilingual Lexicon Induction (BLI) task, word translation task. We discuss two more global applications Cross-lingual document classification (CLDC), and Cross-lingual transfer for natural language inference (XNLI) later. BLI is more direct, and has become the de facto evaluation task for CLWE models. For words in the source language, this task retrieves the nearest neighbors in the target language after alignment to check if it contains the translation. It reports the mean average
211
213
215
precision (MAP) [1], which is equivalent to the mean reciprocal rank (MRR). of the translation.
Unless stated otherwise, reported values baseline methods are taken from [15] and use the Google
217
Translate (GTrans) dictionary from [13 We trained (aligned) using 1k, 3k and 5k source words and evaluated (tested) separate 2k source test queries, unless noted otherwise.
219
Aligmment Algorithms. We evaluated and compared the result several supervised rigid- transformation CLWE models the evaluation benchmarks using proposed methods. All have publically available code, links are found in the reference citation. These include Canonical Correlation Analysis (CCA) [3]. Procrustes (PROC) 2
221
222
10,, Bootstrapping Procrustes (PROC-B) [15. and Discriminative Aatent-Variabll (DLV) 36 as discussed in Section [ We also a rigid alignment. In a few places, we also compare with VECMAP [3] as an example of an unsupervised alignment process. This should only use the geometry of the global embedding structure. e.g., derived from the natural ontology, and our normalization method still helps when using this approach.
38
45
223
consider Lanking-Based Optimization (RCSLS) (2T which
225
228
227
228
4.1
Hyperparameter Tuning
Our main proposed algorithm I-C+SN+L has few simple parameters. To avoid overfitting, we choose these through vross-validation on English (EN) and a held-out set languages Hindi (HI). Russian (RU), Chinese (ZH), Japanese (JA), Turkish (TR). Ten (10) Language pairs of the form EN-X and X-EN were considered. The hyperparameters 3 E {1, 2, 4,5 and #Iter (number of iterations) e
230
232
233
[1, 5) were fine-tuned for I-C+SN+L.
We used the publicly available MUSE translation dictionary g for hyperparameter tuning. The Procrustes alignment algorithm was trained on 5k source words and evaluated 1.5k source test queries. We reported the mean average precision (MAP) in Table [or 3 E {1, 2, 3, 4, 5) and with #lter 2, 3, 4, 5]. We observe the value of B 2 consistently the best threshold (although any B 2 2 performed similarly). These singular values were normalized, and those below / times the
234
236
238
average we judged noise, and left is. However, the result did not change much with respect to the number of iterations.
240
Table 1: Cross- Validation for Hyperparameter Tuning: MAP after Procrustes for 10 language pairs.
3
#Iter=1
#Iter=2
#Iter=3
#lter=4
#lter=5
1
0.363
0.340
0.328
0.322
0.317
2
0.385
0.386
0.386
0.386
0.386
3
0.381
0.384
0.384
0.384
0.384
4
0.381
0.382
0.382
0.382
0.382
5
0.380
0.381
0.381
0.381
0.381
241 The tie between the #Iter hyperparameter was broken using their performance on thirteen English word similarity benchmarks; see Appendix C for more details. In Table below, we report the
242

---

243
average Spearman rank coefficient score on the word similarity task (None means no normalization).
244
(3, #Iter) (2.5) achieved the highest score. So hereafter, we applying I-C+SN+L with the hyperparameter (1,, #lter) = ().
Table 2: Monolingual word similarity Task; Average Spearman rank coefficient
None
(3,tlter) = (2 (3, #lter) = (2,3) (3, #lter) = (2,4) (3, = (2,
0.651
0.67077
0.67101
0.67108
0.67111
246
Note that the proposed approach (I-C+SN+L) only increased this score for these similarity tasks, so showed no signs of distorting inherent information. Although Spectral Normalization does not exactly preserve the linear properties or angular properties (as centering and length normalization do, one each, respectively). it does not suffer ill effects. We hypothesize this is because it is somewhat uniformly stretching words along the major modes of variation, and is effectively removing information not relevant to meaning, like frequency of occurrence. This benign effect is contrast to other spectral adjustments (removing small singular values, or setting all to the same value) shown in Appendix C
247
248
250
252
253
BLI Performance across Normalization and Alignment Algorithms
254
We compare and evaluate the BLI performance (MAP) of various normalization algorithms from previous works to our proposed algorithms. Using the MUSE translation dictionary, we trained CCA, PROC, PROC-B and RCSLS on 5k source words and evaluated on 1.5k source test queries. The following normalization algorithms were used in the comparison analysis: (PCA Removal) ( GeoMediaN (Geometric Median Normalization), C+I. (Mean centering and Length normalization, 1 round), I-C+L (Iterative Mean centering and Length normalization, 5 rounds) [47). SN (Spectral Normalization, 1 round), C+SN+L (Mean centering, Spectral Normalization and Length normal- ization, round), and I-C+SN+L (Iterative Mean centering. Spectral Normalization and Length normalization. 5 rounds). Specifically, evaluated 18 language pairs, i.e., English (EN) from/to Bulgarian (BG), Catalan (CA), Czech (CS), German (DE), Spanish (ES), Korean (KO), Thai (TH) and Chinese (ZH), separate from hyperparameter tuning. The average is reported in Table all results G For almost all algorithms I-C+SN+L achieves the best scores (and especially on L2 EN, often considerably better). The only exceptions
255
257
259
261
263
285
are in Appendix
on non-rigid RCSLS when C+SN+L (with no iteration) or just SN (with C+I.) performs slightly better. So, Spectral Normalization, and in particular I-C+SN+L, is shown the best way to normalize languages before alignment.
267
Table 3: BLI performance (MAP) on aligning EN XL2 and X1, -EN
Methods EN EN-Xia
Methods X EN
Normalization
CCA
PROC
PROC-B
RCSLS
CCA
PROC
PROC-B
RCSLS
None
0.358
0.365
0.377
0.394
0.398
0.399
0.405
0.428
PR
0.394
0.391
0.404
0.373
0.434
0.430
0.442
0.425
GeoMediaN
0.393
0.391
0.400
0.379
0.433
0.432
0.440
0.429
C+I
0.393
0.394
0.408
0.404
0.439
0.437
0.445
0.464
I-C+L.
0.394
0.395
0.410
0.406
0.439
0.438
0.448
0.460
SN
0.391
0.394
0,408
0.405
0.440
0,438
0.451
0.468
C+SN+L
0.395
0,396
0,413
0.407
0.444
0.444
0.458
0.466
I-C+SN+L
0.396
0.398
0.414
0.406
0.445
0.446
0.461
0.466
289
We also compute the average BLI MAP score across all 28 language pair for more direct comparison to prior work [15), summarized in Table g and Appendix [ All results are in Appendix H We compare I-C+SN+L (denoted with SN) against no normalization on various dictionary sizes: 1k, 3k and 5k source words and evaluated 2k source test queries. In all cases, I-C+SN+L significantly improves over the baseline. This includes improvement over RCSLS which is non-rigid, so in
271
272
273
274
principle could "learn" adjustments similar to our normalization in the process of alignment. We also
275
tested on VECMAP, an unsupervised approach; I-C+SN+L preprocessing also improves this result from 0.375 to 0.410.
278
7

---

Table 4: Summary of BLI performance (MAP), average scores for all 28 language pairs. No normalization results from [5]
against I-C+SN+L. (denoted SN).
PROC-B5N
RCSLSSN
PROC5N
PROC-B
CCASN
RCSLS
PROC
DLV
CCA
DLV
Dict
IK
.289
.314
.299
.326
.379
.407
.289
.332
.331
.331
3K
.378
.401
.384
.408
.398
.415
.381
.429
.415
.427
5K
.400
.423
.405
.429
.403
.452
.437
.460
4.3
Downstream Tasks
278 We conclude by demonstrating that Spectral Normalization not only improves in direct translation tasks, but also captures important global structure that generalizes from high resource language (i.c., English, EN) to lower resource languages. In both examples, powerful classifier is trained on the EN embedding (after normalization), and then we demonstrate that after lower resource language (e.g., German, DE) has been normalized and align the analysis task can be directly applied that language. And in particular, adding the simple process of our normalization (I-C+SN+L) dramatically improves the results over not doing that step.
282
283
Cross-linguss Document Classification (CLDC). The CLDC task builds a topic classification
286
using language model on high resource language (in our case English EN) across 15 topics.
The TED CLDC corpus assembled by [19] used for training and evaluation. Following (T5] simple CNN was used to train. Table Summarizess the average Fl-score for all topic classifiers
288
on language pairs. The CLWEs induced by PROCSN PROC-BSM DLVSN and RCSLSSM (using I-C+SN+L) outperformed the baseline result (with no normalization) on the CLDC task, significantly
289
improving the best average score from 0.421 0.461. Glavas et.al. [15] use only 12 of 15 topics, but could not confirm which, so we re-ran all baselines using all 15 topics.
292
Table CLDC performance (micro-averaged Fi scores). Cross-lingual transfer EN-X
Model
Dict
EN-DE
EN-FR
EN-IT
EN-RU
EN-TR
Avg
PROC
5K
.366
.288
.338
.288
.278
.306
PROCSN
5K
.436
.366
.427
.517
.511
.451
PROC-B
3K
.364
.304
.299
.336
.317
.324
PROC-BSN
3K
.448
.396
.423
.522
.517
.461
DLV
5K
.419
.336
.397
.493
.458
.421
DLVSN
5K
.433
.323
.406
.499
.472
.427
RCSLS
5K
.466
.397
.403
.403
.406
.415
RCSLSSN
5K
.468
.500
.443
.488
.394
.459
283
Cross-lingual Natural I Language Inference (XNLI). We evaluated the CLWE on a cross-lingual natural language inference (XNLI) task. We used a multi-lingual XNLI corpus created by [9] which is a collection of sentence pairs from the English MultiNLI corpus [44] translated into 14 languages.
294
285
296
The MultiLLI corpus contains 433k sentence pairs with the labels entailment, contradiction, and
297
neutral. The intersection between XNLI languages and BLI languages result in four XNLI evaluation
pairs: EN-DE, EN-FR. EN-TR and EN-RU. We use the training setup in [5] with the Enhanced 298
Sequential Inference Model (7) on English after normalization. First, we aligned normalized versions
300
of each language onto the normalized EN embedding to obtain the shared cross-lingual embedding, Then we used the 5k test pairs from the XNLI corpus to evaluate each language alignment. Table
302
shows the result for PROC, PROC-B, and RCSLS alignments (DLV and VECMAP transform
EN embedding in the process, were omitted). We compare against the same procedure without

---

normalization from Glavas et.al. [15] (L-C+SN+L normalization denoted SN As in other experiments,
304
305
our normalization improves the average test accuracy with each alignment approach.
Table 6: XNLI performaac (test set accuracy)
Model
Dict
EN-DE
EN-FR
EN-TR
EN-RU
Avg
PROC
5K
.607
.344
.668
.585
.544
PROCSN
5K
.611
.638
.542
.596
.597
PROC-B
3K
.615
.532
.573
.599
.580
PROC-BSN
3K
.624
.638
.548
.601
.603
RCSLS
5K
.390
.363
.387
.399
.385
RCSLSSN
5K
.499
.482
.504
.556
.510
306
5
Conclusion & Discussion
307
We introduce new way to normalize embeddings, based on spectral normalization, for use in creating cross-lingual word embeddings. Our approach generalizes previous approaches, and effectively removes much of the clustering of words based on properties other than the similarity which encodes meaning. When used individually preprocess monolingual embeddings, our approach allows any alignment procedure to find better alignments: resulting in improved performance on direct translation tasks as well as cross-lingual topic classification and natural language inference tasks.
308
309
310
312
Social impacts. The vast majority of NLP research and cutting-edge advancements are in English. This disadvantages those who primarily operate in other languages, with less developed models, or less data to train models. As large language models are the cornerstone of most NLP research and development in English, one of this work's 's main goals is to port these advances to other languages, and those who use them. This will help unlock this technology to many others around the world. As with most models, this accuracy and improvement may vary across tasks and languages.
314
315
316
318
319
W hile language models have many positive use cases including improving accessibility, better recommendations, and increased automation, they have some negative effects as well. These include requiring potentially large computational and hence environmental cost, encoding and exacerbating
320
321
322
bias, and aiding automatically generating fake or deceitful content. While this paper is unlikely to change the desire to use embeddings, it aims to reduce the burden of use and increase the effectiveness lower-resource settings. And in particular to port models trained in English to other languages. This would reduce the cost of retraining in other languages if the English model can be reused, easing environmental costs. We support the maturing efforts attenuating bias in all such embeddings. And while we acknowledge the possibility of this work aiding in the automated creation of deceitful content and the harm can cause, we believe the many benefits outweigh the harms.
323
324
325
326
328
329
L Limitations.
The overarching goal in this line of work is to port the many advances built on embeddings from high-resource settings (like built on the English language), to lower-resource settings (like Turkish). This work can apply powerful model built Turkish after the embeddings
330
English language model
332
(e.g., for natural language inference) and automatically invoke have been aligned. However, the alignment will not be any better than the low-resource embedding.
334 If the embedding is too noisy or limited, then the analysis will likely not be effective.
335
Also, this work focuses on non-contextual embeddings like FastText, but not contextual ones like RoBERTa which have proven almost universally more effective in NI.P on English. While in principle normalization function and alignment could be applied to contextual settings, we are unaware of any technique for learning these mappings. We believe it could be important future work.
396
337
338
339
Data, Code, and Experiments. All existing methods are compared with publicly available code with publicly available data, with links above or in references. The exception is code for CLDC and XNLI is shared by Glavás ef.al. [15]. Everything is run with default parameters; the exception is RCSLS were we follow the suggested hyperparameter selection strategy 21 (with learning rate in
340
342
{1. 10, 25, 50} and epoch number in {10, 20 Our new code for SpecNorm is in Appendix

---

References
344
345
[1] Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of Human Language Technologies: The 2009 Anmual Conference of the North American Chapter of the Association for Computational Linguistics, pages 19-27, Boulder, Colorado, June 2009, Association for Computational Linguistics.
348
347
348
349
350 [2] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural language Processing, pages 2289-2294, Austin, Texas, November 2016. Association for Computational Linguistics.
352
353
354
[3] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789-798, Melbourne, Australia, July 2018. Association for Computational Linguistics. https
355
356
pages
357
//github. wtt.thu...m/aattteeeeeee com//arteexeem/eecmaa
359
[4] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
360
351
362
[5] Elia Bruni, Gemma Boleda, Marco Baroni, and technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 135-145, 2012.
Tran. Distributional semantics in
384
385
[6] Jacques Antonin Chatelon. Donald W Hearn, and Timothy J Lowe. A subgradient algorithm for certain minimax and minisum problems. Mathematical Programmingg 15(1):130-145, 1978.
386
387
[7] Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada, July 2017. Association for Computational Linguisties.
369
371
[8] Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data, 2018
372
[9] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
374
376
377
378
[10] Sunipa Dev, Safia Hassan, and Jeff M Phillips. Absolute orientation for word embedding alignment. Knowledge and Information Systems, 2021.
379
380
[11] Haim Dubossarsky, Ivan Vulić, Roi Reichart, and Anna Korhonen. The secret is in the spectra: Predicting cross-lingual task performance with spectral similarity measures. In EMNLP, 2020.
381
382
[12] James W Eyster, John A White, and Walter W Wierwille. On solving multifacility location problems using a hyperboloid approximation procedure. AlIE Transactions, 5(1):01-06, 1973.
383
384
[13] Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462-471, Gothenburg, Sweden, April 2014. Association for Computational Linguistics.
386
387
[14] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. In Proceedings of the 10th international conference World Wide Web, pages 406-414, 2001.
389
10

---

391
[15] Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. How to (properly) evaluate cross- monss-lingual word embeddings: On strong baselines, comparative analyses, and some miscon- ceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710-721, 2019.
392
383
384
com/ www.m///oooo////////oooooooooooo xling-lniaar eval?utm,
395
source=catalyzee. coo
396
[16] Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed representations without word alignments. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 748- Lille, France, 07-09 Jul 2015. PMLR.
398
400
[17] Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu, Cross- Cross-lingual dependency parsing based on distributed representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1234-1244, Beijing, China, July 2015, Association for Computational Linguistics.
401
402
403
404
405
[18] Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1406-1414, 2012.
406
407
[19] Karl Moritz Hermann and Phil Blunsom. Multilingual models for compositional distributed semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 58-68, Baltimore, Maryland, June 2014. Association for Computational Linguistics.
409
410
411
412
[20]
Geert Heyman, Ivan Vulié, and to combine word-level and character-level representations. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1085-1095, Valencia, Spain, April 2017. Association for Computational Linguistics.
Moens. Bilingual lexicon induction by learning
413
414
415
416
[21] Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979- 2984, Brussels, Belgium, Nectober-Novebbr 2018. Association for Computational Linguistics. https://gi t/tp:hbb........///
418
420
htt///rreeeeeeeeee alignment
422
[22]
Mladen Karan, Ivan Vulié, Anna Korhonen, and Goran Glavaš. Classification-based self- learning for weakly supervised bilingual lexicon induction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6915-6922, Online, July 2020. Association for Computational Linguistics.
428
424
425
426
[23] Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed representations of words. In Proceedings of COLING 20/2, pages 1459-14744 1474, Mumbai, India, December 2012. The COLING 2012 Organizing Committee.
427
428
429
[24] Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049 Brussels, Belgium, October-November 2018. Association for Computational Linguisties.
431
432
433
[25]
Robert F Love, James G Morris, and George O Wesolowsky. Facilities location. 1988.
434
[26] Minh- Kinh-Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In Proceedings of the seventeenth conference on computational natural language learning, pages 104-113, 2013.
435
436
[27] EA Maxwell. 100 great problems of elementary mathematics: their history and solution. By 438
Heinrich Dorme, translated by David. Ustin Pp. x, 393. $2.00. (Dover) The Mathmmatcal Gazette, 50(372):213-213, 1966.
11

---

440
[28]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 1xXiv:130.37850 2013.
441
442
[29]
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguisties: Human language technologies, pages 746-751, 2013.
443
444
445
446
[30]
George A Miller and Walter G Charles. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1-28, 1991.
447
[3]] Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postpro- cessing for word representations. CoRR, (bss/1777 2017.
449
450
[32] Michael I. Overton. A quadratically convergent method for minimizing a sum of euclidean norms. Mathematical Programming, 27(1):34-63, 1983.
451
[33] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics.
453
454
455
456
[34] Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337-346, 2011.
457
458
459
[35] Herbert Rubenstein and John B Goodenough. Contextual correlates of synonymy. Communica- tions the ACM, 8(10):627-633, 1965.
460
[36] Sebastian Ruder, Ryan Cotterell, Yova Kementchedjhieva, and Anders Søgaard. A discriminative latent-variable model for bilingual lexicon induction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 458-444 468, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. https: / com/ wtw.://annnne/attttttaaaaaaaaeea
462
464
485
[37] Vin Sachidananda, Ziyi Yang, and Chenguang Zhu. Filtered Inner Product Projection for Multilingual Embedding Alignment. arXiv e-prints, page ar arXiv:2006.03652, June 2020.
467
468
[38] Samuel L. Smith, David H. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. CoRR, abs/1702.03859,
470
2017.
471
[39] Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. On Difficulties of Cross-Lingus Transfer with Order Differences: A Case Study Dependency Parsing. arXiv e-prints, page 1rXiv:18111 1.00500 November 2018.
473
474
[40] Yehuda Vardi and Cun-Hui Zhang. The multivariate 11-median and associated data depth. Proceedings of the National Academy of Sciences, 97(4): 1743):44231 2000.
[41] Ivan Vulié, Goran Glavaš, Roi Reichart, and Anna Korhonen. Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? arXiv e-prints, page arXiv:1909.01633 September 2019.
477
[42] Ivan Vulic and Marie-Francine Moens. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2015.
480
[43] Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime G. Carbonell. Cross-lingual alignment vs joint training: comparative study and A simple unified framework. CoRR, 1 2019.
483
12

---

485
[44] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018.
486
487
488
Association for Computational Linguistics.
489
[45] Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1006-1011, Denver, Colorado, May-June 2015. Association for Computational Linguistics.
491
482
488
494
495
[46] Dongqiang Yang and David Martin Powers. Verb similarity on the taxonomy of WordNet. Masaryk University, 2006.
496
497
[47] Mozhi Zhang, Keyulu Xu, Ken-ichi Kawarabayashi, Stefanie Jegelka, and Jordan Boyd-Graber. Are girls neko or shōjo? cross- Minsull alignment of non-isomorphic embeddings with iterative normalization. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 3180-31 3189, Florence, Italy, July 2019. Association for Computational Lin- guistics. https:// tigg.b.t...... wtcshhu...omaaaaeeoeeeee.eeeeeee htwp://hagggmziiiiteemmnnntuttt htt.pp/uwwttuuourrrrtttaatrt...o
498
500
Checklist
503
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
505
contributions and scope? [Yes]
508
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
508
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
509
510
2. If you are including theoretical results...
511
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
512
(b) Did you include complete proofs of all theoretical results? [Yes]
513
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
515
516
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
517
518
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
520
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
521
522
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
523
(a) If your work uses existing assets, did you cite the creators? [Yes]
524
(b) Did you mention the license of the assets? [N/A]
525
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See code in Appendix [i]
$26
527
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
529
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
530
531
5. If you used crowdsourcing or conducted research with human subjects...
532
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

534
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
535
536
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "QEbRg0bwa8d", "reviewer": "Reviewer_sMU3", "summary": "The paper proposes a new normalization method for monolingual word embeddings before aligning them between different languages. The normalization is based on spectral properties of the monolingual embeddings and aims to level the contributions of those leading singular values while maintaining those smaller ones at the same time. Further inspired by previous works that centers and units the length of monolingual embeddings, the authors propose an iterative normalization that combines the spectral normalization with centering and unit length normalization. The proposed method achieves better performance in both bilingual lexical induction (BLI) tasks as well as downstream tasks like cross-lingual document classification (CLDC) and cross-lingual natural language inference (XNLI). ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper builds on a well-defined framework of cross-lingual word embedding alignment and focuses on the normalization step. The proposed method is quite novel compared with previous approaches as it emphasizes on the spectrum properties of the word embeddings. The related work is also adequately cited.\n\nQuality: In general, the paper is technically sound. However, line 18 claims \"The best multilingual NLP approaches typically do not jointly learn a single embedding\". This should be supported by citations or other evidence since some of the SOTA multilingual methods like XLM, XLM-R and m-BERT all learn joint multilingual embeddings. About evaluation, table 4, 5 and 6 should include other strong baselines just in the same way as table 3. \n\nClarity: There are few clarity issues of the paper: 1) How is the proposed method \"subsuming various previous approaches\"? 2) What is the position of GeoMediaN? Is it a baseline or a proposed method? In section 3 it is introduced as a proposed method but in section 4 it is treated mostly like one of the baselines. 3) It is better to be more concise in scientific writing and improve the statement like line 57 \"or optimizing something else\". \n\nSignificance: The empirical result is clearly written, and implementation details are also included. However, the performance gain of the proposed method (I-C+SN+L) against strong baseline (I-C+L) is marginal as shown in table 3. And the same comparison is overlooked in table 4, 5 and 6. This insignificance of empirical superiority may limit the contribution of this work to the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I agree with the limitations mentioned by the authors. ", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "H66qSWWUQr-", "reviewer": "Reviewer_qE9E", "summary": "This paper presents a new approach to normalizing (static) word embeddings prior to learning the mapping between them to induce an improved shared cross-lingual word embedding (CLWE) space. The main idea is to improve shared cross-lingual spaces not via learning more sophisticated mapping functions or by designing a new CLWE learning paradigm, but rather to affect the properties of independently trained monolingual embeddings to increase their alignability. In other words, normalization can be seen as a pre-processing step that is orthogonal to the actual mapping/projection functions, and improving normalization can have a positive impact on the final shared space. The core contribution of the paper is therefore the spectral normalization method, built on top of the recent advances and insights from prior work, which yields gains over other normalization techniques from prior work in the intrinsic task of Bilingual Lexicon Induction (BLI) and several transfer tasks (which rely on the CLWEs).", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "UPDATE AFTER READING THE AUTHOR RESPONSE:\n\nI would very much like to thank the authors for providing a detailed response. The response did clarify some of my questions (e.g., how the hparams were chosen), but I still think that the paper should be revised and made stronger and more impactful, e.g., 1) by running similar experiments and empirically demonstrating the usefulness of the proposed method to contextualised embeddings; 2) by comparing to some more recent models (e.g., Ormazabal et al, ACL 2021); 3) by acknowledging other recent research (especially spectral methods) on isometry/isomorphism of word embedding spaces. Further, I am still unsure about the relationship of embed-then-align projection methods and joint learning methods and I need more evidence from prior research that joint training does not work beyond BLI (i.e., I am unsure if the claims made in the response are factually correct). \n\nI do agree with the authors that \"there are numerous applications for embeddings outside of NLP (images, graphs, spatial data, merchants, biology) where there is no concept of contextual embedding.\", but then the authors should also demonstrate the usefulness of their method outside of NLP, not leaving it as speculation only.\n\nAll in all, as said - while I do like the paper and its main goals overall, I still hesitate to recommend it for acceptance.\n\n\n===============\nOriginality/novelty\n===============\n\nWhile the paper isolates and discusses the crucial sub-problem of WE normalization very well, P1) it is heavily based on a plethora of prior research, and therefore feels more like a consolidation of the ideas from prior research; P2) it does not cite some of the more recent research focused on the same goal - improving CLWEs by learning more isomorphic monolingual embedding spaces to facilitate the mapping; P3) it also makes some vague statements about the importance of mapping-based CLWE approaches in modern NLP. I discuss these three points/aspects P1-P3 in what follows. \n\nRegarding P1, the core contribution of the paper is combining the previous findings and normalization methods of Artetxe et al. (ACL 2018) and Zhang et al. (ACL 2019) with recent insights of Dubossarsky et al. (EMNLP 2020) that spectral properties of monolingual WE spaces are largely correlated with the final task performance. This paper makes a step towards the synergy of these two ideas, introducing very simple methods that 'zero out singular values' in order to create spectrally more similar monolingual WEs. This then yields some gains in the standard suite of evaluations (borrowed from the work of Glavas et al. ACL-19). While this new spectral normalization method is a nice new insight, I find it too thin to justify a full-fledged paper, especially since the motivation for why exactly this flavour of spectral normalization works (and some other possible variants don't).\n\nRegarding P2, there has been some recent research that tried to achieve the same effects as this paper, but those works are not cited nor compared against as baselines. Listing a few papers here:\n- https://arxiv.org/pdf/2012.15715.pdf -> this paper also aims to impact the monolingual WE structure to learn better mappings (via context anchoring)\n- https://aclanthology.org/2020.repl4nlp-1.7.pdf -> the motivation of this very relevant paper is exactly the same as the motivation in this paper.\n- https://direct.mit.edu/coli/article/46/2/257/93364/Unsupervised-Word-Translation-with-Adversarial -> this paper also aims to bypass the problems with non-isomorphic spaces through 'translating' the problem to a latent space (induced by autoencoders), other work from the same authors also follows a similar line of thinking.\n\nRegarding P3, unfortunately the work stays at the level of static CLWEs without providing a proper justification nor discussion if these methods can advance contextual word embeddings which are now predominant in modern-day NLP. What is more, can the insights on normalization help us in other tasks? The current evaluation tasks (except for BLI) have been superseded by much more powerful transfer methods, while the gains in BLI (as the task where static CLWEs still hit near-SotA performance) are still very modest, which questions the usefulness of the proposed normalization technique.\n\nFurther, the authors also argue that joint learning of CLWEs cannot match mapping-based approaches. This is not in true in general -> mapping-based approaches have become popular mostly due to their low data demands and the fact that they bypass parallel corpora. However, joint learning with parallel data (or hybrid approaches which combine mapping-based and joint learning) are very competitive, see, e.g.:\n- https://openreview.net/pdf?id=S1l-C0NtwS\n- https://arxiv.org/pdf/1906.05407.pdf\n\n===============\nStrengths\n===============\n\n*The paper is nicely motivated for most parts (although it should stress much more why we require mapping-based approaches and static CLWEs to begin with) and is easy to read.\n*The build-up to introducing the actual spectral normalization method makes it easy to trace how the authors thought about the problem, and the insights (or the lack of them) from prior work.\n*The method is quite simple to apply and seems widely applicable to any mapping-based CLWE induction method, yield to consistent gains (which do vary across different tasks, but it is always beneficial to apply the normalization method).\n*The experiments (when it comes to the focus on static CLWEs) are comprehensive and executed in a sound way.\n\n===============\nWeaknesses\n===============\n\n*The paper is mostly a continuation of similar work in the same subfield, applying and leveraging insights made by other researchers - e.g., it is heavily based on the work of Dubossarsky et al. (EMNLP-20). There doesn't seem to be enough substance for a full publication.\n*Given the results, I am still wondering whether any practitioner will apply this method in practice - also, does the method help with more distant language pairs?\n*The paper also relies on some arbitrary choices (the criterion for zeroing-out singular values; the hparam beta) - some baseline methods like the simple iterative-norm method of Zhang et al. are non-parametric, so perhaps they are already in disadvantage in empirical comparisons? Is there any true impact of doing multiple iterations?\n*The paper focuses only on static CLWEs, which also limits the impact of this work - since the work is largely empirical, it is unclear who will use the main outcomes of the work.\n\n===============\nQuality and clarity\n===============\n\nIt is very clear what has been done, and the experimental setup along with the description of the core method is very clear as well. My main criticism does not refer to the style of the paper (which is very good), but is more concerned with its novelty, impact, comparisons to (relevant) baselines (see above).\n\n===============\nImpact and Significance\n===============\n\nAs mentioned, the paper covers one niche subfield of learning static CLWEs (with a particular suite of models), and given that it focuses on the paradigm which is now largely surpassed, I am not sure that the paper will have any major impact. It might be cited as another normalization method, but unless a stronger theoretical and empirical motivation is provided, as mentioned before in my review - it is unclear who will use the main outcomes of the work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper lists Turkish as an example of a low-resource language, but according to Joshi et al. (ACL-20)'s analysis - there are much lower-resource languages, and this should also be mentioned in the paper.\n\nThe authors state that they focused only on static CLWEs as the limitation of the current work. This is indeed a limitation, but this paper already should have ideally provided some extensions that reach towards contextualised WE learning.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "Ef9ExazBEwK", "reviewer": "Reviewer_z8YG", "summary": "The paper presents a preprocessing technique called ’spectral normalization’ for cross-lingual word alignment and shows improvements across a range of alignment techniques and downstream tasks. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I in many ways really like the paper. The empirical evaluation is thorough, and the normalization trick is neat. I think the paper suffers from three fundamental limitations: \n\na) It is not entirely clear how spectral normalization relates to previous discussions of spectrality (some of the references cited, as well as the ones below). I would have liked to see this fleshed out in more detail. \nb) The normalization trick is the sole contribution; feels a little ’thin’ for a full length paper. \nc) Interest in alignment of word embeddings is on the decline, since word embeddings are not used much any longer. The paper would benefit by the authors’ take on why this problem remains interesting (whether for theoretical or practical reasons). \n\nMissing references for related work discussing spectrality in the context of cross-lingual vector space alignment, e.g.: \n\nhttps://www.aclweb.org/anthology/P16-2080.pdf\nhttps://www.aclweb.org/anthology/Q18-1014.pdf\nhttps://www.aclweb.org/anthology/P18-1072.pdf\n\nSee also Aldarmaki’s thesis: https://www.proquest.com/openview/97f58b5d99e2ed81065054a170f2dcda/1?pq-origsite=gscholar&cbl=18750&diss=y\n\n(I’m omitting older related work, e.g., [http://www.ra.ethz.ch/cdstore/www2010/www/p751.pdf], and basic background, e.g., [https://arxiv.org/abs/math/0203240], which I also think the authors should cite.) \n\nNote the term ’spectral normalization’ has been used before with different meanings. It may make sense to address this somewhere. See, for example, [https://www.aclweb.org/anthology/K19-1064.pdf, https://www.aclweb.org/anthology/2020.findings-emnlp.64/]. \n\nThe paragraph in line 44-50 needs references. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper presents thorough empirical analyses of a normalization technique which can potentially help cross-lingual transfer of NLP models. The focus on word alignment favors morphologically simple languages, though. In §5, you discuss related limitations, but I think this section would benefit from a discussion of morphology. A related, but broader question (which I don't think you necessarily have to address, but should be mindful of), is the impact of strong normative writing systems, and the open problem of how best to approach the majority of low-resource languages that are predominantly spoken languages. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a technique for adjusting the spectral properties of word embeddings that is used to make cross-lingual alignment more straightforward. Experiments show that the proposed method results in better cross-lingual generalisation and bilingual lexicon induction. Overall the method and results are sound, but the normalisation trick is found to be a bit limited as a contribution for this paper, especially given the large number of papers that have appeared related to this issue over the last couple of years (adjusting spectral properties of word embeddings, improving bilingual alignment, etc). Finally, all reviewers remark that this is a well-written, clear paper- in conjunction with a bit more content, it would be a much stronger contribution.
Decision: Reject