Understanding Linearity of Cross-Lingual Word Embedding Mappings
Authors: Xutan Peng
Mark Stevenson
Chenghua Lin
Chen Li
Venue: Accepted by TMLR ()

Abstract: The technique of Cross-Lingual Word Embedding (CLWE) plays a fundamental role in tackling Natural Language Processing challenges for low-resource languages. Its dominant approaches assumed that the relationship between embeddings could be represented by a linear mapping, but there has been no exploration of the conditions under which this assumption holds. Such a research gap becomes very critical recently, as it has been evidenced that relaxing mappings to be non-linear can lead to better performance in some cases. We, for the first time, present a theoretical analysis that identifies the preservation of analogies encoded in monolingual word embeddings as a *necessary and sufficient* condition for the ground-truth CLWE mapping between those embeddings to be linear. On a novel cross-lingual analogy dataset that covers five representative analogy categories for twelve distinct languages, we carry out experiments which provide direct empirical support for our theoretical claim. These results offer additional insight into the observations of other researchers and contribute inspiration for the development of more effective cross-lingual representation learning strategies. 


Published in Transactions on Machine Learning Research (06/2022)
Understanding Linearity of Cross-Lingual Word Embedding Mappings
Xutan Peng
z.peng@shef.ace uk
Mark Stevenson
mark.stevenson &shef uk
Chenghua Lin
c.lin accbefff.cc.cm uk
Chen Li 1
palchenli@ttencent.com
Deportment of Computer Science, The University of Shefffiedd Applied Research Center, Tencent PCG
Reviewed on OpenReview; openrreeee.. net/ per/ffrmm? id- 8luuyXubug.
Abstract
The technique of Cross-Lingual Word Embedding (CLWE) plays fundamental role in tackling Natural Language Processing challenges for low-resource languages. Its dominant approaches assumed that the relationship between embeddings could be represented by linear mapping, but there has been no exploration of the conditions under which this as- sumption holds. Such a research gap becomes very critical recently, as it has been evidenced that relaxing mappings to be non-lincar can lead to better performance in some cases. We, for the first time, present a theoretical analysis that identifies the preservation of analogies encoded in monolingual word embeddings as a necessory and sufficient condition for the ground-truth CLWE mapping between those embeddings to be lincar. On novel cross- lingual analogy dataset that covers five representative analogy categories for twelve distinet languages, carry out experiments which provide direct empirical support for our theoret- ical claim. These results offer additional insight into the observations of other researchers and contribute inspiration for the development of more effective gross-lingual representation learning strategies.
1 Introduction
Cross- LinguaLingual Word Embedding (CLWE) methods encode words from two or more languages in a shared high-dimennsionll space which vectors representing lexical items with similar meanings (regardless of lan- guage) are closely located. Compared with alternative techniques, such as cross-lingual pre-trained language models, CLWE is orders of magnitude more efficient in terms of training corpora and computational power requirement
P As result, the topic has received significant attention as promising means to support Natural Language Processing (NLP) for low-resource languages (including ancient languages) and has been used for a range of applications, e.g., Machine Translation (Herold et al. 2021). Sentiment Analysis (Sun
ct al. 2021 Qucstion Answering (Zhou et al. 2021 and Text Summarisation (Peng ct al. 2021b).
For example, Kim et al. 2020 show that inadequate monolingual data size (fewer than one million sentences) is likely to lead to collapsed performance of XLM [Lample & Conneau 2019) for elynologically cluse language pairs. Meanwhile, CLWE can casily align word cmbbeddnngs for languages such African Amharic and Tigrinya for which only have millions of
tokens (Zhng et al. 2020) available.
[Connean al. 2020 trained 500x Tesla V100 GPUs, whereas the training of VerMap Artetxe al. 2018 can be finished single Titan Xp GPU.
For example, X.M

---

Published in Transactions on Machine Learning Research (06/2022)
The most successful CLWE approach, CLWE alignment, learns mappings between independently trained monolingual word vectors with very little, or even no, cross-lingual supervision (Ruder et al.) 2019). One of the key challenges of these algorithms is the design of mapping functions. Motivated by the observation that word embeddings for different languages tend to be similar in structure (Mikoloy et al.) 2013b), many researchers have assumed that the mappings between cross-lingual word vectors are linear (Frrqqu & Dyer 2014 Lample ct al. 2018b; Li et al. 2021).
Although models based this assumption have demonstrated strong performance, it has recently been questioned. Researchers have claimed that the structure of multilingual word embeddings may not always be similar (Sogaard ct al., 2018: Dubossarsky et al. 2020, Vulié et al. 2020). which led to the emergence of approaches relaxing the mapping linearity (Glavaš & Vulić, 2020 Wang et al.) (021a) or using non- linear functions (Mohiuddin et al.) 2020 Ganesan et al., 2021). These new methods can sometimes out- perform the traditional linear counterparts, causing a debate around the suitability, or otherwise, of lincar mappings. However, to the best of our knowlodge, the majority of previous CLWE work has focused on empirical findings, and there has been no in-depth analysis of the conditions for the linearity assumption.
This paper approaches the problem from a novel perspec- tive by establishing a link between the linearity of CLWE mappings and the preservation of encoded monolingual analogies. Our work is motivated by the observation that word analogies can be solved via the composition of semantics based on vector arithmetic (Mikolov et al.] 2013ep and such linguistic regularities might be transfer- able across languages. More specifically, we notice that if analogies encoded in the embeddings of one language also appear in the embeddings of another, the correspond- ing multilingual vectors tend to form similar shapes (see Fig. D) suggesting the CLWE mapping between them should be approximately lincar. In other words, sus- pect that the preservation of analogy encoding indicates the linearity of CLWE mappings.
king
roi
man
homme
queen
woman
renie
femme
Figure 1: Wiki vectors (see and French (right) analogy word pairs based on PCA (Wold et al. 1987.. NB: We manually ro- tate the visualisation to highlight structural sim- ilarity.
4.3 of English (left)
Our hypothesis is verified both theoretically and empirically. We make a justification that the preservation of analogy encoding should be a sufficient and necessory condition for the linearity of CLWE mappings. To provide empirical validation, we first define indicators to qualify the linearity of the ground-truth CLWE mapping (SLMP) and its preservation of analogy encoding (SPAE). Next, we build a novel cross-lingual word analogy corpus containing five analogy categories (both semantic and syntactie) for twelve languages that pose pairs of diverse etymological distances. We then benchmark SLMP and SPAE on three representative series of word embeddings. In all sctups tested, we observe significant correlation between SIMP and SpaE, which provides empirical support for our hypothesis. With this insight, we offer explanations to why the linearity assumption occasionally fails, and consequently, discuss how our research can benefit the development of more effective CLWE algorithms. We also recommend the use of SPAE to assess mapping lincarity in CLWE applications. We release our data and code at https:: https://github. com cwwp//2o0020222//A1...
This paper's contributions are summarised as:
Introduces the previously unnoticed relationship between the lincarity of CLWE mappings and the preser- vation of encoded word analogies.
Provides a theoretical analysis of this relationship.
Describes the construction of a novel cross-lingutt analogy test set with five categories of word pairs aligned across twelve diverse languages.
Provides empirical evidence of our claim and introduces SPAE to estimate the analogy encoding preser- vation (and therefore the mapping lincarity). We additionally demonstrate that SPAE can be used as indicator of the relationship between monolingual word embeddings, independently of trained CLWEs.
2

---

Published in Transactions on Machine Learning Research (06/2022)
Discusses implications of these results, regarding the interpretation of previous results and well as the future development of cross-lingual representations.
2 Related Work
Linearity of CLWE Mapping, Mikolov et al. (2013b) discovered that the vectors of word translations exhibit similar structures across different languages. Researchers made use of this by assuming that mappings between multilingual embeddings could be modelled using simple linear transformations. This framework turned out to be effective numerous studies which demonstrated that linear mappings are able to produce accurate CLWEs with weak or even no supervision (Artetxe al. 2017 Lample et al. 2018b: Artetxe al.
2018 Wang al. 2020 Li et al. 2021).
One way in which this is achieved is through the application of a normalisation technique called "mean cen- tring". which (for cach language) subtracts the average monolingual word vector from all word embeddings. so that this mean vector becomes the origin of the vector space (Xing et al., 2015 Artetxe et al. 2016 Ruder et al. 2019). This step has the effect of simplifying the mapping from being affine (i.e., equivalent to a shifting operation plus a lincar mapping) to linear by removing the shifting operation.
However, recent. work has cast. doubt on this linearity assumption, leading researchers to experiment with the use of non-linear mappings. Nakashole & Flauger 20188) and Wang et al. (202 pointed out that structural similaritics may only hold across particular regions of the embodding spaces rather than over their entirety. Søgaard et al. (2018) examined word vectors trained using different corpora, models and hyper-parameters. and concluded configuration dissimilarity between the monolingual embeddings breaks the assumption that the mapping between them linear. Patra et al. (2019) investigated various language pairs and discovered that higher etymological distance associated with degraded the linearity of CLWE mappings. Vuuli et al.) 2020)) additionally argued that factors such as limited monolingual resources may also weaken the linearity assumption.
These findings motivated work on designing non-linear mapping functions in an effort to improve CLWE performance. For example, Nakashole (2018) and Wang et al. (2021a) relaxed the linearity assumption by
combining multiple linear CLWE mappings; Patra et al. (2019) developed a semi-supervised model that
loosened the linearity restriction; Lubin et al. (2019) attempled to reduce the dissimilarity between multi-
lingual embedding manifolds by refining learnt dictionaries; Glavaš & Vulid (2020) first trained a globally
optimal linear mapping, then adjusted vector positions to achieve better accuracy; Mohiuddin et al. (2020) used two independently pre-trained auto-encoders to introduce non-linearity to CLWE mappings: Ganesan et al. (2021) obtained inspirations via the back translation paradigm, hence framing CLWF, training to explicitly solve a non-linear and bijective transformation between multilingual word embeddings. Despite these non-lincar mappings outperforming their lincar counterparts in many setups, in some settings the lincar mappings still seem more successful, e.g., the alignment between Portuguese and English word embeddings in Ganesan et al. (2021). Moreover, training non-linear mappings typically more complex and thus requires more computational resources.
Albeit at the significant recent attention to this problem by the research community, it is still unclear under what condition the lincarity of CLWE mappings holds. This paper makes the first attempt to close this research gap by providing both theoretical and empirical contributions.
Analogy Encoding. Analogy is a fundamental concept within cognitive science (Gntner 1983 that has received significant focus from the NLP community, since the observation that it can be represented using word embeddings and vector arithmetic (Mikolov al. 2013c). . A popular example based on the analogy "king is to man as queen is to woman" shows that the vectors representing the four terms (Økings Fman: Bqueen and zwoman) exhibit the following relation:
Tking - Eman RJ Equeen L wwoman-
Since this discovery, the task of analogy completion has commonly been employed to evaluate the quality of pre-trained word embeddings (Mikolov et al. 2013c Pennington al. 2014 Levy & Goldberg 2014a). This
3

---

Published in Transactions on Machine Learning Research (06/2022)
line of research has directly benefited downstream applications (c.g., representation bias removal (Prade & Richard 202Th) and other relevant domains (e-g,, automatic knowledge graph construction (Wang et al. 2021bb). Theoretical analysis has demonstrated a link between embeddings' analogy encoding and the
Pointwise Mutual Information of the training corpus (Arora et al. 2016 Gittens et al. 2017 Allen & Hospedales 2019 Ethayarajh et al. 2019; Fournier & Dunbar] 2021]. Nonetheless, as far as we are aware,
the connection between the preservation of analogy encoding and the lincarity of CLWE mappings has not been previously investigated.
3 Theoretical Basis
We denote ground-truth CLWE mapping as M X - Y, where X and Y are monolingual word embeddings independently trained for languages Lx and Ly, respectively.
Proposition. Encoded analogies are preserved during the CLWE mapping M + M is affinc.
Remarks. Following Eq. (1), the preservation of analogy encoding under mapping can be formalised
Ca 2g By 3:0 = M(xx)) - M(xxs) M(x,) - M(ro):
(2)
where La Bg Iyy 2o E X. If M is affine, for d-dimensional monolingual embeddings we have
M(x) = -Mx +
(3)
where x E X, M E pdxd, and E gdx1
Proof: Eq.
Eq. To begin with, by adopting the mean centring operation in $2 we shift the coordinates of the space of X, ensuring
M( o.
(4)
This step greatly simplifies the derivations afterwards, because from now on we just need to demonstrate that M a linear mapping, i.e., can be written as Mæ. By definition, this equivalent to showing that M preserves both the operations of addition (a.k.a. additivity) and scalar multiplication (a.k.a. homogeneity).
Additivity can be proved by observing that (x, #z) Iz x Ō and therefore,
(x: + æz) - 22
Eq.
2
M(x; + -x,) - -M(x)) = M(xxs) - M(O
Eq M(a; 2.5) -(aa)+
+ M(rg).
(5)
Homogeneity can be proved in four steps.
Step 1: Observe that
D2 - F; similar to Eq. 5 we can show that
0 34 -
M( - M(0)-((x;)) MM-,,)))))).)))) M(
Eq. M(x+)) M(-x+)..
(6)
Step 2: Using mathematical induction, for arbitrary 2i, we show that
Vm E N M(mx.) mM(xx)
(7)
holds, where N is the set of positive natural numbers, as
Base Case: Trivially holds when m
Inductive Step: Assume the inductive hypothesis that m = k (k E N+), i.e.,
M(karz) h:M(xx))
(8)
4

---

Published in Transactions on Machine Learning Research (06/2022)
Then, required, when = 1: 1, M((k + 1)x,)
M(kat) + M(x;)
Eq.
kM(x)) + +M(x))) (k+ 1)M(a.))
Step 3: We further justify that
M(a;)
Vn E ENT, M(
-
(9)
which, due Eq.
trivially holds when = 1; as for 1,
M Di = M(x + (- xzi)
Eq. 5 M(x;) + +M( Dij)
n1
Eq. M(x,) -M( n nー x4)
Eq.
M(x+) (n 1)Ml =
7
directly yiclds M( M(za) i.c., Eq.
n
Step 4: Considering the set of rational numbers Q (0)) Eqs. 4 (6] and 9 jointly
justifies the homogeneity of M for Q. Because Q C IR is dense set, homogeneity of .M also holds over R, see Kleiber & Pervin (1969).
Finally, combined with the additivity that has been already justified above, linearity of CLWE mapping M is proved, Le., Eq. (2
Eq. 33)
Proof: Eq.
3
Eq.
Justifying this direction quite straightforward:
Fa
Fg
Iy.
Dgg
Mæa Mxg Mx. Mxg
Mæa + b (Mæg + b) Mxy + (Mxs + b)
M(xa) - M(xx)) = M(xx) - M(xx).
Summarising the proofs for both the forward and reverse directions, we conclude that the proposition holds. Please note, the high-level assumption of our derivations is that word embedding spaces can be treated as continuous vector spaces, an assumption commoull adopted in previous work, e.g., Levy & Goldberg (2014b), Hashimoto al. (2016), Zhang et al. (2018), and Ravfogel al. (2020). Nevertheless, we argue that the inherent discreteness of word embeddings should not be ignored. The following sections complement
this theoretical insight via experiments which confirm the claim holds empirically.
4 Experiment
Our experimental protocol assesses the linearity of the mapping between each pair of pre-trained monolingual word embeddings. We also quantify the extent to which this mapping preserves encoded analogies, i.e., satisfies the condition of Eq. (2). We then analyse the correlation between these two indicators. A strong corrolation provides evidence support our theory, and vice versa. The indicators used are described in suitable publicly available corpora for our proposed experiments, we
4.1 Unfortunately, there develop novel word-level analogy test set that is fully parallel across languages, namely XANLG (see 6 4.2
The pre-trained embeddings used for the tests are described in $ 4.3
4.1 Indicators
4.1.1 Linearity of CLWE Mapping
Direct measurement of the lincarity of ground-truth CLWE mapping challenging. One relevant approach is benchmark the similarity between multilingual word embedding, where the mainstream and state-of- the-art indicators the so-called spectral-based algorithms (Søgaard et al. 2018 Dubossarsky et. al.
5

---

Published in Transactions on Machine Learning Research (06/2022)
2020j. However, such methods assume the number of tested vectors to be much larger than the number of dimensions, which does not apply in our scenario (see 4.2) Therefore, choose to evaluate linearity via the goodness-of-iit of the optimal linear CLWE mapping, which is measured as
SIMP == -IIM - Yl1f/r with = arr min || Y||FF
where ||e and r denotes the Frobenius norm and the number of X's rows. To obtain matrices X and Y, from X and Y respectively, we first retrieve the vectors corresponding lexicons of ground-truth Lx-Ly dictionary and concatenate them into two matrices. More specifically, if two vectors (represented rows) share the same index in the two matrices (one for each language), their corresponding words form translation pair, i.e., the rows of these matrices are aligned. "Mean centring" is applied to satisly Eq. (4) For fair comparisons across different mapping pairs, in each of X and Y, rows are standardised by scaling the mean Euclidean norm to 1. Generic Procrustes Analysis (not necessarily orthogonal) (Bookstein) 1992) is applied to find M
Large absolute values of SLMP mcan that the optimal linear mapping an accurate model of the true relationship between the embeddings, and vice versa. SIMP therefore indicates the degree which CLWE mappings are linear.
4.1.2 Preservation of Analogy Encoding
To assess how well analogies are preserved across embeddings, we start by probing how analogies are encoded in the monolingual word embeddings. We use the set-based LRCos, the of analogy mining tool
for static word embeddings (Drozd et al., 2016) 3 It provides a score in the range of to 1, indicating the correctness of analogy completion in single language. For the extension in a cross- coss-lingual setup, we further compute the geometric mean:
SPAE v LRC LRCos(Y).
where LRCos(-) the accuracy of analogy completion provided by LRCos for embedding X. To simplify discussion and analysis from now onward, when performing CLWE mappings, by default select the monolingual embeddings that best encode analogy, i.e., restrict LRCos(X) 2 LRCos(YY. SPAE 1 indicates all analogies are well encoded in both embeddings, and are preserved by the ground-truth mapping between them. On the other hand, lower SPAE values indicate deviation from the condition of Eq. (2)
4.1.3 Validity of SPAE
As an aide, we explore the properties of the SPAE indicator to demonstrate robustness for the interested reader. The score produced by LRCos is relative to pre-specified set of known analogies. In theory, low LRCos(X) score may not reliably indicate that X does not encode analogies well since there may be other word pairings within that set that produce higher scores. This naturally raises a question: does SPAE really promise the validity as the indicator of analogy encoding preservation? In other words, it is necessary to investigate whether there exists an unknown analogy word set encoded by the tested embeddings to an equal or higher degree. If there is, then SPAE may not reflect the preservation of analogy encoding completely, as unmatched analogy test sets may lead to low LRCos scores even for monolingual embeddings that encode analogies well. We demonstrate that the problem can be considered an optimal transportation task and SPAE is guaranteed to be a reliable indicator.
As analysed by EEthayarajh et al. (2019). the degree to which word pairs are encoded as analogies in word embeddings is equivalent to the likelihood that the end points of any two corresponding vector pairs form a high-dimensional coplanar parallelogram. More formally, this task is to identify
P arg min Σ (x)) x
(10)
We have tried alternatives including 3CosAdd Mikolov al. 2013a Mul (Levy et al. 2015), verifying that they are less accurate than CoCo similar trends shown in Tab.
PairDistance Lovv & Goldberg 2014a and 3Cos-
cases. Still, the experiments they exhibit.

---

Published in Transactions on Machine Learning Research (06/2022)
Step
Step 2
Step 3
Step
X
X3
X7
V3
X5
V
5-6
7--8
Xg
X2
X
X4
0
Figure 2: An example of solving ()) in Eq. 11 with P = {(11, 02)) (I2,, 244) (as, vs), (177, .z)}. In the figure we adjust the position of æ1, 33, 26 and 27 in the last step, but it is worth noting that there also exists other feasible (-) given p", e.g., to tune 22 T4, Dy and zs instead.
where P is one possible pairing of vectors in X and C(-) is the cost of given transportation scheme. TH()) denotes the corrosponding cost-optimal process of moving vectors to satisfy
(((aa, 2z); (Ry, CP: (x - (7)(2))) -T)(2)))7/(()))
(11)
T.e., the end points of (m))), TB ( (12)) ( and (x0) form parallelogram.
Therefore, in each language and analogy category of XANLG, we first randomly sample vector pairing samples, leading to 1e5 different P. Next, for each of them, we need to obtain TB() that minimises
Easx (()())) in Eq. [0). Our algorithm is explained using the example in Fig. 2 where the cardinality of X and P is 8 and 4, respectively.
Step 1: Link the end points of the vectors within each word pair, hence our target is to adjust these end points so that all connecting lines not only have equal length but also remain parallel.
Step 2: For each vector pair (æa, .z3) E P, vectorise its connecting line into an offset vector as Va-B
Ta - xz.
Step 3: As the start points of all such offset vectors are aggregated at Ö, seek vector p" that minimises the total transportation cost. between the end point of p* and those of all offset vectors (again, note they share start point at 0).
Step 4: Perform the transportation that all offset vectors become p", i.e.,
- (Ka.)z) P. -T(E)) ( p"
In this way, the tuned vector pairs can always form perfect parallelograms. Obviously, p' is at the cost-optimal position (see Step 3), this vector-adjustment scheme is also cost-optimal.
Solving p" for high dimensions is non-trivial in real world and is a special case of the NP-hard Facil- ity Location Problem (a.k.a. the P-Median Problem) (Kariv & Hakimi 1979). We, therefore, use the scipy. . optimize......... fmin implementation of the Nelder-Mead simplex algorithm (Nelder & Mead] 1965) provide a good-enough solution. To reach convergence, with the mean offset vector as the initial guess, we set both the absolute errors in parameter and function value between iterations at 1e4. We experimented with implementing C(.) using mean Euclidean, Taxicab and Cosine distances respectively. For all analogy categorics in all languages, P coincides perfectly with the pre-defined pairing of XANLG. This analysis provides evidence that the situation where on unknown kind of analogy is beller encoded than the ones used does not occur in practice. SPAE is thus trustworthy.
4.2 Datasets
Calculating the correlation between SLMP and SPAE requires a cross-lingual word analogy dataset. This resource would allow us to simultancously (1) construct two aligned matrices X and Y to chock the lincarity of CLWE mappings, and (2) obtain the monolingual LRCos scores of both X and Y. Three relevant resources were identified, although none of them is suitable for our study.
7

---

Published in Transactions on Machine Learning Research (06/2022)
Category
DE
EN
ES
FR
11
Pl.
Budapest
Budapest
Budapest
Budapest Hongrie हार
Budapeszt
CAP
31
Ungarn
Hungary
Hungría
Wegry
XANLGG
GNDR
30
sohn
hijo
fils
aटт बटी
syn
tochter
daughter
hija
fille
córka
NATL
34
Peru
Peru
Perú
Pérou
4
Peru
Peruanisch
Peruvian
Peruano
Péruvien
Peruwianski
G-PL 3
31
kind
child
niño
enfant
q पाश
dziecko
kinder
children
niños
enfants
dzieci
Category
#
EN
B
Fl
HR
LV
RU
SL
XANLGM
ANIM
32
cagle bird lind
kotkas
kotka
orao
Erglis
орeл
orel
lintu
ptica
putns ITTиiаa
ptica
machine
masin
kone
stroj
mašina
мапинна
stroj
G-PL
machines
masinad
konect
strojevi
mašīnas
мапиины
stroji
Table 1: Summary of and examples from the XANLG corpus. # denotes the number of cross-lingual anal- ogy word pairs each language. tSemantic: animal-species/arr, eapital-worldwee male-fema femalle/cxe, natiionnnationalityii.. 1Syntactic: grammar-plural G-PL.
Brychcin et al. (019)) described cross-lingual analogy dataset consisting of word pairs from six closely related European languages, but it has never been made publicly available. Ulčar et al. (2020) open-sourced the MCIWAD dataset for nine languages, but the analogy words in different languages are not paralle
Garneau et al. (021 produced the cross-lingual WiQueen dataset. Unfortunately, a large part of its entries are proper nouns or multi-word terms instead of single-item words, leading to low coverage on the vocabularics of embeddings.
Consequently, we develop XANLG, which believe to be the first. (publiely available) cross-lingual word analogy corpus. For consistency with previous work, XANLG is bootstrapped using established monolingual analogics and cross-lingual dictionaries. XANLG is constructed by starting with bilingual analogy dataset, say, that for Lx and Ly. Within each analogy category, we first translate word pairs of the Lx analogy corpus into Ty, using an available Tx-L.y dictionary. Next, we check if any translation coincides with its original word pair Ly. If it does, such a word pair (in both Lx and Ly) will be added into the bilingual dataset. This process is repeated for multiple languages to form a cross-lingual corpus.
We the popular MUSE dictionary (Tample et al. 2018a) which contains wide range of language pairs. Two existing collections of analogies are utilised:
Google Analogy Test Set (GATS) (Mikolov et al., 2013c the de facto standard benchmark of embedding-basse analogy solving. We adopt its extended English version, Bigger Analogy Test Set (BATS) Gladkova et al., 2016), supplemented with several datasets in other languages inspired by the
original GATS: French, Hindi and Polish (Grave al. 2018), German (Köper et al. 2015) and Span- ish (Cardelino 2019).
The aforementioned Multilingual Culture-Indopendenn (MCIWAD) (Ulčar et al.] 2020).
Word Analogy
Datasets
Due to the differing characteristies of these datasets (e.g., the composition of analogy categories), they are used to produce two separate corpora: XANLGG and XANLGM- Only categories containing at least 30 word pairs aligned across all languages in the dataset were included. For comparison, 60% of the sernantic analogy categories in the commonly used GATS dataset contains fewer than 30 word pairs. The rationale for selecting this value was that it allows a reasonable number of analogy completion questions to be generated b Information in XANLGG and XANLGM for the capital-counte of Hindi was supplemented with manual
4 Persoual communication with the authors. 300 word pairs can be used generate as many as 3480 unique analogy completion questions such Sking:man :: queen:?" (see Appendix A
8

---

Published in Transactions on Machine Learning Research (06/2022)
translations by native speakers. In addition, each analogy included in the data set checked by at least one fluent speaker of the relevant language to ensure that they are valid.
The XANLG dataset contains five distinct analogy categories, including both syntactic (morphological) and semantic analogies, and twelve languages from a diverse range of families (see Tab. [ From Indo-European languages, one belongs to the Indo-Aryan branch (Hindijm); one the Baltic branch (Tatvianjwx), two the Germanic branch (Englishles, Germanjoz), two the Romance branch (Frenchjes, Spanish es) and four to the Slavonic branch (Croatian|aæ, PolishJer. Russian|Re, Slovene/st). Two non-Indo-European languages, EstonianjEr and FinnishIr, both from the Finnic branch of the Uralic family, are also included. In total, they form 15 and 21 languages pairs for XANLGG and XANLGM, respectively. These pairs span multiple etymological combinations, i.e,, intra-languagg--randd (e.g,, ES-PR), Miter-language-brande (e.g., DE-RU) and inter-language-fane family (e.g., HI-ET).
4.3 Word Embeddings
To cover the language pairs used in XANLG, make use of static word embeddings pre-trained the twelve languages used in the resource. These embeddings consist of three representative open-source series that employ different, training corpora, are based on different embedding algorithms, and have different vector dimensions.
Wik 300-dimensional, trained on Wikipedia using the Skip-Gram version of FastText (refer to Bo- janowski et al. (2017) for details).
Crawl 300-dimensional, trained CommonCrawi plus Wikipedia using FastToxt-CBOW.
100-dimensional, trained the CoNLL corpus (without lemmatisation) using
Vec (Mikolov et al., 2013c).
5
Result
Both Spearman's rank-order (p) and Pearson product-moment (r) correlation coefficients are computed measure the correlation between SLMP and SPAE- Note that, is not possible compute the correlations between all pairs due to (1) the number of dimensions varies across embeddings series, and (2) the source and target embeddings have been pre-processed independently for different mappings. Instead, results are grouped by embedding method and analogy category.
Figures in Tab. 2 show that a significant positive correlation between SPAE and SLMP observed for all setups. In terms of the Spearman's p, among the 18 groups, 5 exhibit very strong correlation (p 0.80) (with a maximum 0.96 for CoNLL embeddings on CAP of xANLGg), 4 show strong correlation (0.80 > p 2 0.70), and the others have moderate correlation (0.70 P 2 0.50) (with a minimum at 0.58: CoNLL embeddings ANIM and G-PL of xANLGa).. Interestingly, although we do not assume linear relationship in $ large values for the Pearson's 's are obtained in practice. To be exact, 4 groups indicate very strong correlation, 6 have strong correlation, while others retain moderate correlation (the minimum r value is 0.58; Wiki embeddings on CAP and G-PL of XANLGg). These results provide empirical evidence that supplements our theoretical analysis ($ 3 of the relationship between linearity of mappings and analogy preservation.
In addition, we explored whether the analogy type (i.e., semantic or syntactic) affects the correlation. To bootstrap the analysis, for both kinds of correlation coefficients, we divide the 18 experiment groups into two splits, i.e., 12 semantic ones and 6 syntactic ones. After that, we compute two-treatment ANOVA (Fisher) 1925). For both Spearman's and Pearson's r, the results are not significant. at p < 0.1. Therefore, we conclude that the connection between CLWE mapping linearity and analogy encoding preservation holds across analogy types. We thus recommend testing SPAE before implementing CLWE alignment as an indicator of whether a linear transformation a good approximation of the ground-truth CLWE mapping.
http:///fsstterrre////////ssppts html
http://llattes..cc/c///////sssss
http://rettor.nnppl.mm/ppppsooop

---

Published in Transactions on Machine Learning Research (06/2022)
XANLGe
P 0.65
P 0.78
0
0.96
0.59
0.60
P 0.87
== 0.58
0.71
0.88
r= 0.65
0.65
7 0.80
Wiki: CAP
Crawl: CAP
CoNLL: CAP
Wiki: GNDR
Crawl: GNDR
CoNLL: GNDR
= 0.80
= 0.70
0.67
0.83
0.84
P 0.78
y= 0.59
0.74
r= 0.76
0.62
7= 0.86
0.81
Wiki: NATL
Crawl: NATL
CoNLL: NATL
Wiki: G-PL
Crawl: G-PL
CoNLL: G-PL
XANLGM
0.59
0.75
0.58
0.65
0.59
P 0.58
0.67
0.73
0.62
0.65
0.74
0.76
Wiki: ANIM
Crawl: ANIM
CoNLL: ANIM
Wiki: G-PL
Crawl: G-PL
CoNLL: G-PL
Table 2: Correlation coefficients (Spearman's p and Pearson's r) between SLMP and SPAE- For all groups,
we conduct significance tests to estimate the p-value. Empirically, the p-value is always less than 1e-2 (in most groups it is even less than 1e-3), indicating a very high confidence level for the experiment results. To facilitate future research and analyses, we present the raw SIMP and LRCos data in Appendix B
Although there are strong correlations between the measures, they are not perfect. We therefore carriod out further investigation into the data points in Tab. that do not follow the overall trend. Firstly, we identified that some associated with "crowded" embedding regions, in which the correct answer to analogy question is not ranked highest by LRCos but the top candidate is polysemous term (Rogers et al. 2017). One example is the LRCos score of the CAP analogy for PL's Wiki embeddings, which was underestimated. If we consider the three highest ranked terms, rather than only the top term, then the overall p and r of "Wiki: CAP" (the first cell in Tab. 2 will increase sharply to 0.79 and 0.76, respectively.
Secondly, we noticed the in certain cases the source and target vectors of word pair are too close (i.e. the distance between them near zero). This phenomenon introduces noise to the results of analogy metrics such LRCos [Linzen 2016 Bolukbasi et al. 2016). and consequently, impact SPAE- For example, the mean cosine distance bel ween G-PL pairs is smaller in XANLGM (0.18) than XANLGG (0.24). Therefore, the SpAE for G-PL is less reliable for XANLGM than XANLGG, leading to a lower correlation.
6 Application: Predicting Relationship between Monolingual Word Embeddings
As discussed $2 many scenarios lincar CLWE mappings outperform their nonlinear counterparts, while in other setups nonlinear CLWE mappings are more successful. Therefore, an indicator that predicts the relationship between independently pre-trained monolingual word embedding which helps decide whether to
10

---

Published in Transactions on Machine Learning Research (06/2022)
CLWE method"
CCA
PROC
PROC-B
DLV
ROSLS
SPAE
Seed dict. size
1K
3K
5K
1K
3K
5K
1K
3K
1K
3K
5K
1K
3K
5K
EN-FI
.26
.35
.38
.77
.37
.40
.36
_38
.27
.37
.40
.31
.40
.44
.41
EN-IIR
.22
.30
.33
.2
.31
.34
.30
.34
.23
.31
.33
.27
.36
.38
.32
EN-RU
.34
.43
.45
.35
.45
.46
.42
.45
.35
.44
.47
.40
.49
.51
.46
-HR
.17
.66
.29
.19
.27
.29
.66
.29
.18
.27
.29
.21
.30
.32
.23
FI -RU
.21
.31
.34
23
.31
.34
.32
33
.23
.31
.34
.26
.34
.38
.33
HR-RU
.26
.35
.37
.27
.35
.37
.35
.37
26
.35
.37
.29
.38
.40
.26
Spearman's
.83
.82
.86
.83
.84
.88
.83
866
.84
.84
.87
.87
.88
.90
Table 3: Spearman's p between the Word Translation performance (MRR) of theea--mapping-bant based CLWE methods (Trom Glavaš al. (2019); PROC-B's performance with 5K seed dictionary was not available) and the average analogy encoding preservation score (SPAE).
use linear non-linear mappings without training actual CLWEs, would be beneficial. Use of this indicator has the potential to reduce the resources required to find optimal CLWEs (e.g., some recent approaches need several hours of processing on modern GPUs (Peng et al. 2021a Ormazabal et al. 2021 ), with corresponding roductions in carbon footprint.
The proposed SPAE metric, which can obtained within several minutes on single CPU, can be leveraged as such a metric. A high SPAE score suggests that the linear assumption holds strongly on the ground truth CLWE mapping, so is feasible to train linear CLWE mapping; otherwise, the non-linear approaches are rocommended.
To demonstrate this idea in practice, we revisited a systematic evaluation on CLWE models based on linear mappings Glavaä et al. 2019), which reported Mean Reciprocal Rank (MRR) of five representative linear- sapping-based CLWE approaches on the Word Translation task (the de facto stadard for CLWEs). We focus on six language pairs (EN-FI, EN-HR, EN-RU, FI-HR, FI-RU, HR-RU) as they are covered by both XANLGM and the dataset of Glavaš al. (2019). Additionally, only Wiki embeddings were involved the experiments of Glavaš et al. (2019). Thus, for cach language pair, we aggregated SPAE of different analogy categories for Wiki embeddings, then caleulated the average, SPAE-
Results are shown in Tab. where the Spearman's p between SPAE and Word Translation performance is highlighted. Strong positive correlations are observed in all setups that were tested. These results demon- strate that SPAE provides as accurate indication of the real-world performance linear CLWE mappings, regardless of the language pair, mapping algorithm, or level of supervision (i.e., size of the seed dictionary for training). These results also provide solid support to the main statement of our paper, i.., the ground-truth CLWE mapping between monolingual word embeddings is linear iff analogies encoded in those embeddings are preserved.
7 Further Discussion
Prior work relevant to the linearity of CLWE mappings has largely been observational (see $ sheds new light on these past studies from the novel perspective of word analogies.
2 This section
Explaining Non-Linearity. We provide three suggested reasons why CLWE mappings are sometimes not approximately linear, all linked with the condition of Eq. 2 not being met.
The first may be issues with individual monolingual embeddings (see one such example in the upper part of Fig. 3h. In particular, popular word embedding algorithms lack the capacity to ensure semantic continuity over the entire embedding space (Linzon] 2016). Henco, voctors for the analogy words may only exhibit local consistency, with Eq. (2) breaking down for relatively distant regions. This caused the locality of linearity that has been reported by Nakashole & Flauger (2018). Li et. al. (2021) and Wang et al. (02Ta)
11

---

Published in Transactions on Machine Learning Research (06/2022)
X
blueberry
blue
A
raspberryn
Ogreen
+
red
bull (market)
bear 1V
#
(market)
X
#
Figure 3: Illustration of example scenarios where the CLWE mapping is non-linear. Translations of English (left) and Chinese (right) terms are indicated by shared symbols. Upper: The vector for "blueberry" (shadowed) is ill-positioned in the embedding space, so the condition of Eq. (2) is no longer satisfied. Lower: In the financial domain some Eastern countries (e.g., China and Japan) traditionally use "black" to indicate growth and "green" for reduction, while Western countries (e.g., US and UK) assign the opposite meanings to these terms, also not satisfying the condition of Eq.
The second reason why a CLWE mapping may not be linear is semantic gaps. Despite analogies in our XANLG corpus all are langunge-ngnoslic, the analogical relations between words may change or even dis- appear sometimes. For example, languages pairs may have very different grammars, c.g., Chinese does have the plural morphology (Li & Thompson 1989). so some types of analogy, e.g. G-PL used above, do not hold. Also, analogies may evolve differently across cultures, (see example in the lower part of Fig, 3). These two factors go some way to explain why typologically and etymologically distant language pairs tend to have worse alignment (Ruder et al. 2019).
Thirdly, many studies point out that differences in the domain of training data can influence the similarity between multilingual word emboddings (Søgaard et al. 2018 Artetxe et al. 20188.. Besides, we argue that due to polysemy, analogies may change from one domain to another. Under such circumstances, Eq. 2 is violated and the linear assumption longer holds.
Mitigating Non-Lincarity. The proposed analogy-inspire framework justifies the success and failure of the linearity assumption for CLWEs. As discussed earlier, it also suggests method for indirectly assessing the linearity of a CLWE mapping prior to implementation. Moreover, it offers principled methods for designing more effective CLWE methods. The most straightforward idea is to explicitly use Eqq (2) as a training constraint, which has very recently been practisod by Garneau et al. (2021) 9 Based on analogy pairs retrieved from external knowledge bases for different. languages, their approach directly learnt to better encode monolingual analogies, particularly those whose vectors are distant in the embedding space. It not only works well on static word embeddings, but also leads to performance gain for largo-scale pre- trained pross-lingual language models including the multilingual BERT (Devlin et al. 2019 These results multiple tasks (e.g., bilingual lexicon induction and cross-lingual sentence retrieval) can be seen as an independent confirmation of this paper's main claim and demonstration of its usefulness.
Our study also suggests another unexplored direction: incorporating analogy- information into non- linear CLWE mappings. Existing work has already introduced non-lincarity to CLWE mappings by applying variety of techniques including directly training non-linear functions (Mohiuddin et al. 2020), tuning linear mappings for outstanding non-isomorphic instances (Glavāš & Vulić 2020) and learning multiple linear CLWE mappings instead of single one (Nakashole) 2018 Wang al. 2021a (see $ 2). However, there is lack of thcoretical motivation for decisions about how the non-lincar mapping should be modelled. Nevertheless, the results presented here suggest that ensembles of linear transformations, covering analogy preserving regions of the embedding space, would make a reasonable approximation of the ground ground-truth CLWE mappings and that information about analogy preservation could be used to partition embedding
9They cited earlier preprint the primary motivation for their approach.
12

---

Published in Transactions on Machine Learning Research (06/2022)
spaces into multiple regions, between which independent linear mappings can be learnt. We leave this application as our important future work.
8 Conclusion and Future Work
This paper makes the first attempt to explore the conditions under which CLWE mappings are linear. The- oretically, we show that this widely-adopted assumption holds iff the analogies encoded are preserved across embeddings for different languages. We describe the construction of a novel cross-lingual word analogy dataset for diverse range of languages and analogy categories and propose indicators to quantify lin- earity and analogy preservation. Experiment results on three distinct embedding series firmly support hypothesis. We also demonstrate how our insight into the connection between linearity and analogy preser- vation can be used to better understand past observations about the limitations of linear CLWE mappings, particularly when they are ineffective. Our findings regarding the preservation of analogy encoding provide a test that can be applied to determine the likely success of any attempt to create linear mappings between multilingual embeddings. We hope this study can guide future studies in the CLWE field.
Additionally, we plan to expand theoretical insight to contextual embeddings, inspired by Garneau et al. (2021) who demonstrated that developing mappings that preserve encoded analogies benefits pre-trained cross-lingual language models as well. We also aim to enrich XANLG by including new languages and analogies to enable explorations at an even larger scale. Finally, we will further design CLWE approaches that learn multiple lincar mappings between local embedding regions outlined with analogy- based metrics (see $
Broader Impact Statement
CLWE bridges the gap between languages and is efficient enough to be applied in siluations where limited resources available, including to endangered languages (Zhang al. 2020 Ngoc Le & Sadat 2020). This paper presented theoretical analysis of the mechanisms underlying CLWE techniques which has potential to improve these methods. Moreover, the proposed SPAE metric predicts whether monolingual word embeddings in different languages should be aligned using a linear or non-linear mapping, without actually training the CLWEs. This indicator lowers the computational expense required to identify a suitable mapping approach, thereby reducing the computational power needed and negative environmental effects.
Our analysis relies on the use of analogies and previous work has indicated that these may contain biases, e.g., regarding gender (Bolukbasi et al. 2016 Sun et al. 2019). Any future work that incorporates analogies within the CLWE process should be aware of the potential consequences of any biases that may be contained within the analogies used. On the other hand, there is potential for the findings of this work to be leveraged for bias alleviation in cross-lingual representation learning.
Acknowledgements
We would like to express our sincerest gratitude all voluntoers from Beijing Foreign Studies University who manually annotated and validated the XANLG corpus, as well Guowei Zhang, Guanyi Chen, Ruizhe Li, Alison Sneyd, and Harish Tayyar Madabushi who helped this study, We also thank the official TMLR reviewers for their insightful comments and Angeliki Lazaridou for the action editing.
References
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embeddings. In Pro- ceedings of the 36th International Conference on Machine Learning, pp. 223-231, Long Beach, California, USA, 2019. PMLR. URL http: htttp:://proceedings. mmr. pttpps/////allllnee/p/ http://eeeaaaaaa....rrrr/////aaa html
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Ristoski. latent variable model approach PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399. 2016. doi: 10.1162/tar 00106. URL http://wwwwweeee..o/////ooooo///
13

---

Published in Transactions on Machine Learning Research (06/2022)
Mikel Artetxe, Gorka Labaka, and Encko Agiree. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirioal Methods in
Natural Language Processing, pp. 2289 2294, Austin, Texas, November 2016. Association for Computa- tional Linguistics. doi: 10.18658/v1 1250. URL https://www hwtp::///wwwwwwwwb.b./.b aclweb. http://www...r//aaoooooiieeeeeee http:////ooooog//11----0000
Mikel Artetxe, Gorka Labaka, and Encko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeling of the Associalion for Compulalional Linguislics (Volume 1: Long Papers), PP. 451-462 Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: N0.18653/v1/PMM 1v/11711142 URL hattps://www.att
org/ anthologyyolyy/o /aciooooy/PP1//// P17-1042
Mikel Artetxe, Gorka Labaka, and Encko Agirre. A robust self-lcarning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 789-798, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653 11/805/v1//118/ 1073. URL htttps://wwe aclweb. org/ anthology/P18-1077
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. doi: 10.1162/
tacl_ a 00051. URL.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th Inter- national Conference on Neural Information Processing Systems, NIPS'16, pp. 4356-4364, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Fred L. Bookstein. Morphometric Tools for Landmark Data: Geometry and Biology. Cambridge University Press, 1992. doi:
Tomáš Brycheín, Stephen Taylor, and Lukáš Svoboda. Cross-lingual word analogies using linear transfor- mations between semantic spaces. Expert Systems with Applications, 135:287 295, 2019. TSSN 0957-
4174. doi: http::///bbioor///..00000eeeeeee articlel/p1//05557774414494999
URL (tRL/http://www htti:///www..cieenddrrree..tt com/science/
Cristian Cardellino. Spanish Billion Words Corpus and Embeddings, August 2019. URL https://
crscardellino. Sigaareellioo.ggghhb.../iie/CC
Alexis Comneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual reprosentation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 8440 8451, Online, July 2020. Association for Computational Linguistics. doi: URL h wrgeeoorgg/200000 wwwemminnn....m.e 747
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. Tn Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.11663//11/199 1423. URL hettss://aclaatthoogy.ogg/////
Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. Word embeddings, analogies, and machine learn- ing: Beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 3530, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://www aclweb. www.ww...rg/aae........... otg/aantaoloog//66-33322
Haim Dubossarsky, Ivan Vulié, Roi Reichart, and Anna Korhonen. The secret is in the spectra: Pre- dicting cross-lingual task performance with spectral similarity measures. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2377-2300, Online, November 2020. Association for Computational Linguistics. doi: 20.1863//112020..oo emnn:-mmannlaiii mano-main.118 186. URL https://acllatthoooyy..rr////// emnlp-maint. main. 186
14

---

Published in Transactions on Machine Learning Research (06/2022)
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analogies. In Proceedings of the 57th Arouual Meeting of the Association for Computational Linguistics, pp. Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/11/P19-1BBB URL https://wre acwww///bww....bbb......b...... wttpg//aantooooog11111111 1315
Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational
Linguistics, pp. 462 471, Gothenburg, Sweden, April 2014. Association for Computational Linguistics. doi: 10.1155/11EE44-099 URL https://www. http://wwwwwwww....///.. www.:a.wwwww.wa................a http//annooooooy---1111
R.A. Fisher. Statistical methods for research workers. Edinburgh Oliver & Boyd, 1925, URL http:// psychclassics yorku, ca/Fisher/Methodas
Louis Fournier and Ewan Dunbar. Paraphrases do not explain word analogies. In Proceedings of the 16th Conference of the European Chapter of the Association for Compulational Linguisties: Main Volume, pp. 2129 2134, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. Bacl-main.182 URL
org/2021 eacl-main 182
Ashwinkumar Ganesan, Francis Ferraro, and Tim Oates. Learning a reversible embedding mapping using bi-directional manifold alignment. In Findings of the Association for Computational Lingwistics: ACL- IJCNLP 2021, pp. 3132 3139, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/vv 101553/v1//// wttp/22...inni/i/o acl.ing URL
findings-acl. .766
Nicolas Garneau, Mareike Hartmann, Anders Sandholm, Sebastian Ruder, Ivan Vulié, and Anders Søgaard. Analogy training multilingual encoders. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (14):12884 (14):12884-12992, May 2021. URL https:/ org/ httg/index..ooo
Dedre Gentner. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155- 170, 1983. ISSN 0364-0213. doi:
URL https://www
Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney, Skip-Gram- Zipf + Uniform = Vector Additiv- ity. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 69 76, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 00.18653/v1/P17-111
URL https:://ww. aclweb. www.awwwwwwww.weeeeeeeeoeeeeeeee org/aantaologyy/77-007
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based detection of morphological and scmantic relations with word emboddings: what works and what docsn't. In Proceedings of the NAACL Student Research Workshop, pp. 8 15, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N1 2//16-202.. URL https://aclatthologgy.. wttp:///ooooo.ooo./1//0000000
Goran Glavaš and Ivan Vulié. Non-linear instance- based cross-lingual mapping for non-isomorphic embedding spaces. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7548-7555, Online, July 2020. Association ion for Computational Linguistics. doi: 10. 10.18633/v1 /20p:///1//222.cammmil
675. URL httph: http::://acllathhoogggg org/ 20tgg/200..oo acl mael-main.60 .65
Goran Glavaš Robert Litschko, Sebastian Ruder, and Ivan Vulié. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 710-721, Florence,
Italy, July 2019. Association for Computational Linguistics. doi: //aclanthology. wwtpge/orr//11110001
URL https:
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://ww:.wtt.. htttps:/wwww.aclwebbo aclweb. www.://wwwwwwwww.ooooooo.oooooo. htt..///aantttooog//--------
15

---

Published in Transactions on Machine Learning Research (06/2022)
Tatsunori B. Hashimoto, David Alvarez-Melis, and Tommi S. Jaakkola. Word embeddings metric recovery in semantic spaces. Transactions of the Association for Compulational Linguistics, 4:273 22-6,,66- 2016. doi: (0.1162/tacl a 00098.. URL
Christian Herold, Jan Rosendahl, Joris Vanvinckenroye, and Hermann Ney. Data filtering using cross- lingual word embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulalional Lánguisties: Human Language Technologies, pp. 162 172, Online, June 2021. Association for Computational Linguistics. doi: 11868//v1/0021..naacmmiii //aclanthology. org/2021 http/222...aac11eee
URL https:
Kariv and S. L. Hakimi. An algorithmic approach to network location problems. II: The P-Medians. SIAM Journal on Applied Mathematics, 37(3):539- 560, 1979. doi: 10.1137/0137041. URL hhtpps://oi, org/10. 11111011///113000
Yunsu Kim, Miguel Graça, and Hermann Ney. When and why is unsupervised neural machine translation useless? In Proceedings of the 22nd Annual Conference of the European Association for Machine Transla- tion, pp. 35-44, Lisboa, Portugal, November 2020. European Association for Machine Translation. URL https: //www http://wwwwwawwb..cooo attg///toooogy/0000000 eamt-1.5
Martin Kleiber and W. J. Pervin. A generalized Banach-Mazur theorem. Bulletin of the Australian Mothe- matical Society, 1 1(2):169 173, 1969, doi: 10.007750004977704111111
Maximilian Köper, Christian Scheible, and Sabine Schulte im Walde. Multilingual reliability and "semantic" structure of continuous word spaces. In Proceedings of the 11th International Conference on Computational Semantics, pp. 40-45, London, UK, April 2015. Association for Computational Linguistics. URL https: 7/www www.waclwb..co org/ anthology Saarthhooogy/115-11001 W15-0105
Guillaume Lample and Alexis Comneau. Ciosullngual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Guillaume Lample, Alexis Conncau, Ludovic Denoyer, and translation using monolingual corpora only. In International Conference on Learning Represenlations, 2018a. URL https::/ wwtp://toooedddddtii..t..
Ranzato. Unsupervised machine
Guillaume Lample, Alexis Comneau, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018b. URL httpp::/openrrvvier net/forumnid--HH16aaan
Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Fighteenth Conference on Compulational Natural Language Learning, pp. 171-180, Ann Arbor, Michigan, June 2014a. Association for Computational Linguistics. doi: 10.3115/v1/W14-100 1618. URL httph:t/////tt h/opp:://wwww wwwp://wwwwwwwww...///.// aclweb. wwwp//gg////toooooo
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Z. Ghahra- mani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014b. URL httpps://proceedinng. mew.eeeidddnnnnn.nnp..e.. .df
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguislies, 3:211 225, 2015. doi:
10.1162/tacl a 00134. URL htttps://aclantthologg. httph/ooooooo.ooo11o-e/-e- org/Q15- 1016,
Charles N Li and Sandra A Thompson. Mandarin Chinese: A functional reference grammar, volume 3. Univ of California Press, 1989.
Yuling Li, Kui Yu, and Yuhong Zhang. Learning cross-lingual mappings in imperfectly isomorphic embedding spaces. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2630-266 2242, 2021. doi: 10.1109/TASS..P2222...00 209003093353....
16

---

Published in Transactions on Machine Learning Research (06/2022)
Tal Linzen. Issues in evaluating semantic spaces using word analogies. In Proceedings of the 1st Workshop Evaluating Veclor-Space Represenlations for NLP, pp- 13-18, Berlin, Germany, August 2016. Association for Computational Linguistics, doi: 10.18653/v1/W2/ 2503. URL/htts:://wwe www.eb.llww..orr.. wttp://b...gggaaooooo.....//.// 016-
Noa. Yehezkel Lubin, Jacob Goldberger, and Yoav Goldberg. Aligning veclor-spaces with noisy supervised lexicon. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 460 465, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/ N19-1045. URI. httts://www. btttpp::/www.allwee.orrgg.. wwwwwb.wwwbb.c.... wtt..//www..gggaaoooooo//o////ee
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In 1st International Conference Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a. URL. https: httpss//opeerevieer net/
formm"'d=1ddCaaWkggRGC
Tomas Mikolov, Quoc V. Le, and llya Sutskever. Exploiting similarities among languages for machine translation. CoRR, 1////339944166 2013b. URL http:://rxiiv.c .rgg httpaabs/110.....1
Tomas Mikolov, Tlya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Newral Information Processing Systems Volume 2, NIPS'13 13, pp. 3111-3119, USA, 2013c. Curran Associates Inc. URL http: acm. www.aaa............../.. orgacitaiion cfm?id=2999792. 2999959
Tasnim Mohiuddin, M Saiful Bari, and Shafiq Joty. LNMap: Departures from isomorphic assumption in bilingual lexicon induction through non-linear mapping latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2712-2723, Online, November 2020. Association for Computational Linguistics. doi: N0.18653/v1120222 11066///110222..mmmmmmmmmm2222 URL
https://aallantholooyy wwwp://oooooyyog//222//2///// org/2020.002/ canlp-main 215
Ndapa Nakashole. NORMA:
Neighborhood sensitive maps for multilingual word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 512 522, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/vl (0.18653/v1/D18-11 /1/1111-1/
URL https://www wwtp::wwwwwww.....//..//// http://c..ebboo/aaaaaaooooo///o.
Ndapa Nakashole and Raphael Flanger. Characterizing departures from linearity in word translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 221 227, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 20118633v11PPP 2-366. URL htttps://www.aalwwe..rrrr.. wwwp://wwww....rg//ooooo///////e
J. A. Nelder and R. Mead. A Simplex Method for Function Minimization. The Computer Journal, 7(4):308 313, 1965. ISSN 0010-4620. doi: (0.1098/commnn//7...38 URL https://dol. httt:s://dol.or/11..000 1093///ooo.0///mmal 7.4.308
Tan Ngoc Le and Fatiha Sadat. Revitalization of indigenous languages through pre- pre-processing and neu- ral machine translation: The case of Inuktitut. In Proceedings of the 28th International Confer- ence Computational Linguistics, pp. 4661 4666, Barcelona, Spain (Online), December 2020. In- ternational Committee on Computational Linguistics. doi: /1116/3//1/2222.coolng... otw.:aaa....o..roo//oo..o.....aa 410
URL
Aitor Ormazabal, Mikel Artetxe, Aitor Soroa, Gorka Labaka, and Encko Agirre. Beyond offline mapping: Learning cross-lingual word embeddings through context anchoring. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6479-6489, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 acl- lon..o0n..06. URL https://aclaathhooogg. org/
2021 acl--ong..
17

---

Published in Transactions on Machine Learning Research (06/2022)
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 184 193, Florence, Italy, July 2019. Association for Computational Linguistics. doi: (118653/v1/P11 1018. URL https: //www//wwwwwoo/ aclweb. org/
arthology/P19-1018
Xutan Peng, Chenghua Lin, and Mark Stevenson. Cross-lingual word embedding refinement by & norm optimisation. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Compulational Linguisties: Human Language Technologies, pp. 2690 2701, Online, June 2021a. Association for Computational Linguistics. doi: 10. 20/553/v12021.maam/ mailmmminn22....... 214. URL https: //aclanthology.orgg.
naacl-main 214
Xutan Peng, Yi Zheng, Chenghua Lin, and Advaith Siddharthan. Summarising historical text in modern languages. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- Online, April 2021b. Association for Computational
putational Linguisties: Main Volume, pp.
Linguistics. URL https: sstps://www..r wwwwww. www.:aawwwww....abbbeeeeeeeeeeee htt.://orrr/ooooooo///202220211 htt.:220.1..c1all11111111ae1lll. waw.:ea...lmmmmmmmmemmm 273
https://www.aclweb.oo
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word represen- tation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-111 1162.
URL
Henri Prade and Gilles Richard. Analogical proportions: Why they are useful in ai. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4568 4576. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ 2021/61. URL https://ddae org/ 10. 24963/.ji 2021/621 Survey Track.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding pro- tected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Associ- alion for Computational Linguistics, pp. 7237 7256, Online, July 2020. Association for Computational Lin- URL https:
guistics. doi: (0.18653/v1/220 w2t////v2222..e//e/ acl.00...lmaain.......mm
org/ 202/2020.oo20.0 acl -main. 647
Anna Rogers, Aleksandr Drozd, and Bofang Li. The (too many) problems of analogical reasoning with word vectors. In Proceedings of the 6th Joint Conference on Lerical and Computational Semantics (*SEM 2017), pp. 135 148, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/ 111////111111111111111 URL httpss//acclanthooogg org/S17-1011
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. A survey of gross-lingual word embedding models. Journal of Artificial Intelligence Research, 65(1):569- ()36666633 May 2019. ISSN 1076-9757. doi: 10. 10.1613/jair.111 1...///ar...1164... 116400.. URL https:://oi.orge
1613/jair..... 11640
Anders Sogaard, Sebastian Ruder, and Ivan Vulić. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 778 788, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1 10053/v11P18-107 URL htttp:://// aclweb. argg/anthology/P8--BB0
Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. Cross-cultural similarity features for cross-lingual transfer learning of pragmatically motivated lasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 9403-2414, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 202l.eacl-mann.00 204. URL htttp:://aclaabbboooo..o00//2112 eacl-main. 204.
http:///aaaaaaatog.a.///. org/
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai- Wei Chang, and William Yang Wang. Mitigating gender bias in natural language process- ing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1640-1640, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P19-111 1159. URL http:////cccaaatoooooooorr//////
18

---

Published in Transactions on Machine Learning Research (06/2022)
Matej Ulčar, Kristiina Vaik, Jessica Lindström, Milda Dailidenaité, and Marko Robnik-Sikonja. Multilingual culture-independent word analogy datasets. Tn Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4074 4080, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444 URL https: /acpanthoocl
100.0200..ee1.0000 1.501
Ivan Vulié, Scbastian Ruder, and Anders Sagaard. Are all good word vector spaces isomorphic? In Pro- ceedingg of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. Online, November 2020. Association for Computational Linguistics. doi: 10.18633/ 11.18633/v1/2020. wwtooogg///0/// omnlp-main.257 257. URL httpps://aclanthollgg http:///c.aaaooooooooo////////00 ewnle-mnnnlieeeeeee main.
Haozhou Wang, James Henderson, and Paola Merlo. Multi-adverrarial learning for cross-lingual word em- beddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 463-472, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1 /2021 mwacl-main..n3 URL htttts:://acaanthollggg .orgg
2021 maa.lllmii.......
Meihong Wang, Linling Qiu, and Xiaoli Wang. A survey on knowledge graph embeddings for link prediction. Symmetry, 13(3), 2021b. ISSN 2073-8994. doi:
/sym130304885 URL https://www.nwn com/
2073-8994/13/3/48
Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime G. Carbonell. Cross-lingual alignment joint training: A comparative study and A simple unifidd framework. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe-
view.net, 2020. URL httpp:://ppereeeeerr.. ott.://ewwe...et.eoooooooeeeeeo-
Svante Wold, Kim Esbensen, and Paul Geladi. Principal Component Analysis. Chemometries and Intelli- gent Laboratory Systems, 2(1):37 52, 1987. ISSN 0169-7439. doi: https://tbb.orr/11.010///0111110 74311 (439(87) (87) 80084-9. URL http://www wectt www.science//ccoocccccccccccccce prccaa/c//aatttaaa//////////7//0 ings of the Multivariate Statistical Workshop for Geologists and Geochemists.
Proceed-
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Compulational Linguislics: Human Language Technologies, pp. 1006-1011, Denver, Colorado, May- June 2015. Association for Computational Linguisties. doi: 10.31155v11/115-1004. URL
https:// hww. www.ww.cwbbb.oo org/ www.aaogghoglns-..- 1104
Mozhi Zhang, Yoshinari Fujimuma, and Jordan Boyd-Graber Exploiting cross- cross-lingual subword similarities in low-resource document classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
volume 34, pp.
Yi Zhang, Jie Lu, Feng Liu, Qian Liu, Alan Porter, Hongshu Chen, and Guangquan Zhang. Does deep learning help topic extraction? kernel k-means clustering method with word embedding. Journal of Informetrics, 12(4):1099 1117, 2018. ISSN 1751-1577. doi: URL https:/ //w/:///wwwww/.
Yucheng Zhou, Xiubo Geng, Tao Shen, Wenqiang Zhang, and Daxin Jiang. Improving zero-shot cross- lingual transfer for multilingual question answering over knowledge graph. Tn Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5822-5834, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1, 22255// /01p//211.aaaamm 1ww2000...aal---- canacl-main.4ee URL
nww.l saaacclmmiin44 4665.
19

---

Published in Transactions on Machine Learning Research (06/2022)
A Question Formulations
For an analogy category with t word pairs, (2) four-item elements can be composed. An arbitrary clement,
or : 7:0, can yield eight analogy completion questions as follows:
a:B :: 7:? 3:a :: 0:? no : 0:? 0:3 : y:?
o:7y :: 3:? 8:0 :: oc? 7:0 o:? 0:y :: B:?
Hence, ())
unique questions can be generated.
B
Raw Data for Tab. 2
XANLGe, EN-DE EN-ES
EN-PL
DE-ES
DE-PL
ES-FR
ES-HI
ES-PL
FR-HI
PR-PL
CAP
.16
.21
.17
.36
.33
.21
.18
.36
.22
.22
.35
.25
.35
.23
.33
Wiki
GNDR
.32
.12
.39
.26
.35
.48
.10
.36
.39
.13
.38
.30
.42
NATL
.18
.16
.15
.14
.0
.19
.19
.33
.21
.16
.30
.21
.14
.20
.32
G-PL
.22
.26
.25
.25
.38
.27
.38
CAP
.33
.3
-20
.23
.99
.26
.23
244
.28
.23
.26
.28
.24
.29
.38
CONLL Crawl
GNDR
.57
.58
.65
.66
.57
.59
.64
.56
.57
.57
.58
NATL
.32
.33
.27
.39
.99
322
355
.47
.35
.40
.43
.31
.46
.31
.42
G-PL
.35
.24
.33
.48
.99
.33
.37
.44
42
.33
.47
.33
.48
442
.51
CAP
T31
.58
.32
.55
.39
.58
32
.5
.38
.59
.66
.59
.56
_40
.55
GNDR
.48
.76
.49
.55
.48
.74
.55
.57
.50
.77
.76
.72
.59
.52
.58
NATL
.77
.22
.66
.51
.88
78
.44
.22
.36
.74
.74
.73
.50
.35
.50
G-PL
322
.67
:32
.48
.36
.65
:34
.77
.36
.68
.67
.65
50
.38
.49
EN
EN
RT
TV
RU
XANLGM
ET
F1
HR
LV
RU
SL.
F1
HR
IV
RU
SL
11
LV
RU
SL.
1.
RU
SL.
RU
SL
SL.
ANIM
.50
50
22
.31
.15
.7
.3
.9
.30
440
.28
.22
.20
Wiki
G-PL
25
22
.37
.37
.28
.33
24
.1
.29
28
.66
.0
.29
26
.27
.33
.32
.30
.33
.88
.28
ANIM
.55
.49
.55
511
34
.45
.2
1.0
.45
.37
.23
.28
.38
.24
.43
Crawl
G-PI
228
.43
.47
.43
.45
44
33
45
.7
.43
.37
466
44
.44
.43
.42
.50
.54
.39
.35
.43
ANIM
.54
99
.55
50
53
29
74
46
.3
43
87
31
38
46
64
.77
.9
42
.36
CoNLL
G-PL
.45
40
52
42
40
42
37
.77
41
41
40
81
37
36
39
84
.66
.77
36
.40
1.38
Table 4: Raw SLMP results (the negative sign is omitted for brevity).
Wiki
Crawl
CoNLL
Wiki
Crawl
CoNLL GNDR NATL G-PL
ANIM
G-PL ANIM
G-PL
ANIM
G-PL
CAP
GNDR
NÄTE G-PL
CAP
GNDR MATL G-PLi CAP
.48
.65
.29
.36
.58
.68
.25
.21
.23
.47
.48
.79
.77
.65
.43
.41
.55
ET
.12
.50
.52
1.00
.21
.48
EN
.94
.33
.94
.58
.57
.67
.76
.94
.87
.57
.79
.61
.65
.42
.54
.45
.13
.35
.13
.40
.57
.68
.87
.13
.07
.07
.17
HR.
.17
.20
.50
.68
.07
.11
FR
.92
.27
.76
.13
.65
.50
.85
.87
.48
.14
.24
.35
TV
.19
.10
.39
.27
.23
.29
:30
.42
.07
.58
.59
.59
.32
.32
.37
.31
.16
RU
.36
.40
.61
.87
.42
.55
PT.
.16
.21
.66
.10
.29
.55
.82
84
.45
.45
.38
.52
SL
.42
.23
.39
.81
.12
.39
Table 5: Raw monolingual LRCos results (lett:XANGGo: right: XANLGM).
20


Reviews:
[{"review_id": "92aRLoQxno", "reviewer": "Reviewer_jtuU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper addresses a gap in our understanding of why in some cases it has been shown that a linear mapping for cross lingual word embeddings works well, while in some cases it doesn't. It suggests an indicator, based on how well analogies are preserved in both spaces, that can predict whether a there is a linear mapping between two embedding spaces of two languages. \n- The paper introduces a new dataset for measuring analogy preserving ability of embeddings in cross-lingual setting.\n\n\nWeaknesses:\n- The design of the experiments is a bit narrow and they don't fully support the claims. There are no results reported in the paper that shows the quality of the word embeddings and the linear mappings. E.g., It would be nice to also see the performance of the actual and mapped embedding on some downstream tasks. I think we need to have some results on metrics that indicate the success or failure of the linear mapping models.\n- The conclusions of the paper are based on the correlation between the two quantities $S_{LMP}$ and $S_{PAE}$. Both these can be viewed as notions of similarity between two embedding spaces. Hence, their correlation is not surprising. Taking this into account, and without further experiments to complement these results, I am not sure about the conclusions and their significance. \n- It seems to me that the cross lingual analogy dataset is somewhat small, which can limit the strength of the conclusions that can be made based on the results obtained on this dataset.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is motivated by the gap in understanding the conditions under which the relation ship between word embedding models of different language can be captured by a linear mapping. The authors argue that the existence of the linear relationship is explained by the similarity of the analogy structures between the language pair (or actually word embedding of the languages). They provide theoretical analysis and empirical results to support this argument.\n\nIn the empirical section, to show this, given a pair of word embedding models (for two languages) the authors measure the correlation between two quantities: \n\n(1) How well a linear transformation fits the data ($S_{LMP}$).  \n\n(2) The extent to which analogies are preserved in both word embedding spaces, which is measured using, `LRCos`, an analogy mining tool for static word embeddings (Drozd et al., 2016) that provides a score in the range of 0 to 1, indicating the correctness of analogy completion in a single language. The geometric mean of the analogy completion score for embeddings from the two embedding spaces is then used ($S_{PAE}$). \n\nThey show that these two quantities correlate. I.e., The better both embedding spaces preserve analogies, the better they can be mapped through a linear transformation. The paper suggests this can be viewed as a framework which can explain success and failures of linear cross embedding mapping models in different cases.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper studies the connection between the preservation of analogies and the linearity of cross lingual word embedding mappings. This has been one of the assumption behind linear mapping based cross-lingual word embedding models, i.e., the assumption that there exist similar structures in word embeddings models of different languages suggest a linear mapping between them. Given this, I find the findings of this paper a bit trivial. I appreciate if the authors can elaborate a bit why this is not trivial to empirically and theoretically show the validity of this statement. \n\nI am also a bit puzzled about the significance of the theoretical analysis provided. My understanding is that, by analogies the authors are referring to the structures in the word embedding space (the relations between embeddings), and here they are showing that these structures are preserved by linear transformations, and that if these structures are preserved the mapping is linear.  Does this tell us anything beyond the definition and characteristics of linear mapping in general?\n\nHere are some additional comments independent of the aforementioned concern:\n- One thing that I think is worth showing, is that the results reported in the paper are robust with respect to the dataset used for measuring the perseverance of analogies in the word embeddings with respect to both scale and content. \n- Based on how analogy sets are selected, $S_{PAE}$ metric could potentially be an indicator of the similarity between languages (even independent of the learned embedding), between word embeddings, or how well the word embeddings are trained (their performance on other DS tasks). This is discussed a bit in section 6, but I think it should be accompanied by some empirical analysis to support the discussion and show at what level analogies captured by word embedding models matter.\n- To quantify if there is a linear transformation between two embedding spaces, the authors fit a linear model which is optimised through gradient descent and measure its error. What if we don’t converge to the optimal linear transformation during the optimisation process? This is why I think it's important that at least some complementary results on the performance of the word embedding models and their transformations are also provided.\n- It is not clear, how all this fits into the current state of the field. The paper claims this approach can be used to explain some of the inconsistencies in results from different papers on whether and how the linear mapping approach for cross lingual embeddings can succeed or fail. It would make the paper much stronger, if this can be empirically shown in the paper. E.g., by running these analysis on models and languages studied in the previous works that their findings are discussed in this paper.\n", "broader_impact_concerns": "No comments.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DsEB0DRs2D", "reviewer": "Reviewer_ENQy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work is centered around a novel idea and is practically useful.\n- This paper provides a theoretical proof of the main hypothesis as well as empirical experiments. \n- The authors introduce a new cross-lingual dataset for the word analogy task.\n- The paper is well written and I've enjoyed reading it.\n\nThe paper doesn’t have any strong weaknesses, but I’ll mention one smaller thing I’ve been thinking about while reading the paper:\n- The novel word analogy dataset has been created via translation of existing datasets. In particular, it didn’t seem as if much thought went into the choice of analogy examples. Could this lead to problems? For instance, could it be that some words are homonyms in one language but not in the source language? If so, how could that influence the findings and would it be better to take those words out/substitute them?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate cross-lingual word embeddings, specifically the mapping from the two individual embedding spaces to a common or shared embedding space. The premise of this work is that, while most prior research has assumed that there is a linear relationship between embedding spaces, it is unclear when this assumption holds or how to test it. The authors propose that the preservation of analogies encoded in monolingual word embeddings are a *necessary* and *sufficient* condition for the ground-truth CLWE mapping between those embeddings to be linear. \n\nThe first contribution of this paper is that the authors theoretically prove the above assumption. They then present a new, cross-lingual word analogy dataset, which covers a total of 12 languages and is this paper’s second contribution. Using this dataset the authors conclude by showing that their assumption holds true empirically, which is this paper’s third contribution. (The authors mention that the empirical study is needed as the theoretical proof is based on the simplification that word embeddings spaces can be treated as *continuous* vector spaces, an assumption common in prior research.)\n\nOverall, I believe this is an insight-providing paper that will be of interest to the cross-lingual word embedding community.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I don’t think any major changes are necessary. However, I’m suggesting some smaller changes. In particular, a couple of comments in the paper aren’t needed for a (somewhat) expert audience, so I would suggest removing them:\n- The explanation of R being real numbers, \n- “(i.e., the forward implication)”, \n- “(i.e., the reverse direction)”, \n- “sometimes described as “isomorphism” in previous work”, \n- “Homogeneity needs a proof that seems more complex, which consists of four steps.” → rewrite to something like “Our proof of homogeneity consists of four steps”.", "broader_impact_concerns": "I don’t have any broader impact concerns regarding this work. Accordingly, the broader impact statement which is already in the paper is sufficient in my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AjgLOvP5bA", "reviewer": "Reviewer_ARiR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* As the paper notes, the relationship between analogies and cross-lingual embedding relationships has not been established in the past. This paper does a very nice job of tackling this from both theoretical and empirical perspectives.\n\n* The introduction of the new xANLG dataset is a very nice contribution. This discussion in the paper is vey thorough and does a good job of establishing the relationship with previous data and why those datasets are unsuitable..\n\n* The empirical results are quite thorough and are analyzed well.\n\n* Section 4.1.3 presents an interesting technique (around the computation of S_PAE) that seems at least partially novel to me.\n\n* Section 6 presents a good discussion of limitations and shortcomings of the work.\n\n* The paper is well-written and cites all the relevant prior work to my knowledge.\n\nWeaknesses:\n\n* The core theoretical result of the paper does not really say a lot about practice. As section 6 establishes (and past work has established), these relationships are not linear.  Moreover, critiques of Bolukbasi et al. have noted that some of these analogy encodings only hold if you exclude the word itself from the computation (e.g., king - man + woman ~ king, with queen being in second place).  So we know that both of these phenomena only hold partially in practice, and the theory here does not say much about that setting.\n\n* It would've been nice if the core proposition (or its proof, even) could've somehow engaged with the idea that these relationships aren't perfect.  If the proof reckoned in terms of error bounds, for example, it could possibly give some intuition about what happens in the case where analogies don't perfectly hold.\n\n* As the correlations in the experiments are not perfect, it is hard to interpret whether the correlations are \"good enough\" that the reader believe the central point of the paper, or whether this is evidence that other effects like those mentioned in Section 6 are at play. Section 6 provides a nice accounting of some error cases, but only at a surface intuitive level.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the relationship between cross-lingual word embedding methods and analogical reasoning in word embedding spaces.  Analogies like king-man+woman=queen assume a linear structure of these spaces; the paper proves that this structure exists in two languages iff those two languages' vectors are related by a linear transformation. This proof is fairly straightforward and not very deep mathematically, but it establishes a fact that is new in the literature. Second, the paper collects a new dataset of multilingual analogies, where the analogy pairs are parallel in multiple languages at once. Finally, a set of experiments show correlation between the linearity of the mapping between two spaces and the extent to which analogies are preserved. This last piece requires an interesting and challenging optimization problem to understand how far a set of vectors is from satisfying the analogy criterion.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I do not have any proposed changes. I think this paper is close to a local optimum. While I would like to see a deeper theoretical result and a closer bridging of the theory and the empirical results, I don't think this is easy to do, and the mix that the paper strikes is good and convincing in its own way. Holding it to the standard of TMLR, I believe it appropriately positions its claims and defends them convincingly.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a study on the relationship between cross-lingual word embedding methods and analogies in word embedding spaces. Specifically, the assumption on the well-studied analogies is that there is linear structure in these spaces. The contribution of this work is that it shows (both empirically and theoretically) that a necessary and sufficient condition for this structure to exist is if the vectors in the two languages are related by a linear transformation. The authors also release cross-lingual analogy dataset which is created automatically.

All reviewers have agreed that this is an interesting submission providing useful insights into the world of analogies. In terms of changes, all reviewers have flagged a number of things to be further studied/commented on, which the authors have acted upon during the discussion period.

All in all, I agree with the reviewers that this is a solid contribution and, given the changes during the discussion period, I recommend acceptance as is.
Decision: Accept (poster)