GRI: Graph-based Relative Isomorphism of Word Embedding Spaces
Authors: Muhammad Asif Ali
Yan HU
Jianbin Qin
Di Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Attentive graph Convolutions
Isomorphim
Bi-lingual Induction
TL;DR: We propose Attentive Graph Convolutions to Control the relative isomorphism of Monolingual Embeddings for BLI.

Abstract: Automated construction of bi-lingual dictionaries using monolingual embedding spaces is a
core challenge in machine translation. The end performance of these dictionaries relies on the
geometric similarity of individual spaces, i.e., their degree of isomorphism. Existing attempts
aimed at controlling the relative isomorphism of different spaces fail to incorporate the 
impact of lexically different but semantically related words in the training objective. To 
address this, we propose GRI that combines the distributional training objectives with attentive
graph convolutions to unanimously consider the impact of lexical variations of semantically
similar words required to define/compute the relative isomorphism of multiple spaces. Exper
imental evaluation shows that GRI outperforms the existing research by improving the average
P@1 by a relative score of upto 63.6%.

GRI: Graph-based Relative Isomorphism of Word Embedding Spaces
Muhammad Asif Ali,' Yan Hu,' Jianbin Qin, Di Wang' King Abdullah University of Science and Technology, KSA Shenzhen University, China
(muhammadasif.ali, yan.hu, di di.w | @ kaust.edu.sa. qinjianbin www.bbuunnzuuuuuccr
Abstract
terrible
Automated construction of bilingual dictionar- ies using monolingual embedding spaces is core challenge machine translation. The end performance of these dictionaries relies upon the geometric similarity of individual spaces, i.e., their degree isomorphism. Existing at- tempts aimed controlling the relative iso- morphism of different spaces fail to incorpo- rate the impact of semantically related words in the training objective. To address this, propose GRI that combines the distributional training objectives with attentive graph con- volutions to unanimously consider the impact semantically similar words required to de- fine/compute the relative isomorphism of mul- tiple spaces. Experimental evaluation shows that GRI outperforms the existing research by improving the average P@ by relative score of up to 63.6%. We release the codes for GRT at https:/ thw.. com/ 2sif6827/GR..
horrible
сrpamniii
awful
WAXIIBO
excollent
BLUMiRHO
чудево
great
good
2o6pe
Figure 1: Semantically related tokens for English and Ukrainian languages. These words though lexically varying carry the same semantics and their impact should be unanimously considered.
geometrically similar. This severely limits the ap- plicability of the mapping- based methods to closely related languages and similar data domains. This isomorphism assumption does not hold, especially in case of comain-mismatee and for languages ex- hibiting different characteristics (Conneau et al., 2017; Søgaard et al., 2018: Glavas al., 2019; Patra al., 2019). Other dominant factors identi- fied the literature that limit the end performance of BLI systems include: (i) linguistic differences (ii) algorithmic mismatch, (iii) variation in data size, (iv) parameterization etc. Similar to the super- vised models, the unsupervised variants of BLI are also unable to cater to the above-mentioned chal- lenges (Kim et al., 2020; Marie and Fujita, 2020). Instead of relying on embedding spaces trained completely independent of each other, in the re- cent past there have been a shift explicitly using the isomorphism measures alongside distributional training objective (Marchisio et al., 2022). In order to control the relative isomorphism of monolingual embedding spaces, these models use existing bilin- gual dictionaries training seeds. However, core limitation of these models is their inability to incorporate the impact of semantically relevant tokens into the training objective. This severely de- teriorates the relative isomorphism the resultant lingul embedding space.
1 Introduction
Bilingual Lexical Induction (BI.)) aims the con- struction of lexical dictionaries using different mono-lingual word embeddings. Automated con- struction of bilingual dictionaries plays a signifi- lan-
cant role, especially for resource-constraiigg guages where hand-crafted dictionaries are almost non-existent. It is also key tool to bootstrap the performance of many down-streaming applications, e.g., cross-lingual information retrieval (Artetxe al., 2018a), neural machine translation (Lample et al., 2018).
The most prevalent way for the construction of cross-lingual embeddings is to map the mono- lingual embeddings in a shared space using lin- car and/or non-linear transformations, also known mapping-based methods (Conneau et al., 2017; Joulin et al., 2018; Patra et al., 2019). A core limita- tion of the mapping- based methods their reliance on the approximate isomorphism assumption, i.c., the underlying monolingual embedding spaces are

---

This phenomenon is illustrated in Figure for English and Ukrainian languages. For example, for the English language, we often use terms ["ter- rible", " within the same context with- out a significant change in the meaning of the sen- tence. For these terms, corresponding terms in the Ukrainian language ("crpaumıíää, "жаххиi-- )O") may also be used interchangeably without significant change in the context. Likewise, for the bottom row in Figure the words "goood, " "grear", "excellent") are semantically related words in the English language, with ("Biдмiнно" "uy- ROBO", "rобрe"} as corresponding semantically related words in the Ukrainian language.
of the art by approximately 63.6% for average P@L.
2 Related Work
Due to limited space, we primarily categorize the related work on relative isomorphism of cross- lingual embeddings into: (i) mapping shared space, and (ii) joint training.
Mapping to shared space. These models aim to find linear and/or non-linear transformation for pre-trained word embeddings in order to map them to a shared space. These models rely on the assumption that the embedding models share sim- ilar structure across different languages (Mikolov et al., 2013a), which allows them to independently train embeddings for different languages and learn mapping functions to align them in shared space. Supervised variants in this regard use existing bilin- gual resources, such as parallel dictionaries (Xing et al., 2015; Joulin et al., 2018 8; Jawanpuria et al., 2019). The unsupervised variants use distributional matching (Zhang et al., 2017; Conneau et al., 2017; Artetxe et al., 2018b; Zhou et al., 2019). These models have also been applied to the contextualized embeddings (Aldarmaki and Diab, 2019; Schuster et al., 2019).
To address these challenges, in this paper propose novel framework named: Graph-based Relative Isomorphism (GRI). GRI uses attentive graph convolutions to pay attention to semantically related tokens, followed by using isomorphism met- rics to inject this information into the model train- ing. Later, it combines the isomorphism loss with the distributional training objective to train the com- plcte model.
We argue GRT offers a better alternative for BLI, as allows injecting information about the seman- tie variations of tokens in the training objective, which is a more natural setting in order to control the relative isomorphism of linguistic data. An immediate benefit of the proposed model is obvi- ous in the mismaach settings, where atten- tive graph convolutions mechanism by GRI offer the ability to unanimously analyze and/or model similar concepts represented by lexically varying terms across different corpora. This is also evi- dent by a relatively stable performance of GRI for both domain-sharing and domain-mismatch set- tings (Section 1.... We summarize the core contri- butions of this paper follows:
Joint Training These models put additional con- straints model learning, i.c., hard or soft cross-lingual constraints in addition the mono- lingual training objectives. Similar to the mapping- based models, early works in this domain include the supervised variants relying on bilingual dictio- naries (Ammar et al., 2016; Luong et al., 2015; Gouws et al., 2015). Recently, the unsupervised approaches have gained attention because of their ease of implementation. For instance, Lample et al. (2018) analyzed the performance for concatenated monolingual corpora with shared vocabulary with- out any additional cross-lingual resources. Re- sults show that this setting outperforms many care- fully crafted alignment based strategies for unsuper- vised machine translation. Other unsupervised ap- proaches with good results on benchmark data sets include zero-shot cross- lingull transfer by Artetxe and Schwenk (2019) and cross-lingual pre-training by Lample and Conneau (2019). Marchisio et al. (2022) proposed Iso that introduces multiple different losses along with the skip-gram loss func- tion to control the relative isomorphism of mono- lingual spaces.
1. We propose GRT that combines isomorphism loss functions (guided by graph convolutions) along with the distributional training objective for BLI.
2. We propose attentive graph convolutions for GRI in order to control the relative isomor- phism by sharing information across semanti- cally related tokens.
3. We illustrate the effectiveness of GRT via com- prehensive experimentation. For benchmark data sets, GRI outperforms the existing state

---

A major limitation of these methods is their inability to incorporate the lexico-semantic vari- ations word pairs across different languages in the model training, which severely limits the end performance of these models.
Skip-Gam
Graph Coneelution
Isomorphiis
RALL
RALU
Source
Source
3 Background
In this section, we discuss the notation and math- ematical background of the tools and techniques used in this paper.
Figure 2: Proposed framework for Graph-based Rel- ative Isomorphism(GRD) It combines attentive graph convolutions with the skip-gram to control the relative isomorphism for source U and target V embeddings.
3.1 Notation
U relative to space V, GRI uses attentive graph convolutions. Finally, it uses multiple different iso- morphism metrics along with the skip-gram loss function for model training.
Throughout this paper, we use U C Rexd and V c Raxd to represent the embeddings of the source and target languages. We assume the availability of seeds pairs for both source and target languages, represented by: {wo, vp), (n1, (u),,..(,,,,
We claim the proposed model provides the pro- vision to perform BLI in a performance-cnhancee fashion by using attentive graph convolutions for effective propagation of semantic relatedness of tokens across different languages.
3.2 VecMap toolkit
For mapping across different embedding spaces, we use vecmap toolkit'. We follow Zhang et al. (2019) to pre-process the embeddings, i.e., the em- beddings are unit-normed, mean-centered and unit- normed again. For bilingual induction, we follow the steps outlined by (Artetxe al., 2018 i.c., whitening each space, and solving Procrustes. This is followed by re-weighting, de-whitening, and mapping of translation pairs via nearest-neighhor retrieval. For details, refer to the original work by Artetxe et al. (2018) 8a).
4.3 GRI
In order learn the distributional embeddings for the source language that are geometrically similar to the target embeddings GRI incorporates attentive graph convolutions along with the distributional training objective. GRI relies on the assumption that each language possesses multiple variations of semantically similar tokens that may be used inter- changeably. And, in order to effectively model the relative isomrrphism for the multi multi-lingual data sets this phenomenon needs to be captured explicitly.
4 Proposed Approach
The proposed model (GRI) is based on the as- sumption that sharing information among semanti- cally related words is inevitable in order to control the relative isomorphism of the embedding spaces. From the linguistic perspective, there are arbi- trary number of words semantically related to given word, which makes graphs natural choice to unanimously consider the impact of these words in the end model. We explain the individual com- ponents of GRI in the following sub-sections:
4.1 Problem Definition
this paper, we address a core challenge con- trolling the relative isomorphism for cross-lingual data sets, i.e., incorporate the impact of semanti- cally coherent words for BLI.
4.2 Overview
We propose Graph-based Relative Isomorphism GRI, shown in Figure 2, that aims to learn distribu- tional information the source embedding space U, in such way: (i) U is geometrically similar to the target embedding space V to the best possible extent, (ii) U captures information about the se- mantically related terms in V. In order capture the distributional information GRI uses skip-gram with negative sampling. order to control the geometry and isomorphism of embedding space
4.3.1 Distributional Representation
In order to learn the distributional representations GRT uses skip-gram with negative sampling. For training the skip-gram gram model, we follow the same settings as outlined by Mikolov al. (2013b), i.e., embed word close its contextual neighbors and far from a set of words randomly chosen from the vocabulary. The formulation the skip-gram loss function illustrated in Equation I.
Chttps://githhi ctctt/uu..ooc///raaeaeeeeee com/arrteexeee/eecmp

---

Algorithm 1 Graph Construction
LsG = log ouico T
wcs)+
Input: Embedding (EMB);
Diar = Targetí Dtr+deu+tst)
(1)
Σ E Ey Pooo (-ule wcr)]
Output: Graph: G
1: Pairsprob <- 0: <- 0
2: for (W1, W2) f- Pairs( do
Here the and Mey correspond to the output and in- put vector representation of the word c. Ve the embedding vectors for noise terms. (c(c) corre- sponds to the noise distribution, k is the number of noisy samples. We use k = 10 in our case.
3:
y = scoreEMB (1E1, (11, W/2)
4:
if y 2 thr then
5:
Pairsprob + PairSprob U (Wi, w2)
end if
end for
4.3.2 Capturing Semantics
for pair PairSproo do G GU (edu(edgepaarrr) 10: end for
In order to control the relative isomorphism across the source and target embeddings GRI uses atten- tive graph convolutions under transductive settings in order to share information among semantically related words. The graph construction is summa- rized in Algorithm 1, and explained as follows:
11: return
of the word that accommodates information from semantically related neighbors.
Graph Construction. Inputs for the graph con- struction include: (i) the supervision seed pairs for the target language, (ii) existing pre-trained word2vec embeddings (Mikolov et al., 2013b). The graph construction process proceeds as fol- lows:
Note, in our setting, unlike the existing work by Kipf and Welling (2016), we propose attentive graph convolutions with pair-wise distributional similarity scores as the hard attention weights. The attention weights not updated during the model training. Specifically, we use the following layer- wise propagation mechanism:
Firstly, we organize the target words into all pos- sible pairs, i.e., combinations of two words time. For each word pair, we compute score (co- sine similarity) of their corresponding embedding vectors. The word pairs with scores higher than a threshold (thr) are stored as the probable seman- tically related terms (PairSprot). illustrated in lines (2-6). We observed that using a significantly higher value for thr is beneficial, because: (i) it helps in capturing the most confident word pairs thus over- coming noise, and (ii) only few word pairs end up pairs which makes it computationally ef- ficient. Finally, for all word pairs in Pairsprob. we formulate edges to construct the graph G. For cach word pair, we use the cosine score of corresponding embedding vectors as the attention weight.
( -p(f
(2)
where r= (T I)D 1/2 is the normalized symmetric matrix, D is the degree matrix of I, I is the weighted adjacency matrix learned from graph G with pair-wise scores the attention weights, 1 is the input from previous layer, with I,(0) U Rexd as the input matrix corresponding to the embeddings of the source terms, W; is the learn- able weight matrix, is the non-linear activation function.
Note, the end goal of the attentive convolutions is analyze each word in G in relation to the weighted combination of its semantic neighbors. For this, we surround each word (node) with a group of semantically related words (nodes the graph) and perform weighted aggregation to re- compute the representation of the word. We also allow self- ceffeconnettons for each word, i.e., adding an identity matrix / to r' This will enforce "seman- tically related words" to get similar representations. We observed, that for our problem settings, this at- tentive graph convolution framework outperforms the basic settings with equal contribution from all neighbors (Kipf and Welling, 2016).
Attentive Graph Convolutions. Depending upon the value of thr, graph G surrounds each word by a set of highly confident semantically related words (including their lexical variations). The degree of similarity is controlled by the cosine similarity of embedding vectors. Later, for each word, we aggregate the information contained in the neighbors come up with new representation
'tt / //t/////oeeeee//ee///e google. cww/ wwchhieeeeppp //word// trained using Google-News Corpus of 100 billion words.
For GRI, we use a two-layered network to learn

---

the final embeddings of each word Um Rexd as follows:
Experiments and Results
5.1
Datasets
Un - (TeLL (()L
(3)
In order to set up unanimous platform for com- parative analysis, we use the data settings used by Marchisio et al. (2022). We use the first 1 mil- lion lines from newscrawl-2022 data set for English ("en"), Bengali ("bn") and Tamil ("ta") and the en- tire of newscrawl-2022 data for Ukrainian ("uk") train word embeddings. We used Moses scripts for data pre- pre-processing'. For evaluation, we used pub- lically available train, dev, and test splits provided by MUSE (Conncau et al., 2017). Out of approx 8000-word pairs for each language, we used word pairs 0-5000, 5001-6500, and 6501-8000 as train, test and dev respectively. The train set is used for model training. dev set is used for parameter tuning, and final results are reported using the test set. All these data splits are non-overlapping.
4.4 Isomorphism Loss functions
In order to train the GRI, we experiment with mul- tiple different isomorphism loss functions on top the attentive graph convolution network. Details about each loss function are as follows:
1.2 loss. We use 1.2-norm averaged over the num-
ber of data samples.
C2 U Vila
(4)
Orthognal Procrustus loss. Orthogonal Pro- crustes loss aims to find linear transformation W to solve:
5.2 Baseline Models
Cyroe:
arg min WeR.
U, W Vil2
N
For comparative evaluation, we use independently trained distributional embeddings for the source and target languages as a baseline model. Amongst the existing research, we compare GRI against the prevalent ofte-off-hhe-n work on BLL, i.e., Iso by Marchisio et al. (2022). Vec uses the skip- gram training objective along with isomorphism training objectives. Note, Marchisio et (2022) used exactly the same data settings as that of our proposed model (i.e., GRI), so for performance comparison, we use the numbers reported in the original paper.
(5)
Schönemann (1966) proposed solution W QPT, where PEQ" is the singular value decom-
position of VTU
Procrustus loss with initialization. It follows the same process that of the Procrustus loss with the exception that we initialize the embeddings for the source words with the embedding vectors of their translation vectors corresponding to the target words. The end goal of this setting to analyze the ability of the GRT to propagate the knowledge of the initialized embeddings during the model train- ing. We also allow updating the initialized word embeddings during model training. We denote this loss by Eprociast We use the symbol LISO repre- sent the different variations of isomorphism losses,
5.3 Experimental Settings
For model training, mizer (Kingma and Ba, 2014) with learning rate 0.001; a in Equation 6 is set to 0.7; the value of thr algorithm
Adam opti-
set to 0.5. For experiments, we use embeddings learnt for English language as the target embeddings, and embeddings for other languages, i.c., "ta", "uk", , and "bn", as the source embeddings. For mapping across different spaces, we use Vecmap toolkit with process-flow flow explained in Section 3.2. We use average P@ the evaluation metric. We report mean (μ²) and standard deviation (a) of the results over 5 runs of the experiment. All experiments are performed using Intel Core-19-10900X CPU, and Nvidia 3090Ti GPU.
i.e., L2, Eproe and Eproceat
4.5 The Complete Model
Finally, we combine the distributional training ob- jective with the isomorphism loss function to com- pute complete loss of GRI, as follows:
LGRI aLsG (1 Ciso
(6)
where a is the parameter used control the contri- bution Eso and Liso respectively.
Moses script

---

Methodology
(iv) Error cases.
Raseline
[3
13.9 045)
(0.22)
Voo (1.2)
16.3 04)
16.5 16.5 04)
(+.5))
Voc (Proe-1.2)
(67())
16.0 0.8)
10.7 (+-0.3)
6.1 Domain mismatch
Vee (Proe I.2.
(+ 02)
17.1 (4 0.)
11.8 (+ 0.3)
GRI (Cg)
17.28 0.02)
18.75 (4 0.1)
13.47 (4 0.04)
has been identified as one of the core limitations of existing BLI methods. These methods fail badly in inferring bilingual informa- tion for embeddings trained on data sets from dif- ferent domains (Søgaard et al., 2018; Marchisio et al., 2020).
GRI (Cuoorue)
19,83 (1: 0.05)
21.37 (4 0.08)
15:27 (1 0.01)
GRI (Cyrec)
20.52 (= 0.02)
27.97 (# 2.63)
18.97 (#: 0.2)
Table I: GRT results for the proposed model. We com- pare results with Iso (Marchisio et al., 2022).
5.4 Main Results
The results for the proposed model (GRI) compared with the baseline models are shown in Table 1. We boldface the overall rest-performing scores with the previous totte--o-thheaare underlined.
We claim that incorporating lexical variations for semantically related tokens makes GRT robust the domain mismatch settinss. In order validate these claims for GRI, we re-run the experiments using target embeddings trained on 33.8 million lines of web-crawl data from the English Common Crawl data. The embeddings for the source lan- guages ("bn", "uk" and "ta") are trained using the newscrawl-2020 data. The results for the domain- mismatch experiments for different isomorphism loss functions are reported in Table 2.
These results show the GRI has a relatively sta- ble performance (with low variance), consistently outperforms the baseline and previous state-of- the art scores by a significant margin. For "bn", "uk", and "ta", GRI outperforms the Isc (Marchisio et al., 2022) by 21.4%, 63.6% and 60.7% respec- tively. Especially noteworthy is the performance improvement gained by GRT for the Ukrainian lan- guage. We attribute this performance improve- ment to the fact that the semantic relatedness of the words corresponding to the Ukrainian embed- ding space is relatively better compared to other languages.
These results are compared against the base- line distributional embeddings and best-performing scores of the existing work, i.e., Iso by Marchi- sio et al. (2022). Note, for the domain mismatch experiments,
use exactly same data settings as that of Marchisio et al. (2022), so we report exactly the same numbers as reported in original paper. Comparing the results of our model against the Vec, the GRI improves the performance by 27.74%, 12% and 74.22% for the "bn", "uk" and "ta" languages respectively.
The performance comparison of different iso- morphism loss functions shows that Cprae consis- tently outperforms the Eprocuna and L2 across all data sets. A relatively low performance of Cprocuis compared to the Pprc may be attributed to the fact that randomly initialized embeddings are a better choice compared to the initialization from the seed pairs. The initialization from the seed pairs may not be helpful for the model training to improve the performance at later stages.
Comparing these results against the main ex- periments reported in Table we can see the GRI yields a stable performance for both domain- shared as well as domain mismatch settings. These results show that the attentive graph convolutions indeed allow information sharing across semanti- cally related tokens along with their lexical varia- tions that is in turn helpful in controlling the rela- tive isomorphism of the embedding spaces. Comparing the results for different loss func- tions, we can see that similar to the main experi-
Overall results show the significance of using attentive graph convolutions in controlling the rel- ative geometry of source language for BI... Espe- cially, the ability of the attentive convolutions to accumulate the contribution of semantically related terms plays vital role in controlling the relative geometry of the source embeddings relative to the target embeddings, as is evident from the results in Table 1.
Methodology
Raseline
0.722
10.2 0.43)
7-5
Troee (P2nc-1.2-miit
15.5 0.7)
17.3 0.4
10.9 (.5)
6 Discussion
GRI GRI(C))
13.97 (= 0.02)
17.32 (+ 0.32)
11.93 0.01)
19.75 601)
21.32 0.10)
17.12 0.59)
GRI (Cow))
19.80 (2: 0.50)
26.49 0.50)
18.99 0.20)
In this sub-section, we perform a detailed analysis of the performance of GRI. Specifically, we ana- lyze: (i) Domain mis-match settings (ii) Impact of attentive convolutions, (iii) Isometric metrics, and
Table 2: GRI results for domain mis-match experiments compared with the baseline models, Iso (Marchisio et al., 2022).

---

Melhodology
Peurson Correlation (1)
Eigenvector Similarit())
GRI w/o G-Cone (C2)
16.10 0.35)
16.35 (1: 0.30)
11.23 c2: 0.45)
Methodology
be
uk
ta
bn
uk
GRI wio G- G-Cone (Coretma?
16.75 0.0
16.98 0.30
11.70 0.25)
GRI (CC)
0.47
0.36
0.4
35.55
30.64
69.72
GRI w/o G-Cone (Earac)
0.)
16.10
0.70)
10:65 0.20)
GRI (C (Ceeossun))
0.47
0.36
0,43
31.23
10.92
45,56
GRI (4a)
17.28 (1 0.20)
18.75 (1 0.41)
13:47 (4: 0.04)
GRI (Cpro)
0.49
0.37
0.44
32,16
29.53
62.81
GRI )
19.83 (1: 0.081
21.37 0.08)
15.27 0.011
GRI (ppra))
20.52 (1: 0.021
27,97 2,3))
18.97 (4: 0.200
Table 4: Analysis of different isometry metrics for GRL
Table Analyzing the impact of attentive graph convo- lutions for GRI.
seed translation pairs an indicator of the relative isomorphism of corresponding spaces. We expect our p@ results to correlate positively (t) with Pearson's correlation.
ments the performance of the model for the Pro- crustes loss (Sproc) is relatively higher than the L2 and Eprocina
We compute Pearson's correlation over first 1000 translation seed pairs. Corresponding re- sults are shown in the first half of Table 4. We boldface the best scores. These results show that for all languages, Pearson's correlation for the model GRI (Eproc) is slightly higher compared to other models. Although these results are aligned with our findings in Table 4, however, one note- worthy observation is that Pearson's correlation is not true indicator of the relative performance improvement across different isomorphism losses.
6.2 Impact of attentive convolutions
In this sub-section, we analyze in detail the per- formance improvement of GRI attributable to the attentive graph convolutions. For this, we primarily analyze the performance improvement of GRI with and without attentive graph convolutions. The re- sults of these experiments are reported in Table 3. These results show the significance of attentive graph convolutions that help in improving the per- formance across all three languages. The improve- ment in performance for the "bn", "uk" and "ta" languages is 24.36%, 64.72% and 62. 13% respec- tively.
Eigenvector Similarity. In order to compute the eigenvector similarity of two spaces, we com- pute the Laplacian spectra of the corresponding k-nearest neighbor graphs. This setting is simi- lar to Søgaard et al. (2018), and is summarized as follows. For seed pairs construct unweighted graphs followed by computing the graph Lapla- cians. Later, compute the eigenvalues of the graph Laplacians and retain the first eigenvalues sum- ming to less than 90% of the total sum of eigenval- ues. Finally, we compute the eigenvector similarity as the sum of squared differences between partial spectra. The graphs with similar eigenvalue spectra supposed to have similar structures (a measure of relative isomorphism).
To gain further insight, we also analyzed the output of the model with and without graph convo- lutions in order to dig out which class of translation instances were correctly translated especially due to the attentive convolutions part of GRI. We rum this analysis only for the Ukrainian language be- cause: GRI yields a higher score for the Ukrainian language compared to other languages. All the analyses were performed under the direct supervi- sion of a linguistic expert.
Detailed analyses show that major portion (ap- prox 51 %) of the pairs corrected especially by the graph convolutions belong to the nouns, with 21 % verbs and 20% adjectives. The rest 7% are as- signed to other classes. This analysis shows that the phenomenon of lexical variation is dominant among nouns that results in better performance of GRT compared to the baseline models.
We expect our eigenvector similarity results to correlate negatively (L) with P@1. The experi- mental results are shown in the right half of Ta- ble 4, with the best scores boldfaced. These results show that the eigenvector similarity scores for the model GRI (Eprocana) are better than the other two models. This is in contrast to our findings in Ta- ble 1, where GRI (Cproc) shows relatively better performance.
6.3 Isometric metrics
We also correlate the results of GRI with different widely used isomorphism metrics. Specifically, we use two metrics, namely: (a) Pearson's 's correlation, and (b) Eigenvector similarity. Details about these metrics and the corresponding experimental setting are as follows:
Generally speaking, the results of the isometric metrics do not truly correlate with the P@ 1. These findings are aligned with earlier studies by Marchi- sio al. (2022) that also emphasized the need for better metrics compute the relative isomorphism of the embedding spaces.
Pearson's 's Correlation. We compute Pearson's 's correlation between the cosine similarities of the

---

6.4 Error Analyses
GRT
GRI wio G-ConvlCrred
source
turget
target
source
target
larget
We also analyze the error cases of GRI in order to know the limitations of the model and room for fu- ture improvement. Note, similar section 6.2, we only perform the error analyses for the Ukrainian language and Procrustes loss ppro.) ) All experi- ments were performed with the help of linguistic experts. We separately analyze the errors for the variants of GRI with and without attentive graph convolutions (i.e., GRI; GRI w/o G-Conv) in order to quantify the reduction in error attributable to the attentive graph convolutions.
mail
gone
shurtered
shared
sharing
axis
oil
60g
window
windows
Gomca
merchants
2013
college
teaching
me
nose
rubbing
walked
replacing
overpriced
mappy unnnkin
munaals
templates
nyanon
volcano
100mph
peçjopacx
reform
reforms
piet
growth
deeline
Table 6: Example error cases for the Ukrainian vs En- glish language for the models: GRI (Eproc): ) GRI w/o G- Coree) For each model, first column (source) corresonnd to the Ukrainian words, the second col- umn (target) represents the true target word, third col- umn (target represents model predictions for target words.
In order to better understand the errors from sc- mantic perspective, we categorize the errors into the following types:
respectively. These results clearly indicate that the current performance of GRI is under-reported and there need for better quantification measures (other than P@ 1) in order to compute and/or report the true potential of the model.
Type-a: The predicted target word for P@1 I is semantically close the true target word.
Overall error analyses show the significance of the using attentive graph convolutions to incorpo- rate the lexical variations of semantically related tokens in order control the relative isomorphism and perform BLI in performance-enhancee way.
Type-h: The predicted target word is k-nearest neighbor of the true word for k=5,
We limit the error cases to only the above- mentioned simple types because these types give a rough picture of the relative isomorphism of the different spaces from the semantic perspective. The percentage error counts for both models are shown in Table 5. For the model GRI w/o G-Conv(Cproa) 21.3% errors fall in error Type-a, and 6.5% errors belong to error Type-b. For the model GRocC 50.2% errors fall in Type-a, and 16.6% errors be- long to Type-h. As expected the variant GRI with graph convolutions shows higher percentage for both categories, i.e.. Type-a and Type-b. These numbers clearly indicate that the attentive graph convolutions were not only able correct major portion of errors made by (GRI w/o G-Conv), but also the errors made by the model GRI are either highly semantically related to the target words or a nearest neighbor of the target word.
7 Conclusion and Future Directions
In this paper, we propose Graph-based Relative Isomorphism (GRI) to incorporate the impact of lexical variations of semantically related tokens in order to control the relative isomorphism of cross- lingual embeddings. GRT uses multiple different isomorphism losses (guided by the attentive graph convolutions) along with the distributional loss fashion.
perform BLI in a performance- Experimental evaluation shows that GRI outper- forms the existing research on BLI by a significant margin. Some probable future directions include: (i) extending the concepts learned in this research to contextualized embeddings, and (ii) augment- ing the GRI to focus more on preserving lexico- semantic relations.
In order to gain further insight, we manually look the error cases. For both models, few examples are shown in Table 6. The majority of the predictions made by GRI are indeed correct and closely related to the true target words. For example, it predicts ("mailing", "sharing", "win- dows"] | in place of ("mail", "shared", "window")
Limitations
Some of the core limitations of the proposed ap- proach are as follows: (i) current formulation of GRI is not defined and/or implemented for deep contextualized embeddings which are more preva- lent and better alternate to the distributional embeddings, (ii) existing limitations of the distri- butional embeddings are inherited in the model, which limits the end-performance of GRI. For ex- ample, as pointed out by Ali et al. (2019) the dis- tributional embedding space tends to inter-mix dif-
Type-a
Type-b
GRT w/o G-Conv(Cprod 21.3%
6.5%
GRI (Cpree)
50.2%
16.6%
Table 5: Classification of Error Types

---

ferent lexico-semantic relations, and yield poor performance on specific lask. This phenomenon has a direct impact on GRI especially on control- ling the relative isomorphism of highly interlinked relation pairs, e.g., Antonyms vs Synonyms.
Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. Bilbowa: Fast bilingual distributed represen- tations without word alignments. In International Conference on Machine Learning, pages 748-756. PMLR.
Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukut- tan, and Bamdev Mishra. 2019. Learning multilin- gual word embeddings in latent metric space: a geo- metric approach. Transactions of the Association for Computational Linguistics, 7: 7:107-120.
Acknowledgements. Di Wang, Yan Hu and Muhammad Asif Ali are supported in part by the baseline funding BAS/1/1689-01-00 funding from the CRG grand URF/1/4663-01-00 FCC/1/1976- 49-01 from CBRC and funding from the AI Initia- tive REI/1/481 10-000 of King Abdullah Univer- sity of Science and Technology (KAUST).DE Di Wang is also supported by the funding of the SDAIA- KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST Al).
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilinguall word mapping with a retrieval criterion. arXiv preprint arXiv: A.Kv:/804.4. 07745.
Yunsu Kim, Miguel Graça, and Hermann Ney. 2020.
When and why unsupervised neural machine trans- lation useless? In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation.
References
Diederik P Kingma and Jimmy Ba. 2014, Adam: A method for stochastic optimization. arXiv preprint 142.:944.
Hanan Aldarmaki and Mona Diab. 2019. Context- aware cross-lingual mapping. arXiv preprint a3X34:190..03243.
Thomas Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. arXiv preprint 160.v06000009000
Muhammad Asif Ali, Yifang Sun, Xiaoling Zhou, Wei Wang, and Xiang Zhao. 2019. Antonym-synonym classification based on new sub-space embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence.
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining, arXiv preprint /0ti:/990.0777
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guil- laume Lample, Chris Dyer, and Noah A Smith, 2016. Massively multilingual word embeddings. arXiv preprint arXiv:1602.01922
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018. Phrase- based & neural unsupervised machine translation. arXiv preprint 1oX4v:880400775
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Generalizing and improving bilingual word embed- ding mappings with multi-step framework of linear transformations. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the Ist workshop vector space modeling for natural language processing, pages 151-159.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. arXiv preprint a (ee.:8055000 002.06697.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020.
When does unsupervised machine translation work? arXiv preprint arXiv:2004.05516
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Kelly Marchisio, Neha Verma, Kevin Duh, and Philipp Koehn, 2022. Isovec: Controlling the relative iso- morphism of word embedding spaces. arXiv preprint arXiv:2210.05098.
Benjamin Marie and Atsushi Fujita. 2020. Iterative training of unsupervised neural and statistical ma- chine translation systems. ACM Transactions on Asian and Low-Resource Language Information Pro- cessing (TALLIP).
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou, 2017. Word translation without parallel data, arXiv preprint 11X
Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulie. 2019. How to (properly) evaluate cross- lingual word embeddings: On strong baselines, com- parative analyses, and some misconceptions. arXiv preprint 10l:1992.00000
Tomas Mikolov, Quoc V Le. and llya Sutskever. 2013a. Exploiting similarities among languages for machine translation. arXiv preprint arXiv: 130.4439..168
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-

---

rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision non-isometric embedding spaces. arXiv preprint arXiv:1908.0660
Peter H Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of con- textual word embeddings, with applications to zero-shot dependency parsing. arXiv preprint arXiv:1902.09492.
Anders Søgaard, Sebastian Ruder, and Ivan Vulić, 2018. On the limitations of unsupervised bilingual dictio- nary induction. Xiviv preprint arXiv: 1805.03620
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal trans- form for bilingual word translation. In Proceedings of the 2015 conference of the North American chap- ter of the association for computational linguistics: human language technologies, pages 1006 1011.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume Long Papers), pages 1959-1970.
Mozhi Zhang. Keyulu Xu, Ken-ichi Kawarabayashi, Ste- fanie Jegelka, and Jordan Boyd-Graber. 2019. Are girls neko or shi;= ojo? cross- massullinuull alignment of non-isomorphic embeddings with iterative normal- ization. arXiv preprint arXiv: 1906.0000000 01622.
Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. 2019. Density matching for bilingual word embedding. arXiv preprint arXiv: :xxx:19944 1994.02343


Reviews:
[{"review_id": "0HQuQqOToE", "reviewer": "Reviewer_oHhd", "summary": "", "questions": "- Sec 4.3 Do you have multiple cliques in the graph? Do you get a fully-connected graph across all words? I understand it's a fully-connected graph between each group of semantically similar words but I am wondering if there are connections between those that aren't semantically similar according to your threshold. \n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. It will help a lot in improving the overall content of the paper. Our response for each individual point is as follows:\n\n### Answer to Question:\n\n1. Yes, depending on the value of the threshold we end up with an adjacency graph with multiple cliques. There are no connections between the node pairs with similarity lower than the threshold values, as it will incur noise in the model training.\n\n\n### Presentation improvement:\n\n1. For Figure 1, we will add more details to the figure's content and its explanation in order to make the overall workflow of GRI more explicit and self-explainable.\n    \n2. For Section 4.3, the simplest intuitive explanation of the Attentive Graph Convolutions is to perform a weighted aggregation of lexically varying but semantically related words. This in turn enforces the representation of these words to be close to each other for effective BLI.\n\n3. The simplest intuitive explanation for the interaction of the graph construction and attention mechanism is as follows: \n\n    * Graph construction aims at formulating the graph in a way that: \"lexically different but semantically similar words end up being neighbors in the graph\".\n    \n    * Later, Attentive Graph Convolutions aggregate the information in the graph to re-compute the representation of each word in the graph. We will these details in the final draft to make it more clear.\n\n### Typos:\n\n1. In Sec 4.2 \"best possible extend\" has been updated to: \"best possible extent\".\n\n2. All language names (Ukrainian, Tamil, Bengali, and English) have been upper-cased throughout the draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel Technique: the emphasis on semantically similar words is new and the use of graph convolutions is quite interesting. \n- Comprehensive experiments\n- Strong results", "reasons_to_reject": "none in particular", "questions_to_authors": "- Sec 4.3 Do you have multiple cliques in the graph? Do you get a fully-connected graph across all words? I understand it's a fully-connected graph between each group of semantically similar words but I am wondering if there are connections between those that aren't semantically similar according to your threshold. \n\n", "paper_topic_and_main_contributions": "This paper proposes a novel graph convolution approach for Bilingual Lexicon Induction (BLI). \nThe graphs used in the work captures relations between semantically related words (e.g. \"good\", \"awesome\") and the approach tries to minimize graph isomorphism in bilingual spaces. \n\nThe experiments are very comprehensive, in particular in (1) reporting 5 random trials for each of the three language pairs, (2) conducting different data setups, such as the more challenging but realistic domain mismatch setup in Sec 6.1, and (3) ablation experiments and error analyses in Sec 6.2-6.4. In general, I think this is a nice paper that will help push the frontier of BLI.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Fig 1 and Attentive Graph Convolutions subsection in Sec 4.3 - I think it might be helpful to also  try to explain intuitive what does your graph method do. It may not be very clear to some how the graph attention interacts with your graph construction to create isomorphic results.\n\nTypos:\n- Sec 4.2 \"best possible extend\" -> extent\n- Sec 4.3 english and ukrainian languages. -> uppercase language name"}}, {"review_id": "9V80Ofxar5", "reviewer": "Reviewer_AjeL", "summary": "", "questions": "Lines ~240-244 - why does semantic relatedness imply that graphs are a natural choice?\n4.3.1 - did you implement SkipGram yourself?\nWhich retrained word2vec embedding did you use?  Note that MUSE is fasttext.\n“We aggregate the information contained in the neighbors to come up with a new representation for the word that accommodates information from semantically related neighbors” - how?\nSection 4.3.2 - the math notation is confusing. What is the encircled dot?  What is n_ij?  I recommend explaining only what is necessary, with no jargon or unnecessary equations, as these obscure what you are doing. \n(6) - I do not see that L_iso is defined.  \nLine 402 - “process flow”?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, thanks for a detailed read of the paper. It will significantly help in preparing the final draft of the paper if accepted. The response for each individual point is as follows:\n\n### Contributions of GRI:\n\n1. The main contribution of GRI is the use of Attentive Graph Convolutions to incorporate the impact of \"lexically varying but semantically related words\", {e.g., terrible, horrible, awful, etc.,} for Bilingual Lexical Induction (BLI). The key idea is to surround each word (node in the graph) with a group of semantically related words (though lexically different) and perform weighted aggregation to re-compute the representation of the word. This will enforce \"lexically varying but semantically related words\" to be embedded close to each other, which in turn boosts the end performance of the BLI systems. We will add more details and concise explanations throughout the paper (especially Section 4.3) to make it easy to understand.\n\n### Answers to Questions:\n\n1. **Line 240-244:** It is not the semantic relatedness between words, instead, it is the computations over an arbitrary number of lexical variations for words that are semantically related to a given word that make graphs a natural choice. Details are as follows:\n\n    * In order to capture multiple lexical variations of a given word, GRI first constructs a graph such that each word (node in the graph) is surrounded by an arbitrary number set of semantically related words (i.e., neighbors in the graph).\n    \n    * Later, re-compute the representation of each word by aggregating the information from neighbors over multiple hops. Note it also provides the provision to aggregate the information from immediate and higher-order neighborhoods by stacking multiple layers.\n    \n    \n2. **Skip-Gram:** Yes, we implemented the skip-gram model using Pytorch. Our implementation of skip-gram is primarily inspired by existing works, i.e., online tutorials, skip-gram with negative sampling [1], and IsoVec [6].\n\n3. **Word2Vec Embeddings:** For Graph construction, we use existing Word2Vec Embeddings pre-trained using Google-News Corpus of 100 billion words.\n    \n4. **MUSE:** Yes, MUSE (Multi-lingual Unsupervised and Supervised Embeddings) provides the provision for multilingual embeddings (i.e., fastText embeddings aligned in a common space). However, MUSE learns mapping functions to map independently trained embeddings to a common space, which has severe limitations, e.g., \n\n    * It relies on the assumption of geometric similarity of underlying mono-lingual embeddings. \n    \n    * It performs poorly in domain-mismatch settings as shown by existing works [3, 5]. This is also stated in lines (32 to 51) of the paper.\n\n5. **New Representation:** The key idea is to surround each word (node) with a group of semantically related words (nodes in the graph) and perform weighted aggregation to re-compute the representation of the word. This will enforce \"lexically varying but semantically related words\" to get similar representations.\n    \n6. **Maths Notation:** The mathematical notation used in section 4.3.2 is inspired by earlier work on Graph Convolution by Kipf and Welling [2] with very few additions. We will work out the ambiguous parts to make it more clear.\n    \n7. **Encircled dot:** The encircled dotted is used to represent the Hadamard product, i.e., the element-wise product of the attention matrix $\\zeta$ and the adjacency matrix A.\n    \n8. **$\\eta_{ij}$:** $\\eta_{ij}$ is a typo. It has been corrected and replaced with $\\zeta$ to represent the attention matrix, and $\\zeta_{ij}$ to represent the attention weight between the words $w_i$ and $w_j$. \n    \n    * To re-emphasize, we use a multi-layer network with the following layer-wise propagation mechanism to compute the representation for layer $L^{(i+1)}$ given $L^{(i)}$: $L^{(i+1)} = \\rho(\\widetilde{(\\zeta \\odot A)} L^{(i)} W_{i})$, \n    \n    * Here $\\rho$ and $W_{i}$ are the non-linearity and weight matrix respectively. $L^{(i)}$ is the representation for the i-th layer. \n    \n    * $\\widetilde{\\zeta_{} \\odot A}= \\bar{D}^{-1/2} ((\\zeta_{} \\odot A) + I) \\bar{D}^{-1/2}$ is the normalized symmetric matrix with $\\bar{D}$ as the degree matrix of $(\\zeta_{} \\odot A)$. Details about the normalization trick are available in [2]. We will make it more clear in the paper.\n\n9. **$L_{ISO}$:** We use three different variations for $L_{ISO}$ (explained in line 331 of the paper), i.e., $L_2$, $L_{porc}$, and $L_{{porc}_{init}}$. These are explained in detail in the Section 4.4 of the draft.\n\n10. **Line 402:** It states that the process flow of the Vecmap toolkit [4] for mapping across different spaces is explained in Section 3.2.\n\n### Improvements + Typos:\n\n1. We have updated the word bi-lingual to \"bilingual\" throughout the document.\n\n2. We have replaced all instances of  \"Isovec\" with its correct capitalization: \"IsoVec\". \n\n3. Currently, we discuss examples of lexically different but semantically similar words, {i.e., \"terrible\", \"horrible\", \"awful\" etc.,} in Section 4.3. We will also add a solid and concrete example in the introduction along with the essence of modeling them using a graph. Thanks for mentioning it.\n\n4. Line 152 -- Marchisio's citation has been updated to Marchisio et al. (2022).\n\n5. For the final draft, we will add more details for a better explanation of Fig 1, i.e., skip-gram followed by Attentive Graph Convolutions and finally combining skip-gram and isomorphism loss. We will make the overall workflow of GRI more explicit and self-explainable.\n\n6. Line 181 explains the process flow of the Vecmap toolkit [4]. Dimensionality reduction is an optional step in Vecmap. In our case, it does not significantly impact the final results. We will omit this step from the final draft of the paper.\n\n7. Line 196: the word \"extend\" has been updated to \"extent\".\n    \n\n\n### References: \n\n1. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. \"Distributed Representations of Words and Phrases and their Compositionality.\" NIPS (2013).\n\n2. Kipf, Thomas N., and Max Welling. \"Semi-supervised classification with graph convolutional networks.\" ICLR (2016).\n\n3. Anders Søgaard, Sebastian Ruder, and Ivan Vulic. \"On the limitations of unsupervised bilingual dictionary induction.\" ACL (2018).\n\n4. Artetxe, M., Labaka, G., and Agirre, E. \"Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations.\" AAAI (2018).\n\n5. Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. \"How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions.\" ACL (2019).\n\n6. Kelly Marchisio, Neha Verma, Kevin Duh, and Philipp Koehn. \"IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces.\" EMNLP (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Marchisio et al. (2022) called for the community to create additional loss functions to improve the relative isomorphism of word embedding spaces, and this is one such metric.  The performance vs. the aforementioned work are strong. ", "reasons_to_reject": "My biggest criticism is that I do not feel that the method is explained clearly and concisely.  The explanation in 4.3 seems obscured by unnecessary and unclear math with non-standard notation.  I would be more enthusiastic if the method were explained clearly and concisely, as it is difficult to tell what exactly is happening.  ", "questions_to_authors": "Lines ~240-244 - why does semantic relatedness imply that graphs are a natural choice?\n4.3.1 - did you implement SkipGram yourself?\nWhich retrained word2vec embedding did you use?  Note that MUSE is fasttext.\n“We aggregate the information contained in the neighbors to come up with a new representation for the word that accommodates information from semantically related neighbors” - how?\nSection 4.3.2 - the math notation is confusing. What is the encircled dot?  What is n_ij?  I recommend explaining only what is necessary, with no jargon or unnecessary equations, as these obscure what you are doing. \n(6) - I do not see that L_iso is defined.  \nLine 402 - “process flow”?", "paper_topic_and_main_contributions": "The authors propose a new graph isomorphism loss function that is used in combination with the skip gram loss function to improve the relative isomorphism of static word embedding spaces, therefore improving their ability to be mapped to a shared cross-lingual space as measured by P@1.  This work extends Marchisio et al., (2022) (IsoVec) to a new loss function.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Bi-lingual is typically written as “bilingual”\nIsoVec is the correct captialization\nA concrete example in the intro of what is meant by “lexically different but semantically similar” would be helpful\nLine 152 is missing a year for the Marchisio citation\nFigure 1 could be better explained\nline 181 - I actually do not think that dimensionality reduction happens happens.  I would check the implementation you can. \nLine 196: extend->extent\n"}}, {"review_id": "V5B2BClRIm", "reviewer": "Reviewer_spUb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We will consider these points while preparing the final draft of the paper. The response for each individual point raised in the paper is as follows:\n\n### Contributions of GRI:\n\n1. The main contribution of GRI is to propose Attentive Graph Convolutions to incorporate the impact of \"lexically varying but semantically similar words\" on Bilingual Lexical Induction (BLI).\n\n2. In order to better highlight the main contribution of the paper, we will additional intuitive explanations throughout the paper. We will also add concrete examples in the introduction (in addition to the current example in Section 4.2) to emphasize the impact of Attentive Graph Convolutions compared to existing BLI.\n\n### Reproducibility:\n    \n1. All the methodologies developed in the paper and corresponding results have been computed using freely available public data resources, as stated in Section 5.1 of the paper. These results are easily reproducible. Moreover, upon acceptance, we will make all the codes and data sets publicly available to ensure easy reproducibility of the results.\n\n2. All results reported in the paper are averaged over 5 runs of the experiments as stated in lines 403 to 406 of the paper. \n\n### Typos:\n\n1. All the bibliographic references including (Søgaard et al., (2018), Conneau et al., (2017)) have been corrected to remove duplicate entries. \n\n2. All the arxiv references have been replaced by corresponding peer-reviewed versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contribution of this paper seems to be significant in that the GRI method is proposed, leading to improvements in bilingual dictionary creation and thus in machine translation. The paper address an important problem in machine translation and demonstrates a performance improvement via P@1 of 63.6%. The authors also demonstrate that these results hold in domain mismatch settings, which is a topical concern.", "reasons_to_reject": "The methodology and contributions were not very clear to me, especially when comparing this method against straight BLI, but that is in part because this is not my area. It is also not clear whether these results are very reproducible and how many runs were performed for each set of results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work addresses the challenge of building bilingual dictionaries for machine translation, which is  affected by the alignment of semantically related terms in each language. Prior methods have effectively combined the geometric spaces across languages but have not considered the impact of semantically related terms. This work thus proposes a method called GRI (Graph-based Relative Isomorphism), which is supposed to account for the impact of semantically similar terms using attentive graph convolutions. This is enables effective computation of the relative isomorphism of multiple geometric spaces. The results indicate that the GRI performs existing methods by using the average precision at rank 1 measure, demonstrating that GRI leads to more accurate alignment between terms of different languages.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Incorrect citation for Søgaard 2018-- the same paper appears twice in the Bibliography (once as 2018a and once as 2018b)\n\nIncorrect citation for Conneau 2017-- the same paper appears twice in the Bibliography (once as 2017a and once as 2017b)\n\nMany papers which were cited as arxiv papers have since been peer-reviewed and published; cite the peer-reviewed versions"}}]

Author rebuttal:

Meta-review: This paper proposes a graph convolution approach for the Bilingual Lexicon Induction task. Overall, the reviewers found that the paper presents interesting observations and promising experimental results. Strong experimental results over comparable and the latest method in the field of bilingual lexical induction with the graph isomorphism-inspired method such as IsoVec (Marchiso et al. 2022). However, multiple reviewers pointed out concerns about the clarity of the submission (e.g., math notation [1, 2], lack of definitions of some phrases). The AC would also strongly encourage adding technical details and motivation for selecting components (e.g., why using Graph Convolutional Network but not other graph NN models) but make the paper clear and concise. The AC strongly recommends addressing detailed comments made by Reviewer AjeL to improve the next version of the paper for future readers.


[1] In addition to what reviewers recommended for increasing the clarity on math notations, the AC would also suggest using different styles for loss $L$ vs. layer $L$ e.g., by using $\mathcal{L}$. These small changes pile up and save a lot of mental effort for future readers. 

[2] E.g., the definition of $L_{ISO}$ is not clear from the main text, and adding the explanations mentioned in the rebuttal https://openreview.net/forum?id=IsDxBXUEd8&noteId=FV6vFXbdKa would definitely benefit future readers to easily understand the paper.
Decision: Accept-Findings