A Structure-Aware Generative Adversarial Network for Bilingual Lexicon Induction
Authors: Bocheng Han
Qian Tao
Lusi Li
Zhihao Xiong
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-lingual word embedding
unsupervised
low isomorphic
bilingual lexicon induction

Abstract: Bilingual lexicon induction (BLI) is the task of inducing word translations with a learned mapping function that aligns monolingual word embedding spaces in two different languages. However, most previous methods treat word embeddings as isolated entities and fail to jointly consider both the intra-space and inter-space topological relations between words.  This limitation makes it challenging to align words from embedding spaces with distinct topological structures, especially when the assumption of isomorphism may not hold. To this end, we propose a novel approach called the Structure-Aware Generative Adversarial Network (SA-GAN) model to explicitly capture multiple topological structure information to achieve accurate BLI. Our model first incorporates two lightweight graph convolutional networks (GCNs) to leverage intra-space topological correlations between words for generating source and target embeddings. We then employ a GAN model to explore inter-space topological structures by learning a global mapping function that initially maps the source embeddings to the target embedding space. To further align the coarse-grained structures, we develop a pair-wised local mapping (PLM) strategy that enables word-specific transformations in an unsupervised manner. Extensive experiments conducted on public datasets, including languages with both distant and close etymological relationships, demonstrate the effectiveness of our proposed SA-GAN model.

A Structure-Aware Generative Adversarial Network for Bilingual Lexicon Induction
Bocheng Han Qian Tao'; Lusi Li², and Zhihao Xiong 'oouth China University of Technology, Guangzhou, China 'Old Dominion University, Norfolk, USA Baidu Inc., Beijing, China
202221045875eman scut. edu. cn, an edu. cn lusiliecs.oo edu, xiongzhilnao@aiddu. com
Abstract
been successfully applied in various NLP applica- tions, including machine translation (Artetxe et al., 2018c; Ren et al., 2020), cross- cooss-lingual sentiment analysis (Singh and Lefever, 2020) and text classi- fication (Dong and de Melo, 2019).
Bilingual lexicon induction (BLJ) is the task of inducing word translations with learned mapping function that aligns monolingual word embedding spaces in two different languages. However, most previous methods treat word embeddings isolated entities and fail to jointly consider both the intra-space and inter- space topological relations between words. This limitation makes it challenging to align words from embedding spaces with distinct topological structures, especially when the as- sumption of isomorphism may not hold. To this end, we propose a novel approach called the Stracture-Aware Generative Adversarial Net- work (SA-GAN) model to explicitly capture multiple topological structure information to achieve accurate BLJ. Our model first incorpo- rates two lightweight graph convolutional net- works (GCNs) leverage intra-space topologi- cal correlations between words for generating source and target embeddings. We then employ a GAN model to explore inter-space topolog- ical structures by learning a global mapping function that initially maps the source embed- dings the target embedding space. To further align the coarse-28 grained structures, we develop a pairwise local mapping (PLM) strategy that enables word-specific transformations in supervised manner. Extensive experiments con- ducted on public datasets, including languages with both distant and close etymological rela- tionships, demonstrate the effectiveness of our proposed SA-GAN model.'
Most BLI methods aim to learn a mapping func- tion that aligns word embeddings of two languages into shared embedding space, which allows lever- aging independently trained monolingual embed- dings and then utilizing the learned mapping generate bilingual lexicons (Mikolov al., 2013; Glavaš al., 2019). Thereinto, Mikolov al. (2013) first observed that linear orthogonal map- ping proved to be empirically effective in trans- forming the source embedding space to the tar- get language's space. This mapping was achieved by minimizing the squared Euclidean distance be- tween the translation pairs in given parallel vocab- ulary. They attribute the success of their method to the isomorphic assumption that the two embed- ding spaces exhibit similar geometric structures as they found that the linear projection outperformed its non-linear counterpart with multilayer neural networks. Building upon this work, various BLI methods have been proposed to improve the in- ductive performance by enforcing an orthogonality constraint (Lample al., 2018), normalizing the embeddings (Artetxe al., 201 2018a), relaxing the isomorphic assumption (Patra et al., 2019), leverag- ing the clique-level information (Ren et al., 2020), refining with Coherent Point Drift algorithm (Cao and Zhao, 2018; Oprea et al., 2022), distinguishing the relative orders (Tian et al., 2022), etc. From them, it can be noticed that reliable mapping func- tions can be learned even with weak supervision.
1 Introduction
Bilingual lexicon induction (BLI) has emerged as a crucial task in natural language processing (NLP), focusing on the discovery of corresponding words between two languages using monolingual corpora. Due to its ability to facilitate the transfer of se- mantic knowledge between languages, BLI has
Furthermore, recent advancements have intro- duced several unsupervised models through adver- sarial training learn mapping functions with- out the need for parallel data (Lample et al., 2018; Bai et al., 2019; Mohiuddin and Joty, 2019; Xiong and Tao, 2021), offering a data-driven, seal-
*Corresponding author
'Our code is available on httpps//ggite hht.///gttttu..... com/ SCUtBCH/SAGAN

---

able, and language-independent approach to induce cross-lingual representations from low-resource languages. However, existing adversarial methods focus on word-level alignment and treat the words in the embedding space as isolated entities, ignor- ing the underlying topological structures among words. Therefore, the relationship between words is not preserved and the topologiaal structure of the embedding spaces is not well exploited or involved during training, leading to poor performance com- pared with other non-adversarial methods (Artetxe et al., 2018b; Ren et al., 2020).
enforce an orthogonality constraint on the map- ping function, SA-GAN removes this constraint during adversarial training since the isomorphic assumption may not hold true practically. The learned mapping matrix facilitates the construction of a seed dictionary. To further refine the coarse- grained structures and enhance the seed dictionary, SA-GAN introduces a pairwise local mapping al- gorithm (PLM). This algorithm can learn word- specific transformations for different words based on their nearest neighbors within the seed dietio- nary. By doing so, our method reduces reliance on isometry and achieves improved BLI perfor- mance in fully unsupervised manner. To verify the effectiveness of SA-GAN, we conduct exten- sive experiments with sixteen different language pairs, comprising both etymologically distant and close languages to thoroughly test our model per- formance with varying degrees of isomorphism between monolingual spaces. Experimental results show that our model can achieve comparable per- formance to state-of-the-art unsupervised methods in most cases and even surpass previous supervised ones. Our main contributions can be summarized as follows:
In addition, conventional BLI methods typically assume that the embedding spaces of different lan- guages are nearly isomorphic, and they learn a global linear mapping function shared by all words based on this assumption. However, recent stud- ies (Søgaard et al., 2018 8; Patra et al., 2019) have found that the isomorphic assumption may not hold strictly due deviations in the distributions of word embeddings for different languages. Conse- quently, the performance of BLI methods might be degraded, especially for the language pairs far from isometry. In such a case, a globally-aligned mapping function may not be an optimal solution. There have been some approaches that attempted to alleviate this problem by learning personalized mapping functions for different words employ- ing supervised non-linear mapping latent space (Glavaš and Vulić, 2020; Tian et al., 2022; Mohiud- din et al., 2020). However, the supervised signals are all indispensable to their proposals and can- not be applied to the unsupervised learning setting without any labeled data.
We develop novel adversarial framework SA- GAN to explore both the intra-space and inter- space topological information for unsuper- vised BLI. It integrates two GCNs and a GAN to learn a linear mapping function through adversarial training without imposing an thogonality constraint, providing greater flex- ibility in aligning different languages where the isomrrphic assumption may not hold.
To address these challenges, we propose a novel unsupervised model called structure-aware gener- ative adversarial network (SA-GAN) to explicitly capture multiple topological structure information for accurate BLI. Specifically, given source lan- guage and target language, SA-GAN first views the embedding space of each language as a graph and utilizes two lightweight graph convolutional networks (GCNs) to encode two embeddings for ex- ploring the intra-space topological structures. With the extracted structural information, we formulate the learning of mapping function in fashion that admits an adversarial game. SA-GAN employs GAN model to learn linear mapping matrix, allowing for the global mapping of the extracted source embeddings into the target embedding space. Unlike previous adversarial methods that usually
We propose pairwise local mapping (PLM) algorithm, which enables the learning of word- specific transformations. PLM utilizes topo- logical information from the nearest neighbors in the seed dictionary refine the alignments and alleviate the reliance on isometry.
We conduct extensive experiments over popu- lar benchmarks, and the results demonstrate that our model outperforms existing unsuper- vised methods and even outperforms super- vised state-of- ofte-of-thee-ar -te--att methods.
2 Methdology
In this paper, we denote the source and target language word embeddings as X E dddx and

---

X
GCN
D
0/1
GCN
P
anchor pars
[talln-Lnngaa other words
structure extraction
adversarial training
pairwise local mapping
Figure I: An overview of our proposed SA-GA SA-GAN framework.
Y E Rdxmm where and m are the numbers words and Y, respectively, and d stands for the embedding size. Our proposed SA-GAN method consists of three major components, includ- ing structure extraction, adversarial training, and pairwise local mapping, as shown in Figure 1. Each module has its role to play while targeting different goals. By the strategy of splitting, each module can focus more on its task and improve the overall performance while reducing the complexity, Specifically, given two monolingual embeddings X and Y. we first capture the topological infor- mation of each language via two lightweight GCN modules. After that, a global mapping matrix is learned via adversarial training, which transforms the source word embeddings into the target em- bedding space. Finally, SA-GAN designs novel PLM algorithm to learn word-specific transforma- tion, which alleviates the reliance on isometry. We will next formally introduce the model.
matrix to describe the similarities between the words i and vy in the graph.
In |xi Xyl12,
i J =
Aij 0,
(1)
where x; and x, are the word embeddings for node 0y and node vj, respectively.
The basic idea of GNN is to learn node repre- sentations in graph by incorporating information from neighboring nodes through iterative aggre- gation and transformation processes. During the aggregation process, the neighhoring node features are aggregated to generate combined represen- tation for each node. During the transformation process, the combined representations undergo a transformation to generate refined node represen- tations using neural network layers for capturing more complex topological relationships. A well- known example of traditional GNN is the Graph Convolutional Network (GCN) (Kipf and Welling, 2017). GCN leverages convolutional layers the graph structure to perform neighborhood aggrega- tion and transformation as follows.
2.1 Structure Extraction
Recently, graph neural networks (GNNs) have been widely utilized in various fields due to their pow- erful ability to extract spatial information from graphs. Inspired by this, we propose to incorporate a GNN module prior to adversarial training to ex- ploit the topological correlations in the embedding spaces by viewing the entire embedding space as graph. In this graph, each word is represented as node and edges connect it to its k-nearest neighbors. The graph can be denoted as G (V, E, A), where V {uI.... 2n represents nodes and n is the total number of vocabulary words in one language; {ei.)" is a set of edges, where each edge Gijj is associated with a weight A in adjacency
x -xAX((-))
(2)
ー 1 (A+I)D
where (1) and (l-1) denote the embedding rep- resentations after / and (2 - 1) layers propagation for all the nodes and (0) X; Á is the nor- malized and regularized adjacency matrix; I is identity matrix. which is added connections; D a diagonal node-degree matrix. (')) is the feature transformation matrix at the I-th layer and a(.) is an activation function. However,
include self-

---

the full gradient descent strategy is often used to train GCN, suffering from high computational com- plexity for large-scale datasets. Hence difficult to fit in with subsequent adversarial training, where mini-batch stochastic gradient descent (SGD) is used for each update. Some rescarchers (Hamilton et al., 2017) have proposed mini-batch SGD for GCN to alleviate the problem, but the overheads of these methods are still large.
Specifically, we train generator G to learn a linear mapping matrix W deceive discrimi- nator D. The generator G aims to map the word embeddings from the source language to the target language through ((²²) = W G can be trained with the loss function as follows:
Lgp
Σ log(D(Wa,))
(7)
#cC
Motivated by He et (2020), we propose a simplified GCN by removing the activation func- tion (.) and the feature transformation matrices { [W Wh defined as follows:
The discriminator is trained to distinguish between the mapped source embeddings W W (W21,.., Win and the target embeddings Y {u1, .... Fm using the cross-entropy loss:
X (x
(4)
DDG- m
Σ
log (
(8)
D=
(5)
Σ
log(1 D(WA))
n
Furthermore, to reduce the computation time, we construct a ne-nearest graph to preserve the edge connections of the top K, nearest neighbors for each node and keep the adjacency matrix A as a sparse matrix. Lastly, we combine the embeddings obtained at each layer to produce the final embed- ding matrix:
2,
At each iteration, we optimize the generator loss (Equation (7)) and the discriminator loss (Equation (8) alternately with stochastic gradient updates. Through adversarial training, we can obtain an ini- tial solution of W. Following other GAN-based methods (Lample et al., 2018; Bai et al., 2019; Xiong and Tao, 2021), we further refine the learned mapping W via self-learning strategy in (Artetxe et al., 2018b) by iteratively solving the Procrustes problem and applying a dictionary induction step. In our self-learning, we run five iterations of this process.
X = ar xxx + tajx +... x()
(6)
where {ar)/e are the tradeoff coefficients. It is worth noting that there are trainable pa- rameters of the designed GCN module. In other words, rather than training the propagation process at each iteration, the final embedding matrix only needs to be precomputed once and can be stored as a constant, which greatly decreases the computa- tional cost and memory resource requirements.
Although the word embeddings and Ý contain the structure information using GCNs, they also introduce challenge known as over- smoothing (Li et al., 2018). This issue arises when the words become indistinguishable from each other, especially those words lying in dense areas, leading to poorer performance when induc- ing the bilingual lexicon. To address this concern, we utilize X and Y for finding the initial solution W. Subsequently, we discard X and Y, and the re- maining processes, including self-learning and the PLM algorithm (Section 2.3), are executed using the original embeddings X and Y. This decision is made mitigate the over-smoothing problem and ensure that subsequent steps operate on the unaltered embeddings, thus potentially improving the performance of bilingual lexicon induction.
Two GCN modules are respectively applied to the source language and the target languages Y to form the new embedding representations X with n nodes and Y with m nodes, which contain the topological structure information of the source and target embedding spaces.
2.2 Adversarial Training
With the extracted embedding representations, our goal is to match them for inducing seed dictionary. Recent studies have demonstrated the effectiveness of adversarial training in aligning two distributions (Lample et al., 2018: Xiong and Tao, 2021). Build- ing upon this concept, we employ adversarial train- ing through a GAN in our work to learn mapping function in a fully unsupervised manner.
2.3 Pairwise Local Mapping Algorithm
With structure extraction and adversarial training, we can capture valuable structural information and

---

learn a mapping function that shared globally by all words under the isomorphic assumption. How- ever, several studies (Ruder et al., 2019; Patra et al., 2019) have found that this assumption is not strictly applicable, and it may lead to poor performance in BLI, particularly for language pairs that devi- ate significantly from isometry. In this situation, a global-shared mapping function may not be the optimal solution. To further refine the alignments, we introduce novel PLM algorithm recompute and upgrade the embedding representations for dif- ferent words based on the learned seed dictionary and improve the BLI performance.
that indicates the importance of anchor word 2 in MV to the given source word z, The closer an anchor is to 2;, the larger the importance coefficient gets. However, since the cosine similarity ranges from O and I, we observe that even the anchors that are too far to give useful guideline still get high coefficient, ie. 0.4, which will introduce potential noises to the pairwise mapping. To avoid the influence, we scale the importance using the softmax function with temperature т:
exp(es//T)
aij softmax,(ews) EKEN, exp(eit/T)
(12)
Our PLM algorithm consists of two steps: gen- erating seed dictionary D(Z Y and then uti- lizing the word pairs in this synthetic dictionary to perform a local mapping for each word. Firstly, we induce seed dictionary utilizing the learned map- ping matrix to map the source word embeddings to the target embedding space as follows:
which increases the influence of the nearest neigh- bor anchors even further and decreases for the dis- tant ones. We then compute the new embedding representation of Z₁ with the guidelines of gener- ated dictionary, as follows:
PLM(2,) = z; +
Σ
ask - lyf - -z2) (13)
RENN
Z = WX
where is the rate for updating the word embed- dings.
where Z is the mapped source word representations. With 2 and Y, we can retrieve the translation pairs and build the seed dictionary DZZ YD accord- ing to the cross-domaie similarity local scaling (CSLS) measurement (Lample al., 2018). Specif- ically, given a mapped source word z, we treat the nearest word in the target embedding space as the translation results as
The above steps can be iteratively done for both directions and each iteration, we regenerate the dictionary D with the updated embedding represen- tation the previous iteration to further improve the quality of the synthetic dictionary.
2.4 Training Paradigm
In summary, the proposed approach first extracts the structure information using GCNs and learns a global mapping function in an adversarial manner to map the embeddings of two languages into the same space. In order alleviate reliance on isom- etry, we further apply the PLM algorithm to learn pairwise mapping functions for different words based on the learned seed dictionary. The whole training process of the proposed approach is unsu- pervised and described in Algorithm 1.
CSLS(z, y) = 2 co0s(2,)) rT(Z) - rrs(y) (10)
where (z) is the average cosine similarity be- tween z and its k-nearest neighbors in ryy) is the average cosine similarity between and its k-nearest neighbors in Z. To refine the quality of the dictionary, filter out word pairs in the gener- ated dictionary that are not the Km most frequency words in each language which are usually of low quality and induce word pairs from both directions in the seed dictionary DZz Y
3 Experiment
Secondly, we use the word pairs in this synthetic dictionary to improve the mapped embedding and get pair-wiss local mapping for each word. Given a mapped source word zı, we first obtain its top K.- -earestt neighbor words zp ..., z from D as anchors, denoted as Nr with a coefficient for each anchor point:
3.1 Experimental Settings
Following the common practice of BLI, we eval- uate the performance of induced language pairs with the Precision at 1 (Precision @ 1) metric, which measures the word translation accuracy in compar- ison to gold standard.
Dataset To demonstrate the effectiveness of our (11) SA-GAN model, we leverage the widely used
Gjj = cos(Z,, 2D)

---

Algorithm 1: Training procedure of model
and unsupervised methods (Lample et al., 2018: Artetxe al., 2018b; Bai al., 2019; Mohiuddin and Joty, 2019; Ren et al., 2020; Xiong and Tao, 2021). Please refer to Appendix A for the details. For each baseline model, we report the results the original papers and conduct experiments with the publicly available code if necessary.
Data: Normalized monolingual word embeddings X for source language and Y for target language Build adjacency matrix according Eq.I:
Extract structural information following Eq.4 and get
new embedding representation X and Y:
/* Learn global shared transformation */ for r_epochs do for r_iteraation do for steps do
Implementation details Following previous work, vocabularies of each language are trimmed to the most frequent 200k word embeddings for eval- uation, same for the graph generation in section 2.1. The adversarial model uses 75k most frequent words in each language to feed the discriminator. The original word embeddings are normalized fol- lowing (Artetxe et al., 2018b), including length normalization, center normalization and length nor- malization again to ensure the word embeddings have unit length. The generator G' is a single linear layer. The discriminator is a multilayer per- ceptron with two hidden layers of size 2048 and Leaky-ReLU activation functions. We train our models using stochastic gradient descent (SGD), with a batch size of 32, and a learning rate of 0.1. A smoothing coefficient s = 0.1 is added to the dis- criminator predictions. We train the discriminator more frequently (5 times) than the generator. For the PLM algorithm, the temperature is set to 0.1; the updating rate is set to 0.02; the vocabulary most frequency Km is set to 20,000 in the synthetic dictionary; the number of neighbor words as an- chors K₂ is set to 150; the number of iterations is 10.
Sample a batch from X and Y Update discriminator with Eq.8;
end
Sample a batch from X and Y
10
Update generator W on adversarial loss to fool discriminator with Eq.7;
end
Use validation criterion to save the best model;
13
14 Refine the learned mapping via self-learning
process;
1S Map the source embeddings x to the target embeddings space: Z <- WX;
/* Learn word-specific transformation
*/
16 for m_iteraaions do
17
Build a synthttic dictionary between Z and Y Calculate new embedding representation for each word in Z according to 13
18
19
Build a synthetic dictionary between Z and Y Calculate new embedding representation for cach
20
word in Y according to 13;
end
MUSE dataset (Lample et al., 2018). The Muse dataset consists of monolingual fastText (Bo- janowski et al., 2017) embeddings of 300 dimen- sions trained Wikipedia monolingual corpus and dictionaries for 110 language pairs. According to (Patra et al., 2019), etymologically close lan- guage pairs have lower Gromov Hausdorff (GH) distance compared to etymologically distant lan- guages. Therefore, we evaluate English (En) from/to 4 etymologically close languages with low GH distance: Spanish (Es), French (Fr), Italian (It), and German (De); and 4 etymologically distant languages: Russian (Ru), Chinese (Zh), Hungar- ian (Hu), and Danish (Da) with high GH distance. Comparing both etymologically close and distant language pairs, we can thoroughly test our model performance with varying degrees of isomorphism between monolingual spaces.
3.2 Experimental Results
We report the BLI performance over four etymolog- ically close language pairs(en-es, en-fr, en-it, and en-de) and four etymologically distant pairs (en- ru, en-da, en-hu, en-zh) from the MUSE dataset. The results are presented in Table 1. For our ap- proach, we map the embedding representations of the source language (English) into target embed- ding space (other languages) and evaluate the per- formance of our model in both directions with the corresponding test datasets. should be noted that all results reported in the paper are an average of 5 runs. The 'NA' indicates the authors did not report the number or their code is not publicly available, and *** indicates that the methods fail to converge. Table shows the Gromov-HHusdood (GH) dis- tance the selected language pairs. From the measurements, we can see that etymologically
Baselines We compare
proposed SA- GAN with several SOTA BLI baselines, includ- ing supervised/semi-suprrvisd methods (Lample et al., 2018; Artetxe et al., 201 8a; Jawanpuria et al., 2019; Glavaš and Vulić, 2020; Mohiuddin et al., 2020; Ganesan et al., 2021; Tian et al., 2022),

---

En-Es
En-Fr
En-Ru
En-Da
En-Hu
En-2h
GH Distance
0.31
0.24
0.35
0.29
0.40
0.44
0.40
0.83
Sup/Semt-superree super-sided
(Lample al., 2018)
81.4
81.1
77.3
72.6
63.7
56.3
53.3
42.7
36.7
(Artetxe al. 2018a)
81.9
83.4
82.1
82.4
77.4
77.9
73.5
73.5
50.5
67.3
64.1
56.1
67.7
32.3
43.4
(Jawanpurii et 2019)
81.9
85.5
82.1
84.2
77.8
80.9
74.9
76.7
52.8
67.6
63.1
72.6
57
69.5
49.1
45.3
(Glavoš and Vulié, 2020)
82.4
86.3
84.5
84.9
80.2
81.
76.5
27.5
57
67.1
59,4
70
55.2
70.1
47.9
47.2
(Mohiuddin et al., 2020)
82.9
86.4
82.7
84.2
78.1
81.4
75.5
75.9
52.3
67.8
60.9
70.5
57.5
66.9
42.9
42.0
(Ganesan et al., 2021)
83.1
83.3
83.7
82.9
78.6
76.1
76.1
74.7
55.8
68.7
NA
NA
NA
NA
NA
NA
(Tian al., 2022)
84.1
86.1
83.5
84.3
79.3
81.9
76.5
72.9
58.1
68
59.3
69.1
57.5
65.9
51.7
45.9
Unsupervased
(.amll al., 2018)
81.7
82.3
77.4
76.
72.2
59.1
57.5
53.5
63.7
32.5
31.4
(Artetxc al. 2018b)
82.3
84.7
82.3
83.6
78.8
79.5
74.1
65.5
64.5
67.9
56.4
67
37,5
37.8
(Bai 2019)
82.3
84.3
82.5
83.7
78.4
77.9
74.9
73.5
65.8
57.7
64.6
52.5
43,4
36.7
(Mohiuddin and Joty. 2019)
82.7
84.7
82.8
83.7
79
79.6
75.4
74.3
46,9
64.7
64.5
66.8
56.1
(Xiong and Tao, 2021)
82.8
83.9
82.5
82.3
78.6
77.9
75.3
77.9
47
63.2
58.3
64.6
52.3
63.1
37,8
35
(Ren et al., 2020)
82,9
85.3
82.9
83.9
79.1
79.9
75.3
76.1
49.7
64.7
NA
NA
NA
NA
38.9
35.9
(Oprea et al,, 2022)
83.3
85.4
83.4
84.1
NA
NA
75.8
75.8
49.5
64.0
NA
NA
NA
NA
NA
NA
SA-GAN without PI.M
83.1
85.3
82.9
84
80.2
79.9
76.9
74.7
49.7
64.9
67
70.6
59.8
70.1
41.5
40
SA-GAN
84.2
85.4
83.1
84.4
81.4
80.4
77.5
76.1
50.9
66.2
69.1
71.9
60.5
71.4
41.6
42.1
Table 1: Word translation accuracy (Precision @ 1) on MUSE dataset.
For each metric, underline marks the highest accuracy among all approaches; bold marks the best performance across all unsupervised methods; 'NA' indicates the authors did not report the number their code is not available; *** indicates that the methods fail converge.
close language pairs have lower GH distances com- pared to etymologically distant ones. We com- pare SA-GAN with both existing unsupervised and semi/supervised approachs.. From Table 1, one can clearly see that our proposed method significantly outperforms previous unsupervised methods over most language pairs, and also obtain comparable performance on the rest. Compared with state-of- the-art unsupervised methods, SA-GAN performs better on 14 of 16 language pairs, especially on en- it and en-de with the absolute improvements of 2% to 2.3%, and on ctymologically distant language pairs like en-hu and en-da, with absolute improve- ments of 4.1 % 4.6% over the best baseline.
form poorly or even fail to converge. This is rea- sonable as we capture much richer semantics by extracting the structure of embedding space with the GNN module, which helps learn better map- ping function compared with other methods. More- over, since the distributions of different languages deviate and the isomorphic assumption may not be strietly held (Patra et al., 2019; Søgaard et al., 2018), a global-share mapping is not the optimal solution (Tian et al., 2022). In this situation, an un- supervised PLM algorithm is applied to every word to get personalized mappings, which improves the performance further.
En-Fi
En-He
En-Ro
Avg
Furthermore, compared with the supervised methods, SA-GAN can still achieve competi- tive results and even outperform existing state- of-the-art supervised methods on some language pairs. The performance of our approach on en- it is 81.4%, compared to 80.2% with the best- supervised method. On en-hu, our SA-GAN ob- tains 60.5%, which is 3% better than the super- vised method. Such performance gains demon- strate the superiority of SA-GAN. From table 1, we also find that leveraging the unsupervised pairwise local mapping (PLM) contributes to bilingual lexi- con induction, with a gain of 0.7% on average on etymologically close language pairs and 1.2% on distant language pairs, which is remarkable.
Methods
+-
+-
MUSE
43.7
53.7
38
fail
58.0
66.0
43.2
VECMAP
49.9
63.5
44.6
64.2
71.8
58.6
Adv-M
49.8
65.5
46.1
58.6
62.6
71.9
59.1
Adv-O
49.9
65.5
46.6
59.1
65.4
74.3
60.1
w/o PLM
52.3
64.9
47.1
57.5
66.2
72.3
60.1
SA-GAN
52.7
66.0
48.0
59.7
66.4
72.7
60.9
Table 2: Word translation accuracy (Precision 1) of morphologically rich languages on MUSE dataset. Bold marks the best performance across all methods.
3.3 Results of Morphologically Rich Languages
To better explore our model's robustness, we fur- ther evaluate our method on "difficult" morphologi- cally rich languages, where unsupervised bilingual dictionary induction performs much worse (Sø- gaard et al., 2018). Following Oprea et al. (2022),
From the results, we note that SA-GAN achieves more improvements in etymologically distant lan- guages, where other unsupervised baselines per-

---

we evaluate English (En) from/to 3 morphologi- cally rich languages like Finnish(Fi), Hebr)w(He), and Romanian(Ro), a mixture of isolating or ex- clusively concatenating languages from a morpho- logical point of view (Søgaard et al., 2018). Since Lample et al. (2018); Artetxe et al. (2018c), 8c); Mohi- uddin and Joty (2019); Oprea et al. (2022) consis- tently perform well, they are selected as baselines for the remaining experiments, denoted MUSE, VECMAP, Adv-M and Adv-O respectively. The results are shown in Table 2. From the measure- ments, wc can see that our approach outperforms existing methods on of 6 tasks on morphologi- cally rich language pairs, with gain up to 2.8% on en-fi and 0.8% on average of all languages, which further shows the robustness and effectiveness of our framework.
Setting
co-10
SA-GAN
80.2
79,9
76.9
74.7
49.7
64.9
41.5
40
who GNNN
80.0
79.5
76.5
74,4
Fail
fail
fail
fail
constraint
80.3
79.7
76.7
74.4
47.1
63.3
Fail
fail
Table 3: Ablation study on adversarial training.
Setting
en
en
boseline
80.2
79.9
76.9
74.7
49.7
64.9
41.5
40
SA-GAN
81.4
80.4
77.5
76.1
50.9
66.2
41.6
42.1
w/o Scale
80.4
80.1
76.6
75.2
49.6
65
40.8
40.5
w/o Cutoff
80.2
79.3
76.5
74,9
49.2
64.1
41.1
80.1
75.8
49.7
41.3
w/o p
68.2 63.6
65.6
56.6
31.8
39.9
28.7
32.2
Table 4: Ablation study on the pairwise local mapping. Bi: bidirectional forwarding; pi updating rate (set to when removed).
The obtained results are presented in Table 4. The baseline in the table is a variant of our approach without using PLM. From the table, we can find that the performance declines over all tasks after removing PLM, revealing the importance of per- sonalized local mappings. As for the different PLM components, we observe that coefficient scaling is necessary to avoid the potential noise introduced by anchor words. The dictionary frequency- based cutoff also has a positive influence on our model, with 1 1.2% gain in en-it and 1 1.3% gain in en-ru. At the same time, the updating rate plays a critical role in the systems. Without updating rate (p = 1). the model performance declines sharply due to overly drastic updates of embeddings. Bidirectional for- warding is also beneficial, which provides op- timal solution by mapping the source and target languages together to a latent space, rather than fixing one of them. In summary, every component of PLM is indispensable to achieving better perfor- mance.
3.4 Ablation Study
To further analyze our approach, we perform abla- tion studies and measure the contribution of each novel component that is proposed in this work. We conduct extensive ablation on translation tasks from 4 language pairs from the MUSE dataset, con- sisting of 2 etymologically close and 2 etymologi- cally distant languages.
Streture extraction and adversarial training Here we study the impact of the structure extrac- tion (GNN) module and orthogonality constraint. To avoid the influence of PL.M, ablation studies are investigated in the setting without the PLM mod- ule, shown in table 3. One can see that model performance will consistently drop in all language pairs and even fail converge in distance language pairs if we further remove the GNN module. After enforeing an orthogonality constraint, the perfor- mance pairs drop (eg. en-ru) and fail to converge (eg. en-zh) in the distant language pairs that are far from isometry. We can get the following conclu- sions: 1) the GNN module can capture much richer semantics by extracting structure information of embedding space, which contributes to learning better mapping function and stable the BLI perfor- mance; 2) strict orthogonality constraint limits the performance of language pairs that are etymo- logically distant and far from isometry.
3.5 Parameter Sensitivity Analysis
We further analyze the performance of PLM with respect to two core hyper- hyper-parameters. (1) the vo- cabulary cutoff with the most frequency Km for synthetic dictionary, and (2) the scaling temper- ature T in Formula 12. The sensitivity analysis is conducted on the en language pair on the MUSE dataset.
Pair-wise local mapping Here we aim to study the importance of the designed PLM algorithm, and the influence of updating rate p. coefficient scaling, dictionary frequency cutoff, and bidirec- tional forwarding components to the PLM module.
Frequency-based vocabulary cutoff The hyper- parameter Km denotes the number of most frequent words each language considered when inducing the synthetic dictionary. As shown in Figure 2(a), On the one hand, when Kn, is too small, the syn-

---

Limitations
11.4
11.7
$12
Although our approach can achieve impressive per- formance, there are still some limitations to be resolved in the future.
11.3
31.
00.5
60.8
60.4
50.6
00.7
20R
436
Edk
10x
100c
0.55
B.23 Temperature scaling
0:33
0.2
0.
SA-GAN requires tuning more hyper- parameters compared to previous methods, which is time-consuming.
Frequency cutoff Ky
(a) Performance Km
(h) Performance VS.
Figure 2: Paramesensittvity analysis.
SA-GAN matches source and target languages by mapping the source embeddings into the target embedding space, rather than mapping them into common latent space. While the performance relies on the target word embed- ding space, the mapping function might be sub-optimal.
thetic dictionary can't obtain enough information to guide the local mapping: On the other hand, when Km is too large, much noise will be intro- duced, which reduces the quality of the dictionary and declines the performance.
Temperature of scaling Figure 2(b) illustrates how the performance varies with different values for scaling temperature т. We can find that a small t helps to increase the influence of the nearest an- chors in the dictionary and decrease for the distant ones, which scales the importance coefficients fur- ther to provide useful guidelines and reduce the potential noise in the dictionary.
Additionally, SA-GAN focuses on aligning single-word embeddings, making it unsuit- able for directly applying to the alignment of multi-word expressions that encompass in- tricate semantic concepts.
Acknowledgements
This work was supported by the National Nat- ural Science Foundation of China under Grant 62276101 and the National Key R&D Pro- gram of China under Grant 2019YFC1510400, 2022YFC3006405.
4 Conclusion
In this paper, we proposed novel unsupervised framework SA-GAN for bilingual lexicon induc- tion. Different from previous works that generally treat words in the embedding space isolated en- tities, SA-GAN considers each embedding space as graph and utilizes GCN module learn the topological information between words. Addition- ally. SA-GAN employs a GAN to learn a lincar mapping matrix without imposing an orthogonal- ity constraint, thereby transforming both languages into the same embedding space. To further im- prove the performance, especially for the language pairs where the isomorphic assumption may not hold exactly, we propose pairwise local mapping algorithm learn word-specific transformations instead of only applying a shared global mapping to all words. Extensive experiments conducted on the MUSE dataset demonstrate the superior perfor- mance of our model. SA-GAN outperforms exist- ing unsupervised alternatives and even surpasses state-of-the-ate supervised methods, especially for etymologically distant language pairs.
References
David Alvarez-Melis and Tommi Jaakkola. 2018. Mossov-Wsseerstein alignment of word embedding spaces. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 1881-1890, Brussels, Belgium. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Generalizing and improving bilingual word embed- ding mappings with a multi-step framework of linear transformations, In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. A robust self-learning method for fully unsupervised cross mooss-lingual mappings of word embeddings. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 789-798. Melbourne, Australia. As- sociation for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018c. Unsupervised neural ma- chine translation. In International Conference on Learning Reprexentations.

---

Xuefeng Bai, Hailong Cao. Kehai Chen, and Tiejun Zhao. 2019. A bilingual adversarial autoen- coder for unsupervised bilingual lexicon induction. IEEE/ACM Transactions on Audio. Speech, and Lan- guage Processing, 27(10):1639-164)
Yedid Hoshen and Lior Wolf. 2018. Non-adversarial unsupervised word translation. In Proceedings of the 2018 8 Conference on Empirical Methods in Natu- ral Language Processing, pages 469-478, Brussels, Belgium. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
Pratik Jawanpuria. Arjun Balgovind. Anoop Kunchukut- tan, and Bamdev Mishra. 2019. Learning multilin- gual word embeddings in latent metric space: A geo- metric approach. Transactions of the Association for Computational Linguistics, 7:107-120.
Hailong Cao and Tiejun Zhao. 2018. Point set reg-
istration for unsupervised bilingual lexicon induc- tion. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, 18, page 3991-39977. AAAI Press.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Fdouard Grave. 2018 Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Procexsing, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
Xilun Chen and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, pages 261-270, Brussels, Belgium. Association for Computational Linguistics.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In International Conference on Learning Representations.
Xin Dong and Gerard de Melo. 2019. A robust self- learning framework for gross-lingust text classifi- cation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNIP-IJCNI.P), pages 6306-63 10, Hong Kong, China. Association for Com- putational Linguisties.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Interna- tional Conference on Learning Representations.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi- supervisd learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Ashwinkumar Ganesan, Francis Ferraro, and Tim Oates. 2021. Learning a reversible embedding mapping us- ing bi-directional manifold alignment. In Findings of the Association for Computational Linguistics: ACI-
Yaoyiran Li, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulić. 2022. Improving word translation via two-stage contrastive learning. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 4353-4374, Dublin, Ireland, Association for Computational Linguistics.
IJCNLP 202]. pages 3132-3139, Online. Association for Computational Linguistics.
Goran Glavak, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross- lingual word embeddings: On strong baselines, com- parative analyses, and some misconceptions. ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 710-721, Florence, Italy. Association for Computational Lin- guistics.
Pro-
Yuling Li, Yuhong Zhang, Kui Yu, and Xuegang Hu. 2021. Adversarial training with wasserstein distance for learning cross-lingual word embeddings. Applied Intelligence, 51(1 (111:7666-7678
Goran Glavaš and Ivan Vulić. 2020. Non-linear instance-based cross-lingual mapping for non- isomorphic embedding spaces. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 7548-7555, Online. As- sociation for Computational Linguistics.
Antonio Valerio Miceli Barone. 2016. Towards cross- lingual distributed representations without parallel text trained with adversarial autoencoders. In Pro- ceedings of the Ist Workshop on Representation Learning for NLP, pages 121-126, Berlin, Germany, Association for Computational Linguistics.
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Tomas Mikolov, Quoe V Le, and llya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv: 1309.4168.
Tasnim Mohiuddin, M Saiful Bari, and Shafiq Joty. 2020. LNMap: Departures from isomorphic assump- tion in bilingual lexicon induction through non-linear mapping in latent space. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 2712-2723, On- line. Association for Computational Linguistics.
Xiangnan He,, Kuan Deng, Xiang Wang, Yan Li, Yong- dong Zhang, and Meng Wang. 2020. Lightgeu: Sim- plifying and powering graph convolution network for recommendation. In Proceedings of the 43rd Inter- national ACM SIGIR conference on research and de- velopment in Information Retrieval, pages 639-648.

---

methods and Tasnim Mohiuddin and Shafiq Joty. 2019. Revisiting adversarial autoencoder for unsupervised word trans- lation with cycle consistency and improved training. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- fional Linguislic:: Human Language Technologies, Volume (Long and Short Papers), pages 3857-3867, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Arab Emirates, Association for Computational Lin- guistics.
Zhihao Xiong and Qian Tao. 2021. An unsupervised riossllingual word alignment approach based cyele-gan and hybrid training. In 2021 Interna- rional Conference on Electrical, Computer and En- ergy Technologies (ICECET), pages 1-6. IEEE.
A Related Work
Silviu Vlad Oprea, Sourav Dutta, and Haytham Assem. 2022. Multi-stage framework with refinement based point set registration for unsupervised bi-ling bi-lingual word alignment. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 1089-1097, Gyeongju, Republic of Korea. In- ternational Committee Computational Linguistics.
The basic idea for Bilingual lexicon induction (BLI) is to learn cross-lingual mappings which trans- form word embeddings of different languages to the same embedding space, and then induce Bilin- gual lexicons from the Iearned gross-lingual embed- dings. Based on the availability of a seed dictionary, we divide related work into the following two cat- egories: superviised/semi-uperrvviii unsupervised methods.
Barun Patra, Joel Ruben Antony Moniz. Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision non-isometric embedding spaces. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 184-193, Florence, Italy, Association for Computational Linguisties.
A.1 Supervised/Semi-suerrvii Methods
Mikolov et al. (2013) first observe that the word embedding space of one language can be trans- formed into another using linear mapping. based on the isomorphic assumption that monolingual word embeddings exhibit similar geometric properties across languages. Artetxe et al. (2018a) propose a multi-step framework that generalizes a substan- tial body of previous work. The core steps include normalization, whitening, orthogonal mapping, re- weighting, de-whitening, and dimensionality reduc- tion. Joulin et al. (2018) use a supervised method RCSLS which optimizes the CSLS distance in an end-to-end manner for the supervised matching pairs. Jawanpuria et al. (2019) propose to map both the source and target word embeddings to the common latent space via two orthogonal transfor- mations.
Shuo Ren, Shujie Liu, Ming Zhou, and Shuai Ma. 2020, A graph-basee coarse-to-fine method for unsuper- vised bilingual lexicon induction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3476-3485, On- line. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulíé, and Anders Søgaard. 2019, A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569- 631.
Vin Sachidananda, Ziyi Yang, and Chenguang Zhu. 2021. Filtered inner product projection for crosslin- gual embedding alignment. In International Confer- ence on Learning Representations.
Pranaydeep Singh and Els Lefever. 2020. Sentiment amalysis for Hinglish code-mixed tweets by means of cross-linggul word embeddings. In Proceedings of the The 4th Workshop on Computational Approaches Code Switching, pages 45-51, Marseille, France. European Language Resources Association.
Previously methods learned global-shared linear transformations based on the isomorphic assump- tion. However, several researchers have found that the isomorphic assumption may not hold all the time, especially for distant language (Søgaard et al., 2018). Patra et al. (2019) observe that the language pairs with high Gromov-Hausdorf (GH) distance cannot be aligned well using orthogonal transfor- mation and proposed semi framework which re- laxed isomorphic assumption by jointly optimizing weak orthogonality constraint in the form of a back-translation loss. Mohiuddin et al. (2020) de- sign a semi-supervised model that uses non-linear mapping in the latent space to learn cross-lingus word embeddings, which is also independent of the
Anders Spgaard, Sebastian Ruder, and Ivan Vulić, 2018 On the limitations of unsupervised bilingual dictio- nary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers). pages 778-788 Melbourne, Australia. Association for Computational Linguistics.
Zhoujin Tian, Chaozhuo Li, Shuo Ren, Zhiqiang Zuo, Zengxuan Wen, Xinyue Hu, Xiao Han, Haizhen Huang, Denvy Deng. Qi Zhang, and Xing Xie. 2022. RAPO: An adaptive ranking paradigm for bilingual lexicon induction. In Proceedings of the 2022 Con- ference on Empiricul Methods in Natural Language Processing, pages 8870-8883, Abu Dhabi, United

---

isomorphic assumption. Glavaš and Vulié (2020) propose a supervised word-specific transformation after learning a single global rotation matrix, thus the final mapping function is globally non-linear which performs well in distant language pairs. The PLM algorithm in this paper is inspired by the literature, but differs in that, in comparison with this work, we propose different transformation framework that can be applied to the unsupervised approach without any labeled data. Sachidananda et al. (2021) align embeddings to isomorphic vec- tor spaces, using pairwise inner products. Li et al. (2022) improve word translation via two-stage con- trastive learning. Tian et al. (2022) propose a ranking-based bilingual lexicon induction model which provides sufficient discriminative capacity to rank the candidates.
other frameworks which focus on direct bidi- rectional mappings between the source language and target language, Bai et al. (2019) train two au- toencoders jointly to transform the source and the target monolingual word embeddings into a shared embedding space to capture the cross- loossll fea- tures of word embeddings. Li et al. (2021) observe that low-frequency words tend to be densely clus- tered in the embedding space, to overcome this issue, they introduced noise function to disperse dense word embeddings and a Wasserstein critic network to preserve the semantics of the source word embeddings.
On the other hand, non-adversarial approaches have also been proposed for unsupervised cross- lingual word alignment. Hoshen and Wolf (2018) use the principal component of monolingual word embeddings to build initial alignment and then it- eratively refined the alignment using a variation of the e Iterative Closest Point (ICP) method used in computer vision. Artetxe et al. (2018b) explore the similarity of the embeddings to learn an initial dictionary in an unsupervised way and improve it with robust self-learning approach. Alvarez- Melis and Jaakkola (2018) cast the problem optimal transport problem and measure the sim- ilarity between pairs of words across languages using wesses--aserrttein distance. Cao and Zhao (2018) propose to use the Coherent Point Drift (CPD) algorithm to map the whole source embed- dings to the target embedding space. Inspired by Cao and Zhao (2018), Oprea et al. (2022) employ the CPD algorithm to perform an iterative two-step refinement on the initial global mapping trained by CycleGAN. However, both of them focus on global mapping under the isomorphic assumption. Ren et al. (2020) leverage the Ker-oKerossch (BK) al- gorithm to extract clique-level information, which is not only semantically richer than what a single word provides but also reduces the bad effect of the noise in the pre-trained embeddings.
Nevertheless, all these methods still require su- pervised signals and cannot be applied to the unsu- pervised learning setting without any labeled dic- tionary.
A.2 Unsupervised Methods
Recently fully unsupervised methods have been proposed to induce a bilingual dictionary by align- ing monolingual word embedding spaces. A typ- ical research line is based on adversarial training. Miceli Barone (2016) proposes adversarial au- toencoder framework to map the source language word embeddings the target language, where an encoder aims make the transformed embeddings not only indistinguishable by the discriminator but also recoverable after a reversed mapping by the decoder. Although promising, the reported per- formance is not satisfying. Lample et al. (2018) are the first to show very impressive results for unsupervised word translation where rough ro- tation matrix is first learned using the adversarial framework and further refined with a self-learning process. Based on the previous work (Lample et al., 2018). Chen and Cardie (2018) propose an adver- sarial training framework in the multilingual setting which not only considers one pair of languages at a time but explicitly exploits the relations between all language pairs. Mohiuddin and Joty (2019) revisit the adversarial autocneoder for unsupervised word translation and includes cycle consistency and in- put reconstruction constraints to guide the mapping. Xiong and Tao (2021) propose an unsupervised ap- proach via bidirectional feature mappings based on cycle-GAN and hybrid training. In contrast to
B Supplementary Experiments
B.1
Case Study
To better demonstrate the effectiveness of our model bilingual lexicon induction, we give some examples of the dictionary inferred with our method, comparing with that inferred by two adver- sarial methods Mohiuddin and Joty, 2019 and Bai et al., 2019, denoted as Adv-M and Adv-B respec- tively. We choose the language pair English- Danish

---

examples, as shown in Table
tence. Table 6(b) shows sentence translation trieval results on the Europarl corpus. On the en-fr language pairs, our model obtains the best score with up to 3.5% improvements. Besides, our pro- posed method performs the best on the averaged accuracy, which depicts that SA-GAN provides better performance in sentence translation retrieval tasks.
Query
Adv-M
Adv-B
Ours
Gold
uccuracy
nojagtighed termiske
nojagtighed varmeledning panthers
nojugtighed
nujagtighed termiske raiders
thermal
termiske
raiders
raiders
raiders
smoke
flammer
rog
rog rog kekkenet
kitchen kelkenhavvn
kokkenet
kokkenet
Table
Word translation examples for Anglish-Damise
En-De
En-Es
Avg
Methods
In the first example, both approaches find the cor- rect translations. In the following four examples, our approach SA-GAN successfully induces the correct translations with similar meanings, while Adv-M and Adv-B fail to find all correct trans- lations for the given queries, even having signifi- cantly different meanings for their induced words compared with the gold translations. From these examples, we find that our method produces bilin- gual lexicons with higher quality. This is because our approach can effectively utilize the topologial structure of the embedding spaces, and pair- wise mapping is learned for every different word to alle- viate the reliance on isometry, which improves the BLI performance even further.
(Lample et al., 2018)
70.8
71.3
71.2
71.1
71.1
(Arteixe et al., 2018b)
71.9
71.9
72.1
72.1
72.0
(Mohiuddin and Joty, 2019)
fail
72.0
72.4
71.8
72.1
(Opren et al., 2022)
72.6
72.5
73.0
72.8
72.7
SA-GAN
72.6
72.6
72.5
72.5
72.6
En-Fr
En-Es
Avg
Methods
(Lample al., 2018)
69.1
69.9
75.1
73.9
72.0
(Arteixe al., 2018b)
69.6
69.3
74.7
74.4
72.0
(Mohiuddin and Joty, 2019)
68.0
71.0
75.0
75.7
72.4
(Oprea al., 2022)
70.2
70.9
76.7
76.3
73.5
SA-GAN
73.7
74.4
75.4
76.0
74.9
(h)
Table 6: Performance for (a) Pearson's correlation score(() (%) for word similarity task, and (b) sentence trans- lation retrieval accuracy.
B.2 Downstream Tasks
To better test model's robustness and effec- tiveness, include more downstream tasks, i.e.. Semantic Word Similarity and Sentence Transla- tion Retrieval tasks as in the lample2018 8word and oprea-etal-202 2002--2022222222221
Semantic word similarity We evaluate the qual- ity of cross-lingual embeddings with the task of Semantic Word Similarity, which aims eval- uating how well the cosine similarity berween words of different languages correlates with human- annotated word similarity scores. As shown in Table 6(a), our proposed SA-GAN has better Pearson's 's correlation to buman-annotated scores across languages on the en-de and de-en language pairs and achieves comparable performance and es-en, indicating that our model provides good alignment across languages.
Sentence translation retrieval This task goes from word to sentence level and studies sentence trans- lation retrieval. Following (Lample et al., 2018), the sentences represented a bag of words. and the IDF-weighted average of word embeddings of the sentence is used its sentence embedding.
The closest sentence from the target language is returned as its translation of the given source sen-


Reviews:
[{"review_id": "KmbxHQUgtG", "reviewer": "Reviewer_qVkS", "summary": "", "questions": "My major question is about the GCN - what does it do, if it does not learn anything? How is, then, the structural information propagated within the graph?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to *Reviewer qVkS* \n\n## Comment1 : \n\"Although the structural information has not been explicitly used in the current problem statement, it has been implicitly used in few previous works on bilingual mapping induction. Please see: \"Multi-Stage Framework with Refinement based Point Set Registration for Unsupervised Bi-Lingual Word Alignment\". Oprea et al., COLING 2022. \"Point Set Registration for Unsupervised Bilingual Lexicon Induction\". H. Cao and T. Zhao, IJCAI 2018. As such, a proper discussion and comparison is necessary in the current paper.\"\n### Response:\nThank the reviewer for helpful comment. We have cited the two references and added some discussions as follows. \n\nBoth Cao et.al [1] and Oprea et.al [2] propose to use the Coherent Point Drift (CPD) algorithm [3] to preserve the underlying global topological structure by aligning source and target embeddings into a shared embedding space. Specifically, Cao et.al [1] used the CPD algorithm to map the whole source embeddings to the target embedding space. Different from [1], Oprea et.al [2] used a CycleGAN to map them into an initial globally aligned embedding space, and then employed the CPD algorithm to perform an iterative two-step refinement on the initial global mapping. However, they focus on the global mapping under the isomorphic assumption. \n\nDifferent from their methods: \n\nFirstly, our PLM method captures the local inter-space structural information based on the nearest word pairs neighbors, and learns word-specific transformations instead of a global-shared mapping. Therefore, our method is more suitable for the language pairs of low isometry.\t We can directly see the improvement brought by PLM from the experiments, as shown in table 1. The BLI performance is further improved with PLM, with a gain of 0.7% on average on etymologically close language pairs and 1.2% on low-isomorphic language pairs, which is remarkable.\n\nSecondly, our method explicitly explores intra-space structural correlations using GCN before we train the transformation by GAN (Oprea et al. use CPD after GAN is trained), which provides much rich semantic information to the embeddings and that can be used as a-priori information to the following GAN training. With the help of explicitly intra-space structural information, the generator can generate much more accurate and semantically consistent cross-language word embedding samples, which greatly stabilizes the training process, especially for the missions on low-isomorphic language pairs, where the other GAN-based unsupervised methods (Lample et al[4]. and Mohiuddin et al.[5]) usually fail to converge (eg, En-Fi or En-Zh according to LNMAP[6]). For our method, we can also observe in Table 2 that the performance will consistently drop in all language pairs and even fail to converge in low-isomorphic (distance) language pairs if we further remove the GCN module, which indicates the intra-space structural extraction before adversarial training will greatly contribute to GAN’s training and stabilize the BLI performance.\n| Method | En-Es  | Es-En   | En-De   | De-En   | En-Fr  | Fr-en   | En-Ru    | Ru-En    | Avg   |\n|---|---|---|---|---|---|---|---|---|---|\n|(Oprea et al., 2022)|\t83.3|\t**85.4**|\t75.8|\t75.8|\t**83.4**|\t84.1|\t49.5|\t64.0|\t75.2|\n|SA-GAN|\t**84.2**|\t**85.4**|\t**77.5**|\t**76.1**|\t83.1|**84.4**\t|**50.9**|\t**66.2**|\t**76.0**|\n\nIn addition, as suggested by the reviewer, we further added a comparison with Oprea et.al [1] as suggested in Table 1 of revised manuscript. It can be seen that our approach outperforms the Oprea et.al [2]’s methods on most language pairs (7 of 8 language pairs), with a gain of 0.8% on average and a larger improvement is achieved in low-isomorphic languages (1.4% on En-Ru and 2.2% on Ru-En). The added experiments can prove the effectiveness of our approach. The updated results will be shown in Table 1 in the revision.\n\nWe thank the reviewer for providing the relevant references. Making proper discussion and comparison with these references is very helpful for us to understand how the structural information extraction contributes to BLI task. We have added these references in the revised manuscript and more comparison has been made. We also hope our work can spark further advancement in structural information extraction and contributes to widely used languages in world of low isometry. \n\n[1] Cao, Hailong, and Tiejun Zhao. \"Point Set Registration for Unsupervised Bilingual Lexicon Induction.\" IJCAI. 2018.\n\n[2] Oprea, Silviu Vlad, Sourav Dutta, and Haytham Assem. \"Multi-Stage Framework with Refinement Based Point Set Registration for Unsupervised Bi-Lingual Word Alignment.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n[3] Myronenko, Andriy, and Xubo Song. \"Point Set Registration: Coherent Point Drift.\" IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 32.12 (2010).\n\n[4] Lample, Guillaume, et al. \"Word translation without parallel data.\" International Conference on Learning Representations. 2018.\n\n[5] Mohiuddin, Muhammad Tasnim, and Shafiq Joty. \"Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training.\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.\n\n[6] Mohiuddin, Muhammad Tasnim, M. Saiful Bari, and Shafiq Joty. \"LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n\n\n## Comment2 :\nThe GCN that is proposed has no learnable parameters, so it is a static matrix transformation operation. So, why is the term GCN used - isn't it misleading? Further, if it is not learning anything, it is a simple aggregation function, as far as I understood. Then the structural information is not propagating through the entire graph - this is counter-intuitive because the author(s) claim that structural information usage is a key feature of the proposed framework. Am I missing something here?\n\n### Response:\nWe thank the reviewer for the careful review and valuable suggestions. For traditional GCN, each graph convolutional layer can be divided into two operations: embedding propagation (EP) and embedding transformation (ET). Concretely, GNN first executes the EP with the normalized adjacency matrix $\\hat{A}$ to the embedding matrix $X$:\n$$EP(X)=\\hat{A}X,$$\nThen, the propagated feature $\\hat{X} = EP(X)$ will be transformed with the learnable transformation matrix $W$ and the activation function $\\sigma(·)$:\n$$ET(\\hat{X})=σ(\\hat{X}W).$$\nFor our simplified GCN module, it can be seen as only EP is kept, and the transformation ($W$) and the activation function in ET operation are removed. Therefore, **the structural information can still propagate through the graph through multiple EP operations.**\n\nWe propose to use GCN without $W$ for several reasons:\n\n1, **Removing $W$ in GCN doesn’t mean that it learns nothing.** The core step of GCN is the propagation operation. According to Thomas [1], even the GCN model with random weights (eg, $W=I$) can still serve as a powerful feature extractor for nodes in a graph. \n\n2, **Recent work shows that the transformation operation ($W$) is not necessary in GCN.** Previous works [2-3] have shown that the true effectiveness of GNNs lies in the EP operation rather than the ET operation inside the graph convolution. Specifically, SGC [2] reduces the entire procedure of GCN to a simple feature propagation step and their experimental evaluation demonstrates that these simplifications not only do not negatively impact accuracy in many downstream applications but also enable the model to scale to larger datasets and yields up to two orders of magnitude speedup over FastGCN. For specific downstream tasks, i.e. recommender systems, lightGCN [3] observes that feature transformation and nonlinear activation have no contribution to the performance and even increase the training difficulty. On the contrary, removing them leads to significant accuracy improvements and prevents overfitting problem. \n\n3, **As a matter of fact, our simplified GCN module can be seen as a special “decoupled GCN” [4-6].** Unlike traditional GCN, where the EP and ET operations are stacked together in each convolutional layer of GCN, the decoupled GCNs try to decouple these two operations. Thereinto, one pattern is the Disentangled Propagation and Transformation (DPT), where EP is executed in advance then training the neural network composed of multiple ET operations (also called Propagation then Training as described in [6]). For our method, the simplified GCN module can be seen as the EP operation. After that, the generator (one linear layer) in GAN is acted as the ET operation. Together, they form a decoupled GCN and the structural information is propagating through the entire graph through this decoupled GCN.\n\nWe can also see that our GCN module contributes to the performance through the Ablation Study in Table 2. The model performance will consistently drop in all language pairs and even fail to converge in distance language pairs if we further remove the GCN module. We can get the following conclusion: GCN module can capture much richer semantics by extracting structure information of embeddings space, which contributes to learning a better and robust mapping function which stabilizes the BLI performance;\n\nThanks for the valuable suggestion. For better understanding, we will add “Why  it works” discussion in the revised section, and a more detail experiment will be added to explore the effects of W in the revised manuscript.\n\n[1] Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.\n\n[2] Wu, Felix, et al. \"Simplifying graph convolutional networks.\" International conference on machine learning. PMLR, 2019.\n\n[3] He, Xiangnan, et al. \"Lightgcn: Simplifying and powering graph convolution network for recommendation.\" Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 2020.\n\n[4] Liu, Meng, Hongyang Gao, and Shuiwang Ji. \"Towards deeper graph neural networks.\" Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020.\n\n[5] W. Zhang, Z. Sheng, Y. Jiang, Y. Xia, J. Gao, Z. Yang, and B. Cui, “Evaluating deep graph neural networks,” arXiv preprint arXiv:2108.00955, 2021.\n\n[6] Dong, Hande, et al. \"On the equivalence of decoupled graph convolution network and label propagation.\" Proceedings of the Web Conference 2021. 2021.\n\n\n## Comment3 :\nFor experiments, I have 2 comments - (i) addition of performance on word similarity and sentence translation tasks as in the MUSE paper (and others) would lend more credibility to the robustness and effectiveness of the framework. (ii) addition of morphologically rich languages like Finnish, Hebrew, etc and low-resource languages in the experiments would be good to have (minor point).\n### Response:\n1, Thanks for suggestions. As suggested by the reviewer, we e added the experiments of word similarity and sentence translation tasks following MUSE paper, as shown in table A and table B.\n\n|TableA: Sord Sim|En-De|\tDe-En|En-Es|Es-En|\n|---|---|---|---|---|\n|(Lample et al., 2018b) \t|\t0.708|\t0.713\t|\t0.712|\t0.711|\n|(Artetxe et al., 2018b) \t|\t0.719|\t0.719|\t\t0.721|\t0.721|\n|(Mohiuddin and Joty, 2019) \t|\tfail\t|0.72|\t\t0.724|\t0.718|\n|SA-GAN without PLM\t|\t0.721|\t0.721\t|\t0.723|\t0.723|\n|SA-GAN\t|**0.726**\t|**0.726**\t\t|**0.725**\t|**0.725**|\n\nTable A shows that SAGAN achieves a better Pearson’s correlation to human-annotated scores across languages (except it) – providing better alignment across languages.\n\n|TableB: Sentence Trans|\t\ten-es|\tes-en\t|\ten-fr|\tfr-en|\n|---|---|---|---|---|\n|(Lample et al., 2018b)\t|\t75.1|\t73.9|\t\t69.1|\t69.9|\n|(Artetxe et al., 2018b)\t\t|74.7\t|74.4\t\t|69.6\t|69.3|\n|(Mohiuddin and Joty, 2019)\t\t|75.0|\t75.7|\t\t68.0\t|71.0|\n|SA-GAN without PLM|\t\t73.3|\t73.7\t|\t71.2|\t71.8|\n|SA-GAN \t|\t**75.4**|\t**76.0**\t|\t**73.7**|\t**74.4**|\n\nTable B shows sentence translation retrieval result on Europarl corpus. Table B depicts that SAGAN provides better sentence translation retrieval accuracy, with up to 3.4% score improvements.\n\n2, To better test our model’s robustness and effectiveness, we added more experiments on morphologically rich languages, i.e., Finnish (Fi), Hebrew (He), and Romanian (Ro) following Oprea [1]. The results ae presented in Table C. From the measurements, we can see that our approach outperforms existing methods on all of six tasks on morphologically rich languages pairs, with a gain up to 1.4% on En-He and 0.8% on average of all languages, which further shows the robustness and effectiveness of our framework.\n\n|TableC: morphologic|\t\ten-fi|\tfi-en\t|\ten-he\t|he-en\t|\ten-ro|ro-en|\tavg|\n|---|---|---|---|---|---|---|---|\n|(Lample et al., 2018b)|\t\t43.7|\t53.7\t|\t38\t|fail|\t\t58.0|\t66.0\t|\t51.9|\n|(Artetxe et al., 2018b)\t\t|49.9|\t63.5| 44.6|57.7\t|64.2\t|71.8\t|\t58.6|\n|(Mohiuddin and Joty, 2019)\t|\t49.8|\t65.5\t|\t46.1|\t58.6\t|\t62.6|\t71.9\t|\t59.1|\n|(Oprea et al., 2022) [1]\t\t|49.9|\t65.5|\t\t46.6|\t59.1|\t\t65.4|\t**74.3**\t|\t60.1|\n|SA-GAN without PLM\t|\t52.3|\t64.9|\t\t47.1|\t57.5|\t\t66.2|\t72.3|\t\t60.1|\n|SA-GAN  \t\t|**52.7**\t|**66.0**\t\t|**48.0**\t|**59.7**\t|**66.4**\t|72.7\t\t|**60.9**|\n\n[1] Oprea, Silviu Vlad, Sourav Dutta, and Haytham Assem. \"Multi-Stage Framework with Refinement Based Point Set Registration for Unsupervised Bi-Lingual Word Alignment.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n## Comment4 :\nMy major question is about the GCN - what does it do, if it does not learn anything? How is, then, the structural information propagated within the graph?\n### Response:\nWe thank the reviewer for the careful review and valuable suggestions. A detail response can be found in the response of the above comment 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles an important problem in the domain of multilingual NLP, wherein such dictionary induction is vital in certain scenarios.\n2. The intuition of directly incorporating embedding space structural information is interesting, and might spark further advancement in this direction.\n3. Experimental results depicts the effectiveness of the proposed approach.", "reasons_to_reject": "1. Although the structural information has not been explicitly used in the current problem statement, it has been implicitly used in few previous works on bilingual mapping induction. Please see:\n\"Multi-Stage Framework with Refinement based Point Set Registration for Unsupervised Bi-Lingual Word Alignment\". Oprea et al., COLING 2022.\n\"Point Set Registration for Unsupervised Bilingual Lexicon Induction\". H. Cao and T. Zhao, IJCAI 2018.\nAs such, a proper discussion and comparison is necessary in the current paper.\n\n2. The GCN that is proposed has no learnable parameters, so it is a static matrix transformation operation. So, why is the term GCN used - isn't it misleading? Further, if it is not learning anything, it is a simple aggregation function, as far as I understood. Then the structural information is not propagating through the entire graph - this is counter-intuitive because the author(s) claim that structural information usage is a key feature of the proposed framework. Am I missing something here?\n\n3. For experiments, I have 2 comments - (i) addition of performance on word similarity and sentence translation tasks as in the MUSE paper (and others) would lend more credibility to the robustness and effectiveness of the framework. (ii) addition of morphologically rich languages like Finnish, Hebrew, etc and low-resource languages in the experiments would be good to have (minor point).", "questions_to_authors": "My major question is about the GCN - what does it do, if it does not learn anything? How is, then, the structural information propagated within the graph?", "paper_topic_and_main_contributions": "The paper proposes a framework for bilingual word mapping across the independent language embedding space. The author(s) combine structural information of the embedding spaces via GCN, and then employ GAN to learn the mapping function. Experimental evaluation of the presented framework has been conducted on bilingual dictionary creation benchmark dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\"Multi-Stage Framework with Refinement based Point Set Registration for Unsupervised Bi-Lingual Word Alignment\". Oprea et al., COLING 2022.\n\"Point Set Registration for Unsupervised Bilingual Lexicon Induction\". H. Cao and T. Zhao, IJCAI 2018."}}, {"review_id": "fgz4JbGFsC", "reviewer": "Reviewer_vh2V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to *Reviewer vh2V* \n\n## Comment1 :\nThe main contribution of this paper lies in the structure-aware encoder-based model for seed lexicon induction, there should be an experiment to study the quality of seed lexicon.\n### Response:\nIn our original manuscript, as shown in Section Appendix. B.1, we made a Case Study of quality of the seed lexicon for En-Da on MUSE dataset. We also gave some examples of the dictionary inferred with our method and compare with two adversarial methods. From the examples, we can find that our approach SAGAN successfully induces the correct translations with similar meanings, while the other methods fail to find all correct translations for the given queries, even having significantly different meanings for their induced words compared with the gold translations. From these examples, we find that our method produces bilingual lexicons with higher quality.\n\nWe thank the reviewer for the careful review and valuable suggestions. To better analyze the quality of seed lexicon, we will further give some visualization of the retrieval results of our method in a space matter in the revised manuscript.\n\n## Comment2 :\nWhile the paper focuses on bilingual lexicon induction, it would be beneficial to include a downstream task, such as cross-lingual Natural Language Inference (NLI), to demonstrate the potential impact of the proposed method on downstream applications. This would provide further insights into the effectiveness of the approach beyond the specific lexicon induction task.\n### Response:\nWe thank the reviewer for the insightful suggestions. According to the suggestion of the reviewer, we have added some experiments by conducting more downstream tasks, i.e., **Semantic Word Similarity** and **Sentence Translation Retrieval** tasks as in the Lample et.al [1] and Oprea et.al [2] , which lends more credibility to the robustness and effectiveness of our approach, as shown in table A and table B. For each result, we performed the experiments for 5 times and reported the average results.\n\n|TableA: Sord Sim|En-De|\tDe-En|En-Es|Es-En|\n|---|---|---|---|---|\n|(Lample et al., 2018b) \t|\t0.708|\t0.713\t|\t0.712|\t0.711|\n|(Artetxe et al., 2018b) \t|\t0.719|\t0.719|\t\t0.721|\t0.721|\n|(Mohiuddin and Joty, 2019) \t|\tfail\t|0.72|\t\t0.724|\t0.718|\n|SA-GAN without PLM\t|\t0.721|\t0.721\t|\t0.723|\t0.723|\n|SA-GAN\t|**0.726**\t|**0.726**\t\t|**0.725**\t|**0.725**|\n\nFor Semantic Word Similarity task, it evaluates the quality of cross-lingual word alignment based on the correlation between cosine similarity between words in different languages and human-annotated word similarity scores. Table A shows that SAGAN achieves a better Pearson’s correlation to human-annotated scores across languages – providing better alignment across languages.\n\n|TableB: Sentence Trans|\t\ten-es|\tes-en\t|\ten-fr|\tfr-en|\n|---|---|---|---|---|\n|(Lample et al., 2018b)\t|\t75.1|\t73.9|\t\t69.1|\t69.9|\n|(Artetxe et al., 2018b)\t\t|74.7\t|74.4\t\t|69.6\t|69.3|\n|(Mohiuddin and Joty, 2019)\t\t|75.0|\t75.7|\t\t68.0\t|71.0|\n|SA-GAN without PLM|\t\t73.3|\t73.7\t|\t71.2|\t71.8|\n|SA-GAN \t|\t**75.4**|\t**76.0**\t|\t**73.7**|\t**74.4**|\n\nFor Sentence Translation Retrieval task, it studies sentence translation retrieval on Europarl corpus. The sentences are represented as a bag-of-words and the idf-weighted average of word embeddings is considered as its encoding. For each source sentence, the closest sentence from the target language is returned as its translation. Table B shows sentence translation retrieval result on Europarl corpus, which depicts that SAGAN provides better sentence translation retrieval accuracy, with up to 3.4% score improvements.\n\n[1] Lample G, Conneau A, Ranzato M A, et al. Word translation without parallel data[C]//International Conference on Learning Representations. 2018.\n\n[2] Oprea, Silviu Vlad, Sourav Dutta, and Haytham Assem. \"Multi-Stage Framework with Refinement Based Point Set Registration for Unsupervised Bi-Lingual Word Alignment.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n## Comment3 :\nMissing References:\n\nImproving Word Translation via Two-Stage Contrastive Learning (ACL 2022)\n\nFiltered Inner Product Projection for Crosslingual Embedding Alignment (ICLR 2021)\n\n### Response:\nIt is a very good suggestion. These references are very helpful for us to understand BLI task and further research on CLE evaluation and model analysis. According to the suggestion of the reviewer, we have added these references in the revised manuscript in Section of Related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper is well-organized and presented in an easy-to-follow manner.\n2.The motivation behind this approach is reasonable and well supported by experiments.\n3.The paper demonstrates substantial improvements over several baseline methods.", "reasons_to_reject": "The experiments is somewhat weak. \n1) The main contribution of this paper lies in the structure-aware encoder-based model for seed lexicon induction, there should be an experiment to study the quality of seed lexicon.\n2) While the paper focuses on bilingual lexicon induction, it would be beneficial to include a downstream task, such as cross-lingual Natural Language Inference (NLI), to demonstrate the potential impact of the proposed method on downstream applications. This would provide further insights into the effectiveness of the approach beyond the specific lexicon induction task.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on bilingual lexicon induction, presenting a structure-aware approach to effectively model topological structure mapping. The proposed method utilizes a Graph Neural Network (GNN) to generate structure-aware word embeddings and employs a Generative Adversarial Network (GAN) to align two sets of word embeddings. Experimental results demonstrate that the proposed method outperforms several baselines, yielding superior outcomes in the task of bilingual lexicon induction.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Improving Word Translation via Two-Stage Contrastive Learning (ACL 2022)\n2. Filtered Inner Product Projection for Crosslingual Embedding Alignment (ICLR 2021)"}, "extra_text": {}}, {"review_id": "ZSs93fJyp7", "reviewer": "Reviewer_ggx5", "summary": "", "questions": "1. Line 213: Between noes I and j, the negative log of the euclidean distance is computed. Can the authors explain the reasoning here?\n2. Line 371-373: Why are words with low frequency filtered out?\n3. Line 391: If anchor points that are far off have a high coefficient of 0.4, will the softmax output (when scaled) be high too? If baseline values (points which are far away) are high, then the probability will closer to a uniform distribution.\n4. In Table 2: Why does the GAN model fail if there isn’t a GNN? \n5. In Table 3: What does “without p” mean? Is it set to 1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to *Reviewer ggx5* \n\n## Comment1 :\nFrom algorithm 1, we can see that model training is an amalgamation of several strategies in comparison to a singular outlined training algorithm.\n### Response:\nThank you for reviewing and providing valuable comments. We adopt this pattern for several reasons:\n\n1, Each module has its own role to play while targeting different goals. By the strategy of splitting, each module can focus more on its task and improve the overall performance while reducing the complexity. Specially, the GCN module aims to extract structural and semantical information of the embeddings and provide prior knowledge for subsequent GANs to keep the training stable especially for languages in low-isomorphism; the GANs module aims to learn a global mapping, which roughly aligns the word embeddings of two languages, only then can the PLM further fine-turn the embeddings in local area according to the seed dictionary induced by the aligned embeddings; the PLM module aims to learn word-specific local transformations instead of only applying a shared global mapping to all words. Therefore, the final mapping relation between the two languages is non-linear in global, which avoids the pitfalls of isomorphism. We can observe from Table 1, each three module focuses on its task and performs better on 15 of 16 language pairs, especially on etymologically distant language pairs like en-hu and en-da, with improvements of 4.1% to 4.6% in absolute over the best unsupervised baseline, which proves the effectiveness of our strategies.\n\t\n2, The amalgamation of several strategies is very widely adopted in Bilingual Lexicon Induction (BLI) models. Artetxe et.al. [1] propose a multi-step framework which generalizes previous work. Their framework consists of several steps: Normalization, whitening, mapping, re-weighting, de-whitening, and dimensionality reduction. Other works can be considered as a combination of different steps. For most unsupervised methods shown in Table 1, their methods consist of normalization, mapping, re-weighting and de-whitening. i.e., Artetxe et.al [2], Bai et.al. [3] Mohiuddin [4]. Moreover, Oprea et.al. [5] propose an unsupervised multi-stage framework for learning bi-lingual word alignment, by using a combination of adversarial training, refinement procedure, and point set registration, which is also an amalgamation of several strategies. \n\n3, Through modularization, our method can be more easily split and used by other researchers as part of the expansion of their models. For example, our unsupervised PLM module can be adopted to any other existing unsupervised methods as it does not rely on any supervised signal. They only need to feed their output embeddings to the PLM module to get the upgraded embeddings and improve their performances on low-isomorphism languages.\n\n\n[1] Artetxe, Mikel, Gorka Labaka, and Eneko Agirre. \"Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\n[2] Artetxe, Mikel, Gorka Labaka, and Eneko Agirre. \"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings.\" Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.\n\n[3] Bai, Xuefeng, et al. \"A bilingual adversarial autoencoder for unsupervised bilingual lexicon induction.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 27.10 (2019): 1639-1648.\n\n[4] Mohiuddin, Muhammad Tasnim, and Shafiq Joty. \"Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training.\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.\n\n[5] Oprea, Silviu Vlad, Sourav Dutta, and Haytham Assem. \"Multi-Stage Framework with Refinement Based Point Set Registration for Unsupervised Bi-Lingual Word Alignment.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n\n## Comment2 :\nAlthough isomorphism is not an underlying assumption, the learned transformation matrix is linear. In order to avoid the pitfalls, a non-linear mapping can be considered.\n### Response:\nWe thank the reviewer for the careful review and insightful comments. Previous methods learn an  **orthogonal** linear mapping under the isomorphism assumption. As the assumption does not hold, we remove the orthogonality constraint of the mapping function and an improvement is achieved, as shown in Table 2 in original paper. \n\nHowever, after replacing the transformation with a non-linear one (MLP with tanh as the non-linear activation function), our model’s performance consistently drops in all language pairs and the stability of the model is severely degraded in low-isomorphism languages, as shown in Table A below. \n\nTo exclude the influence of PLM, the results are investigated in the setting without the PLM module. We run 5 times for each language and only report the average of success runs. “P@1” indicates the BLI accuracy and “S” is the number of successful runs with >5% accuracy in 5 runs. The results show that, the performance is slightly but consistently drop in etymologically close languages (i.e., en-es and en-it). While for the low-isomorphism language pairs (i.e., en-zh and en-ru), not only the performance drops, but also the stability is severely degraded.  The original SAGAN can successfully run 5 times in 5 runs. However, after replacing the transformation with a non-linear one, it can only successfully run 1 or 2 times in 5 runs and the performance also drops. Therefore, we use the linear mapping in the paper.\n\n|Type|en-es| |en-it\t||\ten-zh\t||\ten-ru\t||\n|---|---|---|---|---|---|---|---|---|\n||P@1\t|S|\tP@1|\tS|\tP@1|\tS\t|P@1|\tS|\n|Linear\t|\t83.1|\t5\t|80.2|\t5|\t41.5|\t5|\t49.7|\t5|\n|Nonlinear\t\t|82.9\t|5\t|80.1|\t5\t|39.5|\t1\t|49.5\t|2|\n\nWe think there are several reasons under this phenomenon:\n\n1, After adding the activation function to the mapping, the geometric structure of the generated embeddings is changed too much, and the distribution is quite different between the mapped source embeddings and the target language. Therefore, it is difficult for the generator to generate enough real samples to fool the discriminator, and the adversarial training cannot continue.\n\n2, It is more suitable for supervised methods to use a non-linear mapping. Indeed, some methods [1-2] have found that the model can be trained successfully and even performs better on low isomorphism language pairs after adding non-linear mapping. However, all of these methods are supervised methods, where supervised signals are indispensable. As for unsupervised methods, they propose many other improvements for the low-isomorphism problem, but still retain the linear mapping, e.g., bai et.al[3] and Mohiuddin[4], which is consistent with the previously common unsupervised method.\n\nAlthough our transformation matrix is still linear, our full model is globally non-linear. This is because: after the linear transformation matrix is learned and the embeddings of two languages are mapped to the same space (till now, model is linear), we further recompute and update the embeddings for each point according to the local structure during PLM procedure. In other words, through PLM, we learn word-specific local transformations instead of only applying a shared global mapping to all words. Therefore, the final mapping relation between the two languages is non-linear in global, which avoids the pitfalls of isomorphism. We can also observe from Table 1 and the ablation study in Table 3, the performances are improved over all tasks after adding PLM.\n\n[1] Mohiuddin, Muhammad Tasnim, M. Saiful Bari, and Shafiq Joty. \"LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n\n[2] Tian, Zhoujin, et al. \"RAPO: An Adaptive Ranking Paradigm for Bilingual Lexicon Induction.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[3] Bai, Xuefeng, et al. \"A bilingual adversarial autoencoder for unsupervised bilingual lexicon induction.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 27.10 (2019): 1639-1648.\n\n[4] Mohiuddin, Muhammad Tasnim, and Shafiq Joty. \"Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training.\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.\n\n\n## Comment3 :\nAlthough the paper shows improvements, it does not ground it. For example, what does it mean to have a 2% relative improvement on en-it in comparison to the state of the art method. The paper can benefit from a small section on error analysis or comparison with an existing SOTA model to outline what the improvement is.\n### Response:\nWe thank the reviewer for the careful review and valuable suggestions. In our original manuscript, as shown in Section Appendix. B.1, we have made a Case Study of quality of the seed lexicon for En-Da on MUSE dataset. We give some examples of the dictionary inferred with our method and compare with two adversarial methods.From the examples, we can find that our approach SAGAN successfully induces the correct translations with similar meanings, while the other methods fail to find all correct translations for the given queries, even having significantly different meanings for their induced words compared with the gold translations. From these examples, we find that our method produces bilingual lexicons with higher quality.\n\nTo better outline what the improvement our model can provide, we further added some experiments on downstream tasks, i.e., **Semantic Word Similarity** and **Sentence Translation Retrieval** tasks as in the Lample et.al[1] and Oprea et.al[2] , which lends more credibility to the robustness and effectiveness of our approach, as shown in table A and table B. For each result, we performed the experiments for 5 times and reported the average results.\n\n|TableA: Sord Sim|En-De|\tDe-En|En-Es|Es-En|\n|---|---|---|---|---|\n|(Lample et al., 2018b) \t|\t0.708|\t0.713\t|\t0.712|\t0.711|\n|(Artetxe et al., 2018b) \t|\t0.719|\t0.719|\t\t0.721|\t0.721|\n|(Mohiuddin and Joty, 2019) \t|\tfail\t|0.72|\t\t0.724|\t0.718|\n|SA-GAN without PLM\t|\t0.721|\t0.721\t|\t0.723|\t0.723|\n|SA-GAN\t|**0.726**\t|**0.726**\t\t|**0.725**\t|**0.725**|\n\nFor Semantic Word Similarity task, it evaluates the quality of cross-lingual word alignment based on the correlation between cosine similarity between words in different languages and human-annotated word similarity scores. Table A shows that SAGAN achieves a better Pearson’s correlation to human-annotated scores across languages (except it) – providing better alignment across languages.\n\n\n|TableB: Sentence Trans|\t\ten-es|\tes-en\t|\ten-fr|\tfr-en|\n|---|---|---|---|---|\n|(Lample et al., 2018b)\t|\t75.1|\t73.9|\t\t69.1|\t69.9|\n|(Artetxe et al., 2018b)\t\t|74.7\t|74.4\t\t|69.6\t|69.3|\n|(Mohiuddin and Joty, 2019)\t\t|75.0|\t75.7|\t\t68.0\t|71.0|\n|SA-GAN without PLM|\t\t73.3|\t73.7\t|\t71.2|\t71.8|\n|SA-GAN \t|\t**75.4**|\t**76.0**\t|\t**73.7**|\t**74.4**|\n\nFor Sentence Translation Retrieval task, it studies sentence translation retrieval on Europarl corpus. The sentences are represented as a bag-of-words and the idf-weighted average of word embeddings is considered as its encoding. For each source sentence, the closest sentence from the target language is returned as its translation. Table B shows sentence translation retrieval result on Europarl corpus, which depicts that SAGAN provides better sentence translation retrieval accuracy, with up to 3.4% score improvements.\n\nFrom these tables, we see that the performance of Semantic Word Similarity and Sentence Translation Retrieval is restricted to the quality of BLI results. As our method provides better word translations, the downstream tasks benefit from ours accordingly.\n\n[1] Lample G, Conneau A, Ranzato M A, et al. Word translation without parallel data[C]//International Conference on Learning Representations. 2018.\n\n[2] Oprea, Silviu Vlad, Sourav Dutta, and Haytham Assem. \"Multi-Stage Framework with Refinement Based Point Set Registration for Unsupervised Bi-Lingual Word Alignment.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n\n## Comment4 :\nLine 213: Between noes I and j, the negative log of the euclidean distance is computed. Can the authors explain the reasoning here?\n### Response:\nWe thank the reviewers for the comments. By doing this, neighbor nodes closer to node i will have higher weights in adjacency matrix. \n\nIn comparison to use “negative Euclidean distance” directly, “negative log“ further scales the distance weights. As in high dimensional space, the nodes are very close to each other. Even node j is “very far” in semantic, it is still close in Euclidean distance (i.e., 0.8), compared to a real semantic close node k (i.e., 0.3 in Euclidean distance). However, after scaling, these two weights are now 0.01 for distant node j and 0.52 for close node k. Therefore, it makes nodes more distinguishable and the local structural information is preserved better by “filtering” these distant neighbors.\n\n\n## Comment5 :\nLine 371-373: Why are words with low frequency filtered out?\n### Response:\nIn word embedding training (using fasttext or word2vec), those low-frequency words are usually of low quality. In extreme cases, words that occur only once may appear anywhere in the embedding space, containing no semantic information but noise which has a negative effect on the quality of seed dictionary. Therefore, filtering the words with low frequency is performed.\n\n\n## Comment6 :\nLine 391: If anchor points that are far off have a high coefficient of 0.4, will the softmax output (when scaled) be high too? If baseline values (points which are far away) are high, then the probability will closer to a uniform distribution.\n### Response:\nWe thank the reviewers for the comments. After the softmax output, the coefficient is not high, since we further scale the weight through temperature controlling. For example, there are two anchor nodes i and j with coefficient of 0.8 and 0.4 separately (with 2 times larger). After softmax with temperature=0.1, the new coefficients will be 0.98 and 0.02 (with 49 times larger). \n\nIf baseline values are high, in this case, we think these points are the exactly anchors that we want. As the words that are semantically similar tend to be close in embedding space, we think these anchors with high value have similar semantic with the center word, and will have the same margin (distant of the word pair<$x_d$ to $y_d$> in seed dictionary) to move. \n\n\n## Comment7 :\nIn Table 2: Why does the GAN model fail if there isn’t a GNN?\n### Response:\nFor unsupervised models, it is difficult to train and have a good performance on low isomorphism languages (eg, En-Zh and En-Ru). In these languages, the unsupervised models even fail to converge. Same result can be found in Mohiuddin et.al [1-2], which directly use the original word embedding to train the GAN-based model.\n\nOur GNN, on the other hand, can effectively extract structural and semantical information of the embeddings and provide prior knowledge for subsequent GANs, which helps the generator generates better samples and stabilizes the training. \n\t\n[1] Mohiuddin, Muhammad Tasnim, M. Saiful Bari, and Shafiq Joty. \"LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n\n[2] Mohiuddin, Muhammad Tasnim, and Shafiq Joty. \"Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training.\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.\n\n\n## Comment8 :\nIn Table 3: What does “without p” mean? Is it set to 1?\n### Response:\nWe greatly appreciate the reviewer for the comments. In our original manuscript, as shown in Line 590, we have denoted the meaning of “without p” and the p is set to 1. For more clarity, the table has been revised as directed in a clearer way and a relevant description is added to the table caption. \n\n\n## Comment9 :\ninduction -> inductive\n### Response:\nThank you for pointing out this problem. The word format has been revised as directed.\n\n\n## Comment10 :\nIs PLM a mapping function or a loss? The authors can clarify if the PLM is a mapping function or an optimization criteria to learn a mapping. The text in the paper and equation appear to be different.\n### Response:\nThank you for your advice. PLM is a mapping function which recomputes and upgrades the embedding representations directly through the equation. As suggested, we reclarify the description in Line 345 in the revised version.\n\n## Comment11 :\nThe interpretation of “_” in Line 494 and Table 1 caption is different. The authors can clarify which one to use.\n### Response:\nWe thank the reviewer for the careful review and valuable suggestions. As suggested, we have changed the mark in Line 494 to “NA” to indicate “the authors did not report the number or their code is not publicly available”.\n\n## Comment12 :\nThe authors can add information if the improvements are absolute or relative.\n### Response:\nThank the reviewer for advice. The improvements are absolute and the description in Line511 has be revised as suggested.\n\n## Comment13 :\n“hold” -> “held”.\n### Response:\nThank you for pointing out this problem. The Verb tense has been revised as directed. In addition, we have carefully proofread the entire paper to improve the expression quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is written well with few mistakes. Some corrections have been suggested.\n2. The authors propose a method that exploits the manifold structure in the embedding space in order to learning a mapping between two embedding spaces.\n3. The papers have extensive experiments that analyze different aspects of the pipeline and show the importance of each individual component.", "reasons_to_reject": "1. From algorithm 1, we can see that model training is an amalgamation of several strategies in comparison to a singular outlined training algorithm. \n2. Although isomorphism is not an underlying assumption, the learned transformation matrix is linear. In order to avoid the pitfalls, a non-linear mapping can be considered.\n3. Although the paper shows improvements, it does not ground it. For example, what does it mean to have a 2% relative improvement on en-it in comparison to the state of the art method. The paper can benefit from a small section on error analysis or comparison with an existing SOTA model to outline what the improvement is.", "questions_to_authors": "1. Line 213: Between noes I and j, the negative log of the euclidean distance is computed. Can the authors explain the reasoning here?\n2. Line 371-373: Why are words with low frequency filtered out?\n3. Line 391: If anchor points that are far off have a high coefficient of 0.4, will the softmax output (when scaled) be high too? If baseline values (points which are far away) are high, then the probability will closer to a uniform distribution.\n4. In Table 2: Why does the GAN model fail if there isn’t a GNN? \n5. In Table 3: What does “without p” mean? Is it set to 1?", "paper_topic_and_main_contributions": "This paper proposes a GAN based approach to align word embedding spaces for the Bilingual Lexical Induction (BLI) task. The paper proposes a graphical convolutional network (GCN) to learn the topology of the embedding space in a given language and then utilizes a GAN to learn cross space correlations. They showcase the performance of the method of a diverse set of datasets.\n\n*Main Contributions*\n1. The paper proposes a new approach for Bilingual Induction between language pairs. They use a GCN to learn intra-space topology and a GAN to map two spaces together.\n2.  Additionally, they propose a pairwise local mapping algorithm to train learn embeddings based on their nearest neighbors.\n3. The paper has experiments on 16 languages pairs.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 066: induction -> inductive\n2. Line 398: Is PLM a mapping function or a loss?  The authors can can clarify if the PLM is a mapping function or an optimization criteria to learn a mapping. The text in the paper and equation appear to be different.\n3. The interpretation of “_” in Line 494 and Table 1 caption is different. The authors can clarify which one to use.\n4. Line 511: The authors can add information if the improvements are absolute or relative.\n5. Line 539: “hold” -> “held”."}}]

Author rebuttal:

Meta-review: The paper proposes a new method for word-embedding-based bilingual lexicon induction (BLI) using generative adversarial networks (GAN). The way GANs are used is taken from previous work. The main innovation is using a graph convolutional network to inform word embeddings about the topological structure of the embeddings, i.e., about its neighbors in the space (although one review disputed the terminology, claiming it is not really GCN).

Overall, the reviewers mainly criticized somewhat weak experimental results and somewhat limited novelty. During the discussion period, the reviewers raised their scores from rather negative to rather positive.
Decision: Accept-Findings