DM-BLI: Dynamic Multiple Subspaces Alignment for Unsupervised Bilingual Lexicon Induction (2024)

Abstract: Unsupervised bilingual lexicon induction (BLI) task aims to find word translations between languages and has achieved great success in similar language pairs. However, related works mostly rely on a single linear mapping for language alignment and fail on distant or lowresource language pairs, achieving less than half the performance observed in similar language pairs. In this paper, we introduce DMBLI, a Dynamic Multiple subspaces alignment framework for unsupervised BLI. DM-BLI improves language alignment by utilizing multiple subspace alignments instead of a single mapping. We begin via unsupervised clustering to discover subspaces in source embedding space. Then we identify corresponding subspaces in the target space using a rough global alignment. DM-BLI further employs intra-cluster and inter-cluster contrastive learning to refine precise alignment for each subspace pair. Experiments conducted on standard BLI datasets for 12 language pairs (6 richresource and 6 low-resource) demonstrate substantial gains achieved by our framework. We release our code at https://github.com/huling2/DM-BLI.git.

heading: 1 Introduction
text: Unsupervised bilingual lexicon induction (BLI) has shown to be a key multilingual NLP task to align cross-lingual word embeddings (CLWEs) (Mikolov et al., 2013a; Ruder et al., 2019) and bridge lexical gap between low and rich-resource languages (Eder et al., 2021; Marchisio et al., 2022).
Existing BLI approaches can be roughly divided into two categories: mapping-based methods (Conneau et al., 2017; Artetxe et al., 2018; Ren et al., 2020; Li et al., 2022) and generation-based methods (Gonen et al., 2020; Ghazvininejad et al., 2023; Li et al., 2023). Mapping-based methods aim to
∗*Corresponding author
align monolingual embeddings from various languages into a shared CLWEs space via linear or non-linear projections. Generation-based methods leverage the machine translation capacities of large language models (LLMs) (Briakou et al., 2023) to directly generate word translations via zero-shot or few-shot prompting. Mapping-based methods are superior to generation-based methods in unsupervised settings, especially are far superior on low-resource languages (Li et al., 2023), primarily due to the unbalanced training corpus size of each language supported by LLMs (Zhu et al., 2023a).
The existing fully unsupervised mapping-based approaches still need to carefully address two issues. First, these approaches rely on a strong assumption that monolingual word embedding spaces are isomorphic and the mapping matrices should be under orthogonal constraint, but this assumption does not hold true for all languages (Søgaard et al., 2018; Glavaš et al., 2019), especially for distant language pairs (Ormazabal et al., 2019; Vulić et al., 2019). Therefore, weak orthogonal constraints have been proposed to tackle this issue (Mohiuddin et al., 2020; Glavaš and Vulić, 2020).
Second, a global mapping matrix does not consistently perform optimally across all subspaces
2041
(Nakashole, 2018; Wang et al., 2020). As shown in Figure 1, subspaces exhibit inconsistent structural similarity. With a global orthogonal mapping, BLI accuracy varies among different subspaces: the highest accuracy is 54.3% in subspace 0 and the lowest accuracy is 6.89% in subspace 5. To alleviate the issue, recent research proposed a multiadversarial learning method (Wang et al., 2020) and a graph-based paradigm (Ren et al., 2020) to learn or refine a specific mapping for each subspace. However, in these approaches, multiple subspaces assigned by initial mappings are static. Once initial solutions of these mappings are not good enough, they may get stuck in poor local optima.
Different from previous methods, we propose a Dynamic Multiple subspaces cross-lingual alignment framework for fully unsupervised Bilingual Lexicon Induction, named DM-BLI. It leverages intra-cluster and inter-cluster contrastive learning to achieve precise alignment at subspace level for both source and target languages, along with dynamically updating the subspace assignment of each word. DM-BLI starts by clustering the embeddings of source language to establish multiple valid subspaces. Then, we induce an initial solution to discover corresponding multiple subspaces in the target language. Finally, we iteratively refine a pair of specific mappings for each subspace pair until convergence is reached.
In summary, we make the following contributions:
• We propose a dynamic multiple subspaces cross-lingual alignment framework for the BLI task, which achieves customized mappings for each subspace pair.
• To boost the performance of our model, we design a contrastive learning framework including intra-cluster and inter-cluster level based on unsupervised clustering to dynamically update the subspace assignment, avoiding falling into local optima.
• We conduct extensive experiments to demonstrate the effectiveness of our method on twelve language pairs including six richresource and six low-resource language pairs, and DM-BLI achieves significant improvements especially for distant and low-resource language pairs.

heading: 2 Related Work

heading: 2.1 Cross-lingual Word Embedding
text: Bilingual lexicons can be induced via nearest neighbour retrieval on CLWEs, which represent lexical words from two or more languages in a shared space.
Based on whether parallel corpora are used or not, CLWEs approaches can be categorized into three groups: supervised (Faruqui M, 2014; Zou W Y, 2013; Vulić I, 2015), semi-supervised (Artetxe M, 2017; Patra et al., 2019), and unsupervised approaches (Conneau et al., 2017; Artetxe et al., 2018). Because parallel corpora are not available for many languages, unsupervised approaches gain much more attention.
But unsupervised methods do not require any seed dictionary at all, it is more difficult to induce a reliable initial solution which plays a crucial role in alignment. Therefore, GAN-based adversarial training (Zhang et al., 2017), optimal transport solution (Alvarez-Melis and Jaakkola, 2018), Autoencoder (Mohiuddin and Joty, 2019), and graphbased alignment (Ren et al., 2020) were utilized to better match embedding distribution and find a better initial solution in a fully unsupervised way.
Based on the type of pre-trained monolingual embeddings, CLWEs can be divided into two groups: static CLWEs and contextual CLWEs. Most works focused on static word embeddings (Ruder et al., 2019), which can be derived by Word2Vec (Mikolov et al., 2013b) or fastText (Bojanowski et al., 2016). However, static embeddings lack contextual information to capture polysemy. Therefore, contextual embeddings, generated from monolingual and multilingual pre-trained language models (Devlin et al., 2019; Lample and Conneau, 2019), were utilized as input monolingual embeddings. However, they cannot surpass static embedding in the BLI task based on the same mapping technologies even with much more training time (Vulić et al., 2020; Liu et al., 2021).

heading: 2.2 Bilingual Lexicon Induction
text: Bilingual lexicon induction (BLI) aims to derive word translations from monolingual corpora in two different languages. The performance of BLI is heavily impacted by language differences, with significant variation across different language pairs.
BLI methods tend to perform well on semantically similar and resource-rich language pairs but struggle with distant or low-resource language
pairs. For example, unsupervised BLI accuracy on English-Spanish exceeded 80%, while under 40% on English-Chinese (e.g. Conneau et al., 2017; Wang et al., 2020; Ren et al., 2020). Therefore, there is an increasing interest in addressing the challenges of distant or low-resource language pairs alignment.
However, there is no standardized criterion for defining low-resource language pairs. For instance, Zhang et al. (2022) used language frequency on Twitter as a criterion, while Goyal et al. (2022) classified languages into four types based on their bitext resource levels with English. Compared to high-resource languages like English, Spanish, and Chinese, many languages face similar challenges as extremely low-resource languages despite having a decent amount of resources. Additionally, CLWEs often perform poorly for truly low-resource languages due to the inferior quality of embeddings (Michel et al., 2020). As a result, most previous studies have focused on relatively low-resource languages like Finnish and Hindi rather than on absolutely low-resource languages (Mohiuddin et al., 2020; Tian et al., 2022).
To address challenge on disatant and lowresource language pairs, Taitelbaum et al. (2019) suggested leveraging auxiliary languages to bridge the gap between semantically distant and lowresource language pairs. Based on the observation that words are naturally grouped into different semantic subspaces and the BLI accuracies of different subspaces are not uniform, Wang et al. (2020) proposed a multi-adversarial learning method to learn a specific mapping for each subspace. However, this GAN-based method was less robust and its assignment of subspaces was fixed initially which would bring the noise of initial solution.
Different from previous work, we propose a dynamic multiple subspaces alignment framework for unsupervised BLI to achieve more robust and precise alignment at subspace level for both source and target languages, along with dynamically updating the subspace assignment of each word.

heading: 3 Methodology

heading: 3.1 Formulation
text: Let X ∈ RN∗d and Y ∈ RM∗d be the normalized pre-trained monolingual embeddings for source and target languages, where N and M denote the number of words and d denotes the vector dimension. Our goal is to find the optimal mapping ma-
trices WX and WY , facilitating the mapped embeddings XWX and YWY to be in a shared CLWEs space.
Figure 2 illustrates the four procedural steps of our BLI method: multiple subspaces clustering on the source language, initial alignment, intracluster and inter-cluster contrastive refinement, and bilingual lexicon induction.

heading: 3.2 Multiple Subspaces Discovery
text: Multiple subspaces discovery contains the first two steps in Figure 2: multiple subspaces clustering and initial alignment. It aims to discovery subspaces pairs {Xi, Yi} from the source embeddings X and target embeddings Y , where i = 1, 2...K and K is the number of subspaces.
Firstly, multiple subspaces clustering is only carried on source language embedding X to obtain K subspaces, denoted as X = {X1, X2, ..., XK}. Xi ∈ RNi∗d represents the i-th subspace, where Ni is the number of words in Xi. A major challenge in multiple subspaces clustering is to determine the optimal number of subspaces in advance. To tackle this issue, we use a parameter-free hierarchical clustering called First Integer Neighbor Clustering Hierarchy (FINCH) (Sarfraz et al., 2019) to provide a reference number K. Then, K-means algorithm (MacQueen et al., 1967) is used to cluster X into K subspaces.
Secondly, an initial alignment is conducted for identifying corresponding K subspaces in the target language Y , denoted as Y = {Y1, Y2, ..., YK}. Yi ∈ RMi∗d represents the i-th subspace, where Mi is the number of words in Yi. To be detailed, we operate the initial alignment following (Artetxe et al., 2018) to get a pair of global initial mapping matrices WXinit and WYinit , with which we can retrieve the translation of each target word in the source language. Subsequently, the subspace index of the target word is set to be the subspace index of its translation.

heading: 3.3 Multiple Subspaces Contrastive
text: Refinement
A single global mapping does not consistently perform optimally across all subspaces (Nakashole, 2018; Wang et al., 2020). Therefore, the proposed framework will dynamically refine matrices for each subspace pair. This framework contains both inter-cluster and intra-cluster contrastive learning. Inter-cluster contrastive learning ensures the
distinguishability of features among different subspaces pairs, thereby facilitating more effective customized mapping. Intra-cluster contrastive learning brings translation pairs within the subspace pair closer together, while push non-translation pairs further apart, thus achieving finer-grained alignment. The whole refinement process will be completed subspace by subspace.

heading: 3.3.1 Inter-cluster Contrastive Learning
text: Given the subspace pair {Xi, Yi}Ki=1, inter-cluster contrastive learning aims to bring the whole subspaces Xi closer to Yi, while pushing it away from other non-corresponding subspaces Yj,j ̸=i.
We introduce optimal transport distance as the metric to evaluate distance of two subspaces distribution, in our work Wasserstein distance (Han et al., 2022) has been applied. Compared to simple distance metrics like Euclidean distance, the Wasserstein distance considers the overall structure of probability distributions, making it robust to outliers and capable of capturing geometric nuances more effectively. The Wasserstein distance between the distributions of two subspaces can be calculated as:
Dw(Xi, Yi) = min
Ni∑
j=1
Mi∑
k=1
T jkc(w Xi j , w Yi k ) (1)
where c(wXij , w Yi k ) is the transport cost between words wXij ∈ Xi and wYik ∈ Yi, and T jk represents the transport plan between wXij and w Yi k .
Based on the K pairs of subspaces, we calculate a bi-direction inter-cluster contrastive learning loss as follows:
Ls2t = − 1
K
{ log (e−Dw(Xi,Yi)/τ )
+ ∑
j ̸=i log (1− e−Dw(Xi,Yj)/τ )
}
Lt2s = − 1
K
{ log (e−Dw(Yi,Xi)/τ )
+ ∑
j ̸=i log (1− e−Dw(Yi,Xj)/τ )
} (2)
where τ is a temperature parameter. To be specific, the aforementioned process is applied to the sampled distribution of subspace, where the proportion of samples is determined by a preset threshold.
Finally, we obtain the final inter-cluster contrastive loss Linter as below, where λ is the tradeoff set to be 0.5 between two directions:
Linter = λ ∗ Ls2t + (1− λ) ∗ Lt2s (3)

heading: 3.3.2 Intra-cluster Contrastive Learning
text: Given the subspace pair {Xi, Yi}Ki=1, intra-cluster contrastive learning is to ensure word pair (wXij , w Yi k ) are closer, which are translations to each other in Xi and Yi. Based on the mapping matrices WX and WY , we can initially construct a bilingual dictionary D by retrieving the translation of each target word in the source language, where D ={ (wXi1 , w Yi 1 ), (w Xi 2 , w Yi 2 ), ..., (w Xi l , w Yi l ) } and l is the number of translation pairs in D. However, the quality of D depends on the quality of mapping matrices. To alleviate the noise brought by the current solution, we selectively sample high-confidence word translation pairs from D, where confidence is determined by the similarity gap between the selected translation and the second candidate translation with the source word.
Based on the sampled translation pairs Ds, the intra-cluster contrastive learning loss can be defined as:
Lintra = − |Ds|∑
i=1
log esim(w
x i ,w y i )/τ
∑|Ds| j=1 esim(w x i ,w y j )/τ (4)
where |Ds| is the number of sampled translation pairs and τ is a temperature parameter. Ultimately, the loss of the whole contrastive refinement can be defined as follows:
L = Linter + Lintra (5)

heading: 3.4 Multiple Subspaces Dynamic Updating
text: A single round of subspace assignment may introduce noise from the initial solution, potentially causing CLWEs to fall into local optima. Therefore, we propose to dynamically adjust the subspace assignment of each word in target language during the process of updating WX and WY .
To clarify, the assignment of multiple subspaces in source language X = {X1, X2, ..., XK} is fixed once the clustering process is completed. For word wYi in target language, its translation from source language wXi is retrieved based on XWX and YWY . The subspace index of wXi will be assigned to wYi . Upon updating WX and WY , the subspace assignment of wYi will be adjusted accordingly to maintain consistency whenever its translation changes.
As we mentioned before, the whole refinement process will be operated subspace by subspace. For
each subspace Yi in target language, the whole dynamic updating procedure stops until convergence is reached. Convergence can be determined by measuring the overlap of target words within Yi between the current and previous rounds. Besides, once a subspace has achieved convergence, its assignments are finalized, ensuring that the words in this subspace remain unchanged. The whole methodology is summarised in Algorithm 1.
Algorithm 1: Dynamic Multiple Subspaces Alignment for Unsupervised BLI
Input: Monolingual word embedding spaces X , Y
Output: {WXi}Ki=1, {WYi}Ki=1 1 {Xi}Ki=1 ← Apply Clustering on X; 2 WXinit ,WYinit ← Initial Alignment ; 3 {Yi}Ki=1 ← Calculate XWXinit , Y W Yinit ; 4 for i ≤ K do 5 Initialize WXi = WXinit , WYi = WYinit while not convergence do 6 WXi ,WYi ← Optimize loss L
Xi ← Keep Xi fixed Yi ← Update Yi with WXi ,WYi
7 return {WXi}Ki=1, {WYi}Ki=1;

heading: 4 Experiment Setup
text: We evaluate our framework in both supervised and unsupervised BLI tasks on 12 language pairs, which contain 6 rich-resource language pairs: Spanish (ES), German (DE), Russian (RU), Arabic (AR), Japanese (JA) and Chinese (ZH), all cross-lingual to English (EN) and six low-resource language pairs: Finnish (FI), Hindi (HI), Turkish (TR), Indonesian (ID), Bulgarian (BG) and Catalan (CA), all cross-lingual to English (EN). Following previous research (Mohiuddin et al., 2020; Tian et al., 2022), we select relatively low-resource language pairs as low-resource setting rather than truly lowresource languages.

heading: 4.1 Dataset
text: We use fastText vectors trained on Wikipedias (Bojanowski et al., 2016) as monolingual word embeddings. We use the widely used MUSE bilingual lexicon (Conneau et al., 2017), released by Facebook, as ground truth lexicon. MUSE provides 110 bilingual lexicons and each lexicon contains the 6,500 most frequently used words in each language,
split in a test set of 1,500 words and a training set of 5,000.

heading: 4.2 Baselines
text: Baselines are divided into supervised and unsupervised two lines as described below. We run the released code of each baseline in our experiments. Supervised BLI •MUSE: Conneau et al. (2017) learned an orthogonal map by minimizing the Euclidean distance between the supervised translation pairs. • VecMap: Artetxe et al. (2018) used a multi-step framework consisting of several steps: whitening, orthogonal mapping, re-weighting, de-whitening, and dimensionality reduction. • BLISS: Patra et al. (2019) proposed a semisupervised approach with a weak orthogonality constraint in the form of a back-translation loss. • CL-BLI: Li et al. (2023) proposed a robust and effective two-stage contrastive learning framework to combine static and contextual embeddings. Unsupervised BLI • MUSE: Unsupervised MUSE (Conneau et al., 2017) used adversarial training and iterative Procrustes refinement. • VecMap: Unsupervised VecMap (Artetxe et al., 2018) used intra-linguistic word similarity information to induce initial solution. • Ad. : Mohiuddin and Joty (2019) proposed a adversarial auto-encoder framework, where adversarial mapping was done at the latent embedding space. • BLOOM7B (Workshop et al., 2022): It is a decoder-only Transformer language model that supports 46 natural languages. 7B parameters version was used in our experiment. • Llama13B (Touvron et al., 2023): It is a decoderonly LLM which supports 20 languages. 13B parameters version was used in our experiment. • GPT-3.5 (Brown et al., 2020): It is a decoderonly LLM with 175B parameters, supported by 38 languages. GPT-3.5-turbo was used in our experiment.

heading: 4.3 Implementation details
text: We choose the most 75,000 frequent vocabularies of each language. The normalization procedure for pre-trained embedding contains three steps: length normalizes the embeddings, then mean centers each dimension, and then length normalizes them again.
For multiple subspaces discovery, the number of subspaces is set to be 9 and we will discuss the impact of this setting later. For inter-cluster contrastive learning, only words with weight above 0.45 are sampled to represent the subspace distribution. For intra-cluster contrastive learning, we only sample the top 20% of word translation pairs sorted descending by confidence.
Following the previous research (Patra et al., 2019), the prompt template for Llama13B is defined as: "Translate from Lx to Ly: wx=>"; the prompt template for GPT-3.5 is defined as: "Translate the Lx word wx into Ly:". Both of them are provided as the best template for each of them in Li et al. (2023).
The evaluation for BLI is done by comparing the bilingual lexicon constructed by each model with the benchmark lexicon MUSE (Conneau et al., 2017) and reporting precision Precision@N for N = 1, 5. Precision@N accounts for accuracy for which the correct translation of the source words is in the N -th nearest neighbors based on CSLS (Conneau et al., 2017).

heading: 5 Result and Discussion

heading: 5.1 Results in low-resource Languages
text: Table 1 summarizes the results of the supervised and unsupervised BLI tasks in low-resource language pairs. In both tasks, our proposed method shows significant improvements, particularly in Precision@5, with an average of 2.16 points higher than the strongest baseline VecMap in the supervised task. In the unsupervised task, our method performs nearly as well as the strong baseline GPT3.5.
In the supervised task, DM-BLI outperforms all the baseline methods on all language pairs, demonstrating the robustness and effectiveness of our framework on low-resource language pairs. In the unsupervised task, DM-BLI outperforms all the baseline methods on four out of six language pairs and archives suboptimal scores in the remaining pairs at Precision@5. It demonstrates that our method is competitive even compared with GPT-3.5, which has 175B parameters and supports 38 languages. The unsatisfied performance of BLOOM7B and Llama13B also suggests that the generalization of LLMs to low-resource languages remains an open challenge.

heading: 5.2 Results in Rich-resource Languages
text: Table 2 summarizes the main results of the supervised and the unsupervised BLI tasks on richresource language pairs.
In supervised tasks, our proposed method achieves significant improvements, with average nearly 1 point higher than the strongest baseline CL-BLI. We achieve the optimal or sub-optimal performance on all the language pairs. Notably, our method achieves a 6.26% improvement over CL-BLI on distant language pairs Japanese to English, demonstrating advantages of multiple sub-
space alignment on distant language pairs.
In unsupervised tasks, DM-BLI achieves the sub-optimal result on rich-resource language pairs. While it outperforms the previous mapping-based SOTA method VecMap but underperforms GPT-3.5. The outstanding performance of GPT-3.5 verifies the potential of the latest generation of LLMs for developing bilingual lexicons with sufficient training and a large amount of parameters. However, BLOOM7B and Llama13B are still far lagging behind the traditional mapping-based method even on rich-resource language pairs, which verifies that it
is difficult to extract lexical information from large language models (Liu et al., 2021).

heading: 5.3 Influence of the Number of Subspaces
text: In this section, we discuss the impact of the number of subspaces on performance of DM-BLI, taking distant language pair JA2EN as an example.
As shown in Figure 3, compared with VecMap who only use a global mapping, our method lets word with same meaning from different languages get much closer in a shared CLWEs space via multiple subspace-level alignments.
Notably, from Figure 3, we can find that even using different numbers of subspaces, DM-BLI still achieved nearly the same results, which shows that it is not sensitive to the number of subspaces and further proves the robustness of our method.

heading: 5.4 Influence of Translation Direction
text: In this subsection, we examine how the translation direction affects BLI results in unsupervised setup. The language pairs we choose as examples are Japanese (JA), Chinese (ZH), Finish (FI), Turkish (TR), from and to English (EN), as shown in Table 3.
From Table 3, we observe the performance differences in the two directions of the language pair. Specifically, the results from English to other languages significantly lag behind those from other languages to English. A part of the reason is that there are more unique English words than nonEnglish words in the evaluation set (Xu et al., 2018).
It also proves that LLMs exhibit unbalanced capacities across languages, performing better at translating into English than translating into non-English (Zhu et al., 2023b).

heading: 5.5 Effect of Multiple Subspaces Alignment
text: Notice that our method focuses on leveraging multiple subspace alignments to achieve better performance for BLI. In this subsection, we discuss the advantages of multiple subspaces alignment from our method DM-BLI, taking low-resource language pair CA2EN as an example.
As shown in Figure 4, on low-resource language pair like CA2EN, we can find that BLI accuracies for all subspaces based on DM-BLI are higher than the strongest mapping-based baseline VecMap. Notably, we also find that unbalanced alignments occur in a generative way via GPT-3.5 as well. Furthermore, LLM’s capability on BLI is still far lagging behind mapping-based approach.
In order to show effect of DM-BLI more intuitively, we sample 2 subspaces for visualization. As shown in Figure 5, via multiple subspaces alignment, translation pairs within the subspace stay closer together than applying a global mapping.

heading: 6 Conclusion
text: In this paper, we propose a Dynamic Multiple subspaces alignment framework for unsupervised BLI, called DM-BLI. Our method utilizes multiple subspaces alignment instead of a single mapping alignment to achieve more accurate alignment on the subspace level. The experiments show that our method can significantly improve the bilingual word induction performance compared with strong baselines even including GPT-3.5, especially for distant and low-resource language pairs. At the same time, the unsatisfied performances of BLOOM7B and Llama13B on all language pairs also suggest that it is difficult to extract lexical information from large language models and the generalization of LLMs to low-resource languages remains an open challenge. In the future, we will consider combining our method with multilingual LLMs to take advantage of these two paradigms.
Limitations
First, public BLI datasets are not enough to support a comprehensive evaluation. In the evaluation standard dictionary, the proportion of ground-truth translations in different categories is uneven. As also discussed in (Li et al., 2023), current evaluation will not work for words that are not included in the gold translations.
Second, we choose the relatively low-resource languages rather than truly low-resource languages in the setting, which fails to address the problem in real low–resource scenarios. Truly low-resource languages will be considered to increase the applicability of proposed framework in our future work.

heading: Future Work
text: In the future, we will consider combining our method with multilingual LLMs to take advantage of these two paradigms. Limitations
First, public BLI datasets are not enough to support a comprehensive evaluation. In the evaluation standard dictionary, the proportion of ground-truth translations in different categories is uneven. As also discussed in (Li et al. , 2023), current evaluation will not work for words that are not included in the gold translations. Second, we choose the relatively low-resource languages rather than truly low-resource languages in the setting, which fails to address the problem in real low–resource scenarios. Truly low-resource languages will be considered to increase the applicability of proposed framework in our future work