UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages (2024)

Abstract: In this paper, we introduce UniBridge (CrossLingual Transfer Learning with Optimized Embeddings and Vocabulary), a comprehensive approach developed to improve the effectiveness of Cross-Lingual Transfer Learning, particularly in languages with limited resources. Our approach tackles two essential elements of a language model: the initialization of embeddings and the optimal vocabulary size. Specifically, we propose a novel embedding initialization method that leverages both lexical and semantic alignment for a language. In addition, we present a method for systematically searching for the optimal vocabulary size, ensuring a balance between model complexity and linguistic coverage. Our experiments across multilingual datasets show that our approach greatly improves the F1-Score in several languages. UniBridge is a robust and adaptable solution for cross-lingual systems in various languages, highlighting the significance of initializing embeddings and choosing the right vocabulary size in cross-lingual environments.

heading: 1 Introduction
text: Recently, multilingual pre-trained language models (LMs) have significantly advanced natural language processing (NLP) tasks, narrowing the performance gap between English and various other languages. Multilingual pre-trained models such as XLM-R (Conneau et al., 2020) and mBERT (Devlin et al., 2019) are currently strong models for effectively cross-lingual transfer (Hu et al., 2020; Artetxe et al., 2020; Le et al., 2024). However, these models pose a limitation that they are pretrained on a limited set of approximately 100 languages, leaving a substantial void for the vast array of the world’s nearly 7000 languages (van Esch et al., 2022). The resultant disparity disproportionately affects low-resource languages that are
* Equal Contribution. †Corresponding author.
not covered in their pre-trained corpora (Wu and Dredze, 2020; Pfeiffer et al., 2020), impeding their performance compared to their high-resource counterparts.
Recent efforts propose the use of adapters to mitigate the knowledge gap in low-resource languages prior to transferring knowledge for specific tasks (Pfeiffer et al., 2020; Üstün et al., 2020; Ansell et al., 2021). These methods adapt the pre-trained LMs to a new language by utilizing monolingual data, enabling the model to acquire a robust representation of the target language before receiving knowledge from the source language. Despite enhanced performance in languages not included in the pre-trained corpora, these approaches still exhibit poor performance in languages with unseen scripts (i.e., the scripts that are not presented in the pre-training corpora; see Figure 1). To address the issue of unseen scripts, existing studies (Artetxe et al., 2020; Pfeiffer et al., 2021) propose acquiring a new vocabulary embedding for newly discovered languages. However, these methods heavily rely on manually configuring the vocabulary size and initializing the embedding matrix.
Furthermore, recent Cross-Lingual Transfer Learning studies focus on English due to its abundant pre-trained data and impressive task performance, our experiments reveal that high performance in English tasks does not necessarily guar-
3168
antee successful transfer to other languages, particularly low-resource languages. Therefore, we suggest an automated method utilizing the LMs to identify the most suitable set of source languages for knowledge aggregation, leading to notable performance improvements over single-source language transfer.
Our research empirically tested the effectiveness of newly random initialized embeddings and fixed vocabulary size. We then introduce an efficient technique for determining the optimal vocabulary size for new languages, utilizing the syntactic and semantic insights from the pre-trained LMs. In addition, we present an innovative method for transferring knowledge from multiple sources, which allows the model to choose the best combination of source languages to improve the overall performance. Our results contribute to the ongoing discussion about managing linguistic diversity in NLP, particularly for languages with limited resources, emphasizing the importance of a detailed and inclusive strategy in creating multilingual pre-trained LMs.
We evaluate our approach on sequence tagging tasks (e.g. NER, POS) and classification (e.g. NLI) with two strong baselines, mBERT and XLM-R, and observe a significant increase in the F1 and accuracy score 1. In summary, our contributions are:
• We propose a novel approach to automatic search for a suitable vocabulary size to adapt to a new language. • We propose a new strategy to initialize the embedding that leverages the syntactic and semantic knowledge encoded in the pre-trained LMs to address the missing tokens when adapting to low-resource languages. • We propose a method to aggregate multisource transfer learning to enhance the performance on cross-lingual transfer tasks. We show that multi-source can outperform effective multi-language learning.

heading: 2 Methodology
text: Our proposed framework includes five stages as illustrated in Figure 2. In the following section we will detail each stage of the framework: 1) Vocabulary size searching, 2) Language-specific embedding initialization, 3) Model adaptation to new
1https://github.com/TokisakiKurumi2001/ UniBridge
languages not covered in the pre-training data, 4) Downstream task training, 5) Multi-source transfer downstream task inference.

heading: 2.1 Vocabulary size searching
text: Whether training from scratch or starting with a pre-trained language model, every NLP practitioner faces the task of determining the appropriate vocabulary size. Thus, choosing a suitable vocabulary size requires exhaustive searching (i.e., the whole training and testing process is required to determine the best vocabulary size). For UniBridge, the vocabulary is determined by using only CPU and is not time-consuming as it does not require any language model training phases. This is achieved by leveraging the average log probability (ALP, Zheng et al. (2021)). The algorithm for vocabulary size searching is illustrated by Algorithm 1.
Algorithm 1 Vocabulary size searching algorithm. Require: D: monolingual data, contains a list of
words sentences, vi: initial vocabulary size, vm: maximum vocabulary size that the system should not exceed, δv: increased step of vocabulary size, ϵs: a threshold for stopping the algorithm.
1: v ← vi 2: t← build tokenizer with vocab size v on D 3: sprev ← ALP (D, t) 4: ∆s =∞ 5: while ∆s > ϵs do 6: v ← v + δv 7: if v > vm then 8: v ← vm 9: t← build tokenizer with v on D
10: Break the loop 11: else 12: t← build tokenizer with v on D 13: scurr ← ALP (D, t) 14: ∆s = scurr − sprev 15: sprev ← scurr 16: end if 17: end while 18: return Tokenizer t with vocab size v
The concept of Average Log Probability (ALP) was introduced by Zheng et al. (2021), who argue that ALP is related to the effectiveness of subsequent tasks.
ALP (D, t) = 1|t(D)|
|t(D)|∑
j=1
|sj |∑
k=1
logpuni(skj ) (1)
For more details, readers are advised to refer to the work of Zheng et al. (2021). It is worth noting that although ALP has a high correlation with downstream tasks, the work did not provide a solution to find an optimal vocabulary size. Therefore, in this work, we propose using the ‘degree of changes’ in the ALP score, e.g., ∆s. Initially, the starting vocabulary size is chosen and the ALP is calculated. Through a series of increases in the vocab size by δv, we can calculate the difference between the current ALP and the previous one. Thus, the algorithm will stop when the difference reaches a specific threshold ϵs. This threshold indicates that the optimal vocabulary size has been obtained. Continuing to increase the size will result in similar or worse performance. Therefore, we stop the algorithm to maintain the efficiency of training. Additionally, our method stands out from traditional grid search by using the ‘degree of changes’ of the ALP score indirectly, rather than directly as in grid search.

heading: 2.2 Language-specific embedding initialization
text: When training for a new language, using a randomly initialized embedding can lead to prolonged training times for optimal performance, especially in low-resource settings with a dataset size of around 10K samples. In such cases, strategically initializing the embedding proves to be more effective than a random approach. While FOCUS (Dobler and de Melo, 2023) demonstrates the use of a pre-trained LM’s embedding for initialization, it depends heavily on a simple lexical overlapped alignment for subsequent stages, thus decreasing the downstream task performance. To address this gap, our approach initializes the new embedding by leveraging the pre-trained LMs in both syntactic and semantic aspects. In the initial stage, we obtain the target tokenizer tT for the new language, with tS being the source tokenizer of the pre-trained LMs. Representing the vocabulary sets as V T and V S for the target and source tokenizers respectively, and embedding matrices as ET [·] and ES [·], we copy the source embedding to the target embedding for the overlapping tokens OL = V T ∩ V S . This method ensures a seamless
integration of knowledge from the pre-trained LMs, addressing both syntactic and semantic aspects of the new language’s embedding initialization.
∀o ∈ OL : ET [o] = ES [o] (2)
Although the number of lexical overlapping tokens can be substantial when utilizing the same script, such as Latin or Han, this phenomenon does not extend to unseen scripts. To address this challenge, we define the non-lexical alignment set as ALT = V
T \ OL and initiate a search for semantically aligned tokens within this set. Despite languages having different scripts, the underlying meanings often converge on similar definitions. To facilitate this alignment, we train two static embeddings—one for the source tokenizer (FS) and another for the target tokenizer (F T ) —using the monolingual dataset D. These embeddings are denoted as FS [·] for the source tokenizer and F T [·] for the target tokenizer. For each token vi in ALT , we calculate the cosine similarity with every token vj in ALS = V
S \ OL, resulting in a matrix Si,j ∈ R|ALT |×|ALS |. A pair of semantically aligned tokens (vi, vj) is defined as a pair of source and target tokens whose embeddings exhibit the highest cosine similarity score to each other, or:
i = argmax l (Sl,j) and j = argmax l (Si,l) (3)
Refer to Equation 3, we define S = {(i, j)|i = argmax
l (Sl,j) and j = argmax l (Si,l)}. Each token that is semantically aligned will have the embedding copied from their counterpart from the source embeddings.
∀(i, j) ∈ S : ET [i] = ES [j] (4)
For the remaining non-aligned tokens, AT = ALT \ Si and AS = ALS \ Sj where Si, Sj is the set of semantically aligned token of the target and source vocabulary (i.e. Si = {i|(i, j) ∈ S}, Sj = {j|(i, j) ∈ S}), we initialize the target embedding using the weighted sum of the aligned target tokens. We compute the cosine similarity between each non-aligned token aT ∈ AT and the set of aligned target tokens (comprising both lexical and semantically aligned tokens) oT ∈ OL ∪ Si.
ca,o = F T [aT ]F T [oT ] ⊤
∥F T [aT ]∥ · ∥F T [oT ]∥ (5)
To obtain the most similar aligned symbols oT for a single symbol aT , we use the same approach in Dobler and de Melo (2023), using sparsemax (Martins and Astudillo, 2016) over ca, where ca is a vector containing all similarity scores from ca,o. Sparsemax is a variant softmax, but it assigns zero to low-probability element. By this, we can overcome the problem posted by skew distribution where some tokens has only one or two similar tokens while others have more. The weight wa,o for each aligned token oT as defined in Equation 6.
wa,o = sparsemaxo(ca) (6)
We denote Sa as a set of similar aligned tokens, which contains oT whose probability is non-zero assigned by sparesemax.
Sa = {oT ∈ OL ∪ Si|wa,o > 0} (7) Using the set Sa and the weight wa,o, the embedding for the non-aligned token aT is calculated as the weighted sum of its most similar aligned tokens.
∀aT ∈ AT : ET [aT ] = ∑
oT∈Sa waT ,oTE
T [oT ] (8)

heading: 2.3 Model adaptation to new languages & Downstream task training
text: Continual pre-training, also known as language adaptation, has proven to be an effective method for enhancing the downstream performance of zeroshot cross-lingual tasks, as demonstrated by studies such as Ke et al. (2023); Alabi et al. (2022); Ebrahimi and Kann (2021). To mitigate the environmental impact and reduce model storage requirements, we opt to pre-train only a portion of the model, aligning with the approach introduced in MAD-X (Pfeiffer et al., 2020).
As in Figure 2, we made some modifications to the MAD-X configuration. Firstly, we initialize a new embedding for UniBridge which is achieved from previous stages and train the embedding together with adapters while still freezing all the pretrained LMs’ parameters. Secondly, we propose using the KL divergence together with the MLM loss (Appendix A). We see that although the frozen parameter in each layer of the pre-trained LMs helps guide the trainable adapters of the new language’s embedding representation into the same pre-trained LM’s embedding space, MLM is not sufficient as it only enforces the adapter to predict
the mask token and this cannot guarantee the new language’s representation is the same as multilingual embedding space encoded by the pre-trained LMs. This limitation prohibits the knowledge transferability of task adaptation since task adaptation takes a source language (usually high-resource languages such as English, Chinese, etc) and transfers the task knowledge directly to the target language without any alignment between the two languages. Therefore, we use KL divergence as a regularizer to guide the model not only to learn the language representation well, but also to maintain the same space as the source language in order to achieve better transferability.
L = LMLM (y, ŷ) + βDKL(πUniBridge(h|x)∥πPLM (h|x)) (9)
y and ŷ are the ground truth and prediction logits of the mask prediction task, respectively. πUniBridge(h|x) is the last hidden state of UniBridge, it is the output of the invertible adapter before goes to the linear classification head for masked predicting. πPLM (h|x) is the last hidden state of the pre-trained LMs, it is the output of the last Transformer layer, as in Figure 2, and is the input of the linear classification head for mask predicting.

heading: 2.4 Multi-source transfer downstream task inference
text: Instead of using one task adapter from one source language, we propose aggregating the knowledge from multiple source languages to derive a better result. For each target language, we compute the harmony weight or similarity distance between languages. Some libraries such as Lang2Vec (Malaviya et al., 2017) provide a similarity score between languages. However it does not cover all the languages. To overcome this problem, we directly use the language model (that UniBridge produced from previous stages) to measure the similarity between languages. In the Appendix D.2, we will provide a detailed comparison between Lang2Vec and UniBridge. This analysis will highlight the differences and similarities between the two approaches, offering insights into their respective performances and effectiveness.
For each target language, we collect K samples of parallel sentences from datasets such as Tatoeba (Tiedemann, 2020) or FLORES-200 (Guzmán et al., 2019; Goyal et al., 2022; Team et al., 2022)
between the target language and a set of N source languages.
We denote DT as a monolingual dataset extracted from the parallel dataset on the target side, DSi is the monolingual dataset extracted from the parallel dataset on the source side of the i-th source language. Each sentence is fed into the UniBridge with the corresponding language adapters and obtains a set of hidden states (i.e., output from the invertible adapter).
Hl = {πlUniBridge(s)|s ∈ Dl} (10)
πlUniBridge is the UniBridge model which use the l adapter; Dl is DT for the target language and DSi for the i-th source languages. The inverse L2 distance between the target hidden state Ht for target language t and source hidden state Hs for source language s will be computed.
dt,s = 1
L2-norm(Ht, Hs) (11)
After that, we compute the softmax over the inverse L2 distance to gain the harmony weight between target language t and set of source languages S = {si}Ni=1.
wt = softmaxs(dt,s) (12)
Using this harmony weight, instead of replacing the task adapter for each different source language during inference like MAD-X, we forward through all the task adapters in parallel. The last logit prediction will be the weighted sum of all the logits predicted by each source language weighted by the harmony weight.
ŷ = ∑
s∈S wt,sŷs (13)
ŷs is the logit prediction from source language s.
The intuition behind the harmony weight is that given a pair of parallel sentence, each sentence is encoded by a different language model. The difference between the hidden states produced by this process turns out to be the difference between languages itself since the sentences convey the same meaning. Therefore, inversing the difference and applying softmax will result in the similarity that we can up-weight for languages, and they could be beneficial to the target language on downstream tasks and, at the same time, down-weight
the languages that are distant from the target language. Through our experiment, we show that multi-source inference outperforms single-source transfer and multi-language learning settings.

heading: 3 Experimental setup
text: Language set: The set of source languages are: English, Chinese, Russian, Arabic and Japanese. For the target languages, we evaluate 14 low-resource languages from WikiANN (Rahimi et al., 2019) whose training set consists of only 100 samples per language, 9 low-resource languages from Universal Dependencies (UD) (Zeman et al., 2023) whose training set consists of just few thousands samples per language and 10 languages from the AmericasNLI (Ebrahimi et al., 2022).
Monolingual data: For the language adaptation part, we extract from the Wikipedia dataset from HuggingFace 2 10K samples for simulating the low-resource settings, each sample consists of 128 words, for each target language. For source languages, the number of samples is 50K per language to simulate the rich-resource languages. For languages in AmericasNLI, we use one side of the translation dataset from Mager et al. (2021).
Tokenizer: We use the SentencePiece (Kudo and Richardson, 2018) to learn the token from the monolingual data with the vocab size determined by our Algorithm 1.
Downstream data: NER: We train UniBridge on the train split of WikiANN for all the source language sets and perform inference for the target language on the test split. POS: We train UniBridge from the train split of UD for all the source languages sets. NLI: We train UniBridge from the train split of XNLI (Conneau et al., 2018) for English, Chinese, Arabic and Russia due to the missing Japanese set.
Baseline: We evaluate UniBridge against the MAD-X framework and zero-shot cross-lingual fine-tuning using pre-trained language models (LMs). In the zero-shot approach, we fine-tune the entire pre-trained LM on the combined training data of all source languages and then directly infer on the target languages. With MAD-X, we adhere to its standard setup, training on monolingual data. To perform multi-language training, we combine training data from all source languages to train a “universal” task adapter. For inference, we swap
2https://huggingface.co/datasets/graelo/ wikipedia
the language adapter for each target language and integrate the “universal” task adapter. For UniBridge, we implement the language adaptation and task training stages as detailed in Section 2.3. During inference, we combine the task adapters from 5 source languages for multi-source transfer and report the F1 score for NER and accuracy score POS, NLI on the target language’s test split.
The hyperparameters for training, inference as well as the computational resources are given in Appendix C.

heading: 4 Results and Analysis
text: We present the result of our method and the baselines in Table 1 and 2 for NER task and Table 3 and 4 for POS tagging task. We report the NLI results in Table 12 and 13 in Appendix D.1. UniBridge outperforms strong baselines such as whole model fine-tuned (XLM-R, mBERT) and MAD-X framework by a large margin, i.e, for the XLMR model, we outperform 11 over 14 languages. For POS tagging task, we outperform both baselines with two different backbone models. We also see this trend in NLI task (Appendix D.1). This highlights the effect of leveraging multiple source languages during inference to help make better decisions since each language contributes knowledge that benefits the model at prediction. Meanwhile, multi-training offers a more robust performance but also introducing more difficulties during training. The fact that UniBridge outperforms strong baselines such as whole fine-tuned model indicates that given a small monolingual and lightweight adaptation using adapters, we can significantly improve the cross-lingual tasks for uncovered languages. Compared to MAD-X, our approach differs from the use of a new embedding layer. For unseen languages, using a more specific layer of embedding can remarkably enhance the performance. Even though MAD-X already uses the invertible adapters as a component to adapt embedding layer to unseen languages, these components may not sufficient for rare languages with unseen scripts such as Amharic (amh), Khmer (khm), Kanada (kan). In addition, to evaluate UniBridge with large (decoder-style) Language Models (LLMs), we expanded our experiments beyond XLM-R and mBERT to include mGPT (Shliazhko et al., 2024) and mBART (Liu et al., 2020). This extension provides a more robust assessment of UniBridge’s effectiveness across different model types, demonstrating its versatility
and potential in leveraging various LLM architectures for improved language representation. The results are presented in Appendix D.8, showcasing the comparative performance and strengths of UniBridge in diverse settings.
Although UniBridge can successfully improve cross-lingual generalization, there are still some inconsistencies in the performance of a language on different tasks, e.g., Amharic (amh), Ligurian (lij), and Sanskrit (san) on NER and POS tasks. We hypothesize that the inconsistency arises from the misalignment in the subspace between the language adapter and the task adapter. One approach to mitigate this misalignment is to regularize the representation so that the newly learned representation is shared between the source and target languages. UniBridge leverages KL divergence as a regularization approach. This may not be sufficient to completely resolve the inconsistency, but given our constrained resources, KL divergence fits our requirements well. We leave other advanced methods, such as optimal transport or contrastive learning, for future work.

heading: 5 Ablation study

heading: 5.1 Contribution of each component
text: We study the contribution of each UniBridge component independently to investigate the critical components of each module. To remove KL divergence, we simply remove the KL loss from equation 9, keeping only MLM loss. To remove the embedding initialization component, we randomly initialize the embedding drawn from the Xavier normal distribution (Glorot and Bengio, 2010). To remove the vocab size search component, we fix the vocab size to 10k for every target language and
use SentencePiece (Kudo and Richardson, 2018). To remove the multi-source transfer, we consider English as the single source language transferred due to its wide use in many cross-lingual transfer works.
We report the mean and standard deviation of the F1 scores between 14 languages of 2 backbone models when applying UniBridge and the components removed from UniBridge in Table 5, details of each language can be found in Appendix D.3. Among components, embedding initialization plays the most critical role since removing it, we experience performance drops of about 39 and 20 for XLM-R and mBERT, respectively. For multi-source transfer component, mBERT experiences a larger drop with 11 F1 drop while XLM-R is down from 45 to 42. However, the standard deviation when removing multi-source transfer is larger than that of UniBridge (XLM-R), indicating that multi-source benefits more languages compared to single language transferred. Although removing KL divergence off the XLM-R improves its performance by 1 F1 score, the standard deviation increases by 1 score. Thus, KL divergence benefits languages in maintaining a more stable improvement among languages. On the other hand, removing KL divergence while using mBERT as a backbone model hurts the performance and drops 3 F1 scores. In order to clarify the effectiveness of KL-Divergence in the other model, we conducted experiments in Appendix D.4. Vocab size searching with dynamic vocab size significantly improves the performance for mBERT backbone with an improvement of the 7 F1 score. This implies that different languages should be applied differently and dynamically to best adapt to their linguistic features.

heading: 5.2 Vocabulary size
text: In this section, we contrast our approach with a novel technique for vocabulary initialization called EXTEND (Wang et al., 2020). EXTEND operates by initially expanding mBERT’s vocabulary to accommodate the new language and then proceeding with pre-training on this language. In our comparison, EXTEND undergoes full fine-tuning for the MLM pre-training task. Subsequently, EXTEND is further fine-tuned using the monolingual data of each target language. Despite its extensive fine-tuning and high computational requirements, EXTEND does not perform satisfactorily on NER in comparison to UniBridge, as illustrated in Table 19 in Appendix D.5. UniBridge offers a much lighter and faster alternative, employing adapters for cross-lingual transfer learning. The lightweight and rapid nature of UniBridge significantly enhances the effectiveness of our method. Furthermore, we present an elaborate Table 20 containing various vocabulary sizes for each target language in the Appendix D.6. Regarding the lexical similarity of subwords in the vocabulary, we offer illustrations of subwords that exhibit similarity in both mBERT and XLM-R. These examples can be
found in Figure 4 within Appendix B.

heading: 5.3 ALP Threshold
text: We conducted experiments using different ALP thresholds to identify the most effective one. We tested threshold values such as 5.0, 10.0, and 15.0 during the pre-training process of UniBridge. In essence, raising the threshold led to a decrease in vocabulary size as the algorithm ended prematurely. As a result, we noticed a decrease in the F1-Score of mBERT and XLM-R as the threshold values increased, as illustrated in Figure 3.
More experiments and ablation study can be found in Appendix D.

heading: 6 Related works
text: Dynamic vocabulary size: It is common among NLP practitioners that the vocabulary size is considered a hyperparameter and requires manual settings. Algorithms such as BPE (Gage, 1994), Word-
Piece (Wu et al., 2016), SentencePiece (Kudo and Richardson, 2018) focus more on how to build a set of predefined number of tokens that statistically retrieved from the monolingual dataset. Some work such as VoCAP (Zheng et al., 2021), XLM-V (Liang et al., 2023) proposed algorithm to dynamically assign a vocabulary size for each language during the multilingual training. However, in monolingual training, there are some few works concerning this problem such as BPE-dropout (Provilkov et al., 2020), VOLT (Xu et al., 2021) learns to have an optimal vocab size via reducing the original vocab size using optimal transport as in VOLT or randomly removing merge in BPE-dropout.
Initialization: Artetxe et al. (2020) proposed to randomly initialize the new embedding for new language adaptation. Meanwhile, Wang et al. (2020), Chau et al. (2020), Pfeiffer et al. (2021) leverage the lexical similarity between the old vocabulary and the new vocabulary to initialize the embedding. On the other hand, there are works that explore the semantic space for initialization. SMALA (Vernikos and Popescu-Belis, 2021) directly finds the aligned token through the highest cosine similarity score. WECHSEL (Minixhofer et al., 2022) and FOCUS (Dobler and de Melo, 2023) use static embedding to find aligned tokens.
Multi-source transfer: Single-source transfer, especially, English-as-the-source-language receives many attentions. Artetxe et al. (2020), Ansell et al. (2021), Tu et al. (2022) leverages the multilingual backbone model, fine-tune on English downstream task and perform zero-shot transfer on target language test’s set. Until recently, researchers have pointed out that using a multilingual training set is more beneficial compared to a single language. DeMuX (Khanuja et al., 2023) investigates the dataset level to accumulate examples that best benefit transferring using active learning. Dossou et al. (2022), Ogunremi et al. (2023) pre-train on the multilingual African dataset before distilling knowledge to downstream tasks.

heading: 7 Conclusion
text: In this paper, we investigate Cross-Lingual Transfer Learning, focusing on languages with constrained resources. Our contribution lies in an algorithm that autonomously determines the optimal vocabulary size for a new language, informed by its monolingual corpus, and an innovative method for initializing a new embedding matrix, drawing from
both semantic and lexical facets of the pre-trained language models. Additionally, we introduce a novel technique for aggregating multi-source transfer learning, enhancing the efficacy of cross-lingual transfer tasks. Our empirical tests demonstrate the adaptability of our method across different models, yielding significant enhancements in performance. A thorough investigation of key elements highlights UniBridge’s effectiveness in various situations, offering an in-depth understanding of the robustness of our approach.
Limitation
UniBridge is trained on the extracted Wikipedia with some heuristic noise filtering. However, we believe that further pre-processing such as language identification and noise filtering pipeline could further produce higher-quality monolingual data, which potentially improve the language adaptation stage. UniBridge incorporates the use of adapter to perform cross-lingual generalization, while this leverages the modular characteristic of adapter, it also inherited some limitation of the adapter itself (Kunz and Holmström, 2024; Alabi et al., 2024).

heading: A Why KL Divergence and MLM Loss work?
text: For the KL-Divergence effect, in contrast to MADX, UniBridge incorporates a novel training embedding. This results in the pre-task adapter representation being more inclined to reflect the characteristics of the target language compared to solely using the language adapter in MAD-X. Consequently, this introduces a misalignment between the language adapter and the task adapter, as each represents a different language.
In our research, we opt for KL-Divergence to regulate the representation to ensure it is shared across both the source and target languages (Kim et al., 2021). KL-Divergence requires less computational resources compared to other methods like layer-wise regularization or optimal transport.
For the MLM Loss effect, it is extremely effective in training encoder-only LMs because it encourages the model to learn rich contextual representations of language and facilitates effective pre-training.
In MLM, a portion of the input tokens is randomly masked, and the model is trained to predict these masked tokens based on the context provided by the surrounding tokens. This forces the model to learn contextual representations of words and phrases of that target language (Sinha et al., 2021). Moreover, by randomly masking tokens, MLM introduces noise into the training process, which can prevent overfitting and encourage the model to learn more generalizable features of the data.

heading: B Similar tokens between pre-trained LM and UniBridge specific tokenizer
text: We illustrate the similar tokens between pre-trained LM and UniBridge specific tokenizer in Figure 4.

heading: C Computational resources and hyperparameter for training, inference
text: All experiments are conducted on T4 machines. Training the UniBridge’s language adapter takes approximately 2.5 hours on a single T4 machine with a batch size of 16. Separately, training UniBridge’s task adapter, takes about 0.5 hours per source language on a single T4 machine with a batch size of 16.
We present the hyperparameters for training and inference for UniBridge and all the baselines’ configurations in Table 6, 7, 8, 9, 10, and 11.

heading: D More experiments and ablation study
text: D.1 Performance of UniBridge on NLI task
We report the performance of UniBridge on the AmericasNLI dataset in Tables 12 and 13.
.
D.2 UniBridge v.s. Lang2Vec
Our method’s reliance on parallel data enables it to capture typological similarities as well as syntactic and semantic relationships between languages. By utilizing parallel sentences, we can develop more nuanced representations that reflect the intricacies of language structures and meanings.
Moreover, the quality and coverage of typological databases can be inconsistent. Although these databases are available for many languages, they often lack completeness and accuracy. In contrast, parallel corpora, while more challenging to obtain, provide direct evidence of language similarities and differences in real-world contexts. Additionally, our method has shown superior performance compared to Lang2Vec in the experiments conducted on the WikiANN dataset in Table 14 and 15.
D.3 Detail performance of each factor
We present the detailed performance of UniBridge on 14 languages on NER task when removing the contributed components in Table 16 and 17.
D.4 Effectiveness of KL Divergence
In contrast to MAD-X, UniBridge employs a novel training embedding, leading to a pretask adapter representation that better captures the characteristics of the target language than solely using the language adapter in MAD-X. To ensure that the output representation is shared between both source and target languages, we use KL-Divergence. This approach is less computationally intensive than methods such as layer-wise regularization or optimal transport (Section 6).
To assess the effectiveness of using KLDivergence within UniBridge, we conducted extensive tests on an alternative Language Model, such as mBART, using the WikiANN dataset. The results in Table 18 indicate that KL-Divergence significantly contributes to the overall performance of UniBridge, enhancing its effectiveness considerably.
D.5 UniBridge v.s. EXTEND
We report the result on NER task compared between UniBridge and EXTEND method in Table
19.
D.6 Vocabulary searching result of UniBridge We report the searched size of the Algorithm 1 for each language in Table 20.
D.7 UniBridge v.s. FOCUS We compared UniBridge initialization and FOCUS initialization. For UniBridge, the whole pipeline is kept the same as discussed in the paper. For FOCUS (Dobler and de Melo, 2023), we replace step 2 discussed in Section 2.2 with FOCUS initialization pipeline while other steps are kept the same as UniBridge. We report the results on NER task in Table 21 and Table 22.
UniBridge surpasses FOCUS in performance across 10 out of 14 languages and 9 out of 14 languages on WikiANN. Among these languages, approximately 10-15% of the tokens exhibit semantic alignment. We theorize that UniBridge’s advantage lies in its ability to leverage these aligned tokens, which facilitates a smoother and quicker convergence during the subsequent MLM training phase compared to FOCUS initialization.
D.8 UniBridge with Large Language Models To evaluate UniBridge with large (decoder-style) Language Models (LLMs), we extended our experiments to include mGPT and mBART, alongside XLM-R and mBERT. This broader assessment demonstrates UniBridge’s versatility and effectiveness across different model types. The results, presented in Table 23, highlight the strengths of UniBridge in diverse settings.

heading: Future Work
text: We leave other advanced methods, such as optimal transport or contrastive learning, for future work