TRANSLICO: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024)

Abstract: The world’s more than 7000 languages are written in at least 293 scripts.1 Due to various reasons, many closely related languages use different scripts, which poses a difficulty for multilingual pretrained language models (mPLMs) in learning crosslingual knowledge through lexical overlap. As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally. To address this problem, we propose TRANSLICO, a framework that optimizes the Transliteration Contrastive Modeling (TCM) objective to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script (in our case Latin2), which enhances uniformity in the representation space for different scripts. Using Glot500-m (ImaniGooghari et al., 2023), an mPLM pretrained on over 500 languages, as our source model, we fine-tune it on a small portion (5%) of its training data, and refer to the resulting model as FURINA. We show that FURINA not only better aligns representations from distinct scripts but also outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks. Additionally, we achieve consistent improvement in a case study on the Indic group where the languages exhibit areal features but use different scripts. We make our code and models publicly available.3

heading: 1 Introduction
text: In recent years, mPLMs have made impressive progress in various crosslingual transfer tasks (Conneau et al., 2018; Hu et al., 2020; Liang et al., 2020). Such achievement is mainly due to the
*Equal contribution. 1https://worldswritingsystems.org/ 2Throughout this paper we use Latin to refer to the Latin
script, not the Latin language. 3https://github.com/cisnlp/TransliCo
availability of monolingual corpora of many languages (Costa-jussà et al., 2022; Adebara et al., 2023; ImaniGooghari et al., 2023), the amelioration of model architectures suitable for scaling up (Vaswani et al., 2017; Peng et al., 2023), as well as the advancement of self-supervised learning objectives (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020). Despite the fact that mPLMs present attractive performance in high-resource languages, those models often gain unsatisfactory results for low-resource languages, especially when the writing systems or scripts are different from the transfer source languages (Muller et al., 2021).
This undesired behavior is related to the script barrier in the representation space, where different scripts are located in different subspaces (Wen-Yi and Mimno, 2023). To tackle this problem, transliteration or romanization4 is leveraged in some recent work (Dhamecha et al., 2021; Muller et al.,
4Romanization is a specific type of transliteration that involves converting non-Latin scripts into the Latin script.
2476
2021; Moosa et al., 2023; Purkayastha et al., 2023): all languages from different scripts are converted into one common script and the language model is pretrained or adapted with transliterated data. For testing and inference, the queries also need to be transliterated, as the model only supports one script, the pretraining or adaptation script of the model.
However, this line of approaches presents two limitations. First, it does not break the script barrier, rather, it circumvents it. The representations from different scripts are still not aligned. Second, for some tasks, e.g., question answering, it is necessary to transliterate the response back to the original script because we cannot assume that end users know the common script. Unfortunately, transliteration and transliterating back to the original script is not immune to information loss (Amrhein and Sennrich, 2020). The romanized words in many languages, e.g., Chinese, Japanese, and Korean, can be converted to different words in their original scripts, which unfortunately leads to ambiguity.
In this paper, we present TRANSLICO, a contrastive learning framework to address the script barrier in the representation space of mPLMs in a way that overcomes the limitations of prior work. To start with, a small portion of the data from the pretraining corpus of an mPLM is used to generate Latin transliteration, using Uroman5 (Hermjakob et al., 2018). Then we create paired data using sentences in their original script and their transliterations. The data is subsequently used by the two objectives: Masked Language Modeling (MLM) and Transliteration Contrastive Modeling (TCM). MLM is applied to both the original sentences and their transliterations; we use TCM to learn betteraligned representations by contrasting the positive pairs (paired data) against negative pairs (the remaining in-batch samples) as shown in Figure 1.
Using Glot500-m (ImaniGooghari et al., 2023) as our source model, we evaluate TRANSLICO both “globally” and “locally”. Specifically, we fine-tune Glot500-m on 5% of its pretraining data of all languages and refer to the resulting model as FURINA. We show that FURINA aligns representations from different scripts better and it generally outperforms the baselines on sentence retrieval, sequence labeling, and text classification tasks for different script groups. Our ablation study indicates MLM and TCM in TRANSLICO are both important for achieving good crosslingual performance. We ad-
5https://github.com/isi-nlp/uroman
ditionally conduct a case study on Indic languages, a group of languages that show areal features and use different scripts. FURINAIndic fine-tuned by TRANSLICO using the data from Indic languages shows consistent improvement over the baseline.
The main contributions of this work are summarized as follows: (i) We present TRANSLICO, a simple but effective framework, to address the script barrier in the representation space of mPLMs. (ii) We conduct extensive and controlled experiments on a variety of crosslingual tasks and show TRANSLICO boosts performance. (iii) We show the framework encourages the representations from different scripts to be better aligned. (iv) In a case study on Indic languages, we demonstrate that TRANSLICO also works for areal languages that have shared vocabulary but use distinct scripts.

heading: 2 Related Work
text: Transliteration refers to converting languages from one script into another script (Wellisch et al., 1978). Transliteration can increase lexical overlap, and therefore it has been shown to substantially improve the performance of neural machine translation for low-resource languages of different scripts (Gheini and May, 2019; Goyal et al., 2020; Amrhein and Sennrich, 2020). Several studies also demonstrate that transliteration can enhance the crosslinguality of mPLMs across various dimensions. For instance, Dhamecha et al. (2021) transliterate seven Indo-Aryan family languages into Devanagari and show that common-script representations facilitate fine-tuning in a multilingual scenario. Purkayastha et al. (2023) show that, by transliterating into Latin, better performance is achieved when adapting mPLMs to new languages, particularly for low-resource languages. More recently, Moosa et al. (2023) focus on Indic languages and show that models directly pretrained on transliterated corpora in the Latin script achieve better performance. However, the downside is that the model only supports one script and loses the ability to deal with the scripts in which the languages were originally written. This is not optimal when we expect predictions or generations in the original scripts. In contrast to this line of work, we aim to directly break the script barrier, instead of circumventing it by limiting the model to one common script. We use transliterations in our finetuning framework to improve the alignment across different scripts: the model after fine-tuning still
supports the scripts it originally did. Contrastive learning is a method for learning
meaningful representations by contrasting positive pairs against negative pairs (Chopra et al., 2005; Hadsell et al., 2006). This type of approach has achieved great success in learning visual representations (Schroff et al., 2015; Oord et al., 2018; Chen et al., 2020; He et al., 2020). Contrastive learning also demonstrates its effectiveness in NLP, especially for learning sentence representations (Gao et al., 2021; Zhang et al., 2022; Wu et al., 2022b; Zhang et al., 2023). One major problem in contrastive learning is how to construct contrastive pairs. For a monolingual scenario, depending on specific downstream tasks, the positive pairs are usually constituted through data transformation or data augmentation strategies (Zhang et al., 2020; Yan et al., 2021; Wu et al., 2022a; Xu et al., 2023a) whereas the negative pairs are typically the remaining in-batch samples (Xu et al., 2023b). In a multilingual scenario where parallel data is available, translations from different languages can be used to construct the positive pairs (Reimers and Gurevych, 2019; Pan et al., 2021b; Chi et al., 2021; Wei et al., 2021). Unfortunately, large parallel corpora are mostly available for high-resource languages. Therefore, aiming to improve crosslinguality, especially for under-represented languages, we start from the perspective of the script and construct positive pairs by using sentences in their original script and their Latin transliterations that can be easily obtained. Then we fine-tune an mPLM with our contrastive framework TRANSLICO. In this way, our work also resembles some post-pretraining alignment approaches (Pan et al., 2021a; Feng et al., 2022; Ji et al., 2023) that fine-tune a PLM using token-level or sentence-level translations.

heading: 3 Methodology
text: We present TRANSLICO, a simple framework to address the script barrier by fine-tuning a PLM on a small portion of the data that is used to pretrain the model. The framework consists of two training objectives: Masked Language Modeling (Devlin et al., 2019) and Transliteration Contrastive Modeling. We illustrate our framework in Figure 2 and introduce our training objectives in the following.

heading: 3.1 Masked Language Modeling
text: The MLM training objective is to take an input sentence X = [x1, x2, · · · , xn], randomly replace
a certain percentage (15% in our case) of tokens by [mask] tokens, and then train the model to predict the original tokens using an MLM head. Formally, let H = [h1,h2, · · · ,hn] be the contextualized representations at the last layer of the Transformer model (Vaswani et al., 2017) (the output of the last Transformer block of the model) given the input sentence X . Following the notations used by Meng et al. (2021), we compute the MLM loss as follows:
LMLM = E [ − ∑
i∈M log pMLM(xi|hi)
]
where M is the set of masked positions in the input sentence X and pMLM(xi|hi) is the probability of outputting the original token giving hi from the vocabulary V , computed by the MLM head.
Instead of only performing MLM for the sentences in their original scripts, we also perform MLM for their transliterations in Latin script: X trans = [x1, x2, · · · , xm], which are obtained by using Uroman (Hermjakob et al., 2018). The MLM loss for transliteration data is referred to as LtransMLM. By doing this, we can improve the crosslinguality of the model across related languages that use different scripts, as transliteration has shown to be effective in capturing morphological inflection (Murikinati et al., 2020) and generating shared subwords between related languages (Muller et al., 2021; Dhamecha et al., 2021; Moosa et al., 2023). The intuition is that, as all sentences are consistently transliterated into Latin using the same tool, this can bring about vocabularies that have more shared subwords that are originally in different scripts. The improved lexical overlap therefore encourages the model to generate more crosslingual representations through MLM objective.

heading: 3.2 Transliteration Contrastive Modeling
text: Modeling a sentence and its transliterations separately does not necessarily lead to a good alignment between two types of scripts. Therefore, we propose to learn more similar and robust representations of a pair of sentences in different scripts using the contrastive learning objective. Sentence-level contrastive learning generally aims to align a positive pair of sentences by distinguishing them from negative or unrelated samples of sentences (Gao et al., 2021; Meng et al., 2021; Zhang et al., 2023). In our framework, we simply let a positive pair be a sentence in its original script and its transliteration in the Latin script, and other sentences in a training batch be the negative samples to contrast with.
Formally, let a training batch for TCM objective be B = {Xorig1 , X latn1 , · · · , X orig N , X latn N }, where N is the batch size, Xorigi is the ith sentence in its original script and X latni is its Latin transliteration. Similar to the setting of Meng et al. (2021), a positive pair (X,X+) consists of a symmetrical contrast pair, i.e., both (Xorigi , X latn i ) and (X latn i , X orig i ), whereas the negative samples are all the remaining sentences in their original scripts and their transliterations in the training batch using a slightly abusing notation: B− = B \ {X,X+}. Then the contrastive loss is defined as follows:
LTCM = E [ − log exp(sim(h,h +)/τ)
exp(sim(h,h+)/τ) + NEG
]
where NEG = ∑
X−∈B− exp(sim(h,h −)/τ),
sim(·) is the similarity measure (cosine similarity is used), τ is the temperature (set to 1 by default), h, h+ and h− are the representations of X , X+ and X− respectively, which are computed by mean pooling the output of the 8th layer. Choosing the 8th layer is based on previous empirical findings, as the first layers are weak in terms of crosslinguality whereas the last layers are too specialized on the pretraining task (Jalili Sabet et al., 2020; Chang et al., 2022). The numerator improves the alignment, i.e., encouraging the model to assign similar representations to similar samples, while the denominator improves the uniformity, i.e., encourag-
ing the representations to be uniformly distributed on the unit hypersphere (Wang and Isola, 2020).
Since all sentences are expected to have representations similar to their Latin transliterations by the contrast training objective, this implicitly encourages sentences in different scripts to be in the same subspace. Intuitively, the Latin script acts as a bridge to connect all other scripts, and therefore all other scripts are better aligned. We show better script-neutrality is achieved in the representation space compared with the original mPLM in §5.2.

heading: 3.3 Overall Training
text: The overall training objective of TRANSLICO is then the sum of the MLM loss (from the original data and the transliteration data) and the TCM loss:
Ltraining = λ1LMLM + λ2LtransMLM + λ3LTCM
where λ1, λ2 and λ3 are the weights for each loss. Following (Meng et al., 2021), we set λ1 = λ2 = λ3 = 1, as we also find the initial losses from each part during training are in similar magnitude. By fine-tuning an mPLM with this overall training objective, the model is expected to (1) not forget the language modeling ability gained in its pretraining phase; (2) be able to model sentences in both their original scripts and in their Latin transliterations and (3) learn to better align representations from different scripts in the same subspace.

heading: 4 Experiments

heading: 4.1 Setups
text: We use Glot500-m (ImaniGooghari et al., 2023), a continued pretrained model from XLM-R (Conneau et al., 2020), as our source mPLM. The training data of Glot500-m is Glot500-c, which contains 1.5B sentences from 511 languages and 30 scripts (534 language-scripts6 in total). For each languagescript, we randomly select 5% sentences from its training set in Glot500-c as the training data. We then concatenate these sentences from all languagescripts and use Uroman (Hermjakob et al., 2018), a tool for universal romanization, to transliterate them into the Latin script. Finally, our training data consists of around 75M pairs (a pair is a sentence in the original script and its Latin transliteration). Examples of Uroman transliteration are shown in Table 1. Note that we also include the sentences originally in Latin script and perform transliteration for them. This is because we want to (1) preserve the model’s ability to model languages in Latin script (2) increase lexical overlap by including data where diacritics are removed (done by Uroman) and (3) improve the overall robustness of the model. We show in §5.1 how the model finetuned in this setting outperforms the model finetuned without Latin data. The fine-tuned model by using the proposed TRANSLICO framework is referred to as FURINA. See §A for detailed hyperparameters. Except for the evaluation performed on the original datasets discussed in the main content below, we also evaluate the resulting models on the transliterated datasets. That is, we use Uroman to transliterate the datasets from the original script to the Latin script, and then perform evaluation on the new datasets. The performance on transliterated datasets is shown and discussed in §D.

heading: 4.2 Downstream Tasks
text: Sentence Retrieval. Two datasets are considered: Tatoeba (Artetxe and Schwenk, 2019) (SR-T) and Bible (SR-B). We select up to 1,000 Englishaligned sentences for SR-T, following the same setting used by Hu et al. (2020); and up to 500 sentences for SR-B. We report the top-10 accuracy for both tasks. Following Jalili Sabet et al. (2020), the similarity is calculated by using the average of contextualized word embeddings at the 8th layer.
6A language-script is a combination of the ISO 639-3 code and the script.
Language-script XLM-R Glot500-m FURINA Language-script XLM-R Glot500-m FURINA Language-script XLM-R Glot500-m FURINA afr_Latn 89.3 87.7 86.7 ajp_Arab 63.0 73.1 70.1 aln_Latn 54.1 49.7 54.2 amh_Ethi 63.9 65.9 63.6 ara_Arab 67.9 65.7 65.3 bam_Latn 25.1 39.5 49.7 bel_Cyrl 86.0 85.7 85.2 ben_Beng 82.0 83.9 82.2 bre_Latn 61.3 60.2 67.8 bul_Cyrl 88.6 88.3 87.3 cat_Latn 86.6 86.9 86.7 ceb_Latn 50.1 66.3 68.8 ces_Latn 84.4 84.4 83.4 cym_Latn 65.8 64.7 65.3 dan_Latn 90.3 90.1 90.7 deu_Latn 88.4 87.9 87.3 ell_Grek 88.0 83.9 82.1 eng_Latn 96.3 96.1 96.1 est_Latn 85.9 82.5 83.7 eus_Latn 71.2 61.1 63.6 fao_Latn 77.6 89.1 89.4 fas_Arab 70.3 71.3 72.1 fin_Latn 85.1 80.3 82.4 fra_Latn 85.9 86.4 87.1 gla_Latn 58.4 60.2 60.6 gle_Latn 66.1 64.8 65.5 glg_Latn 82.7 83.7 84.2 glv_Latn 27.2 52.7 54.0 grc_Grek 64.7 72.6 70.8 grn_Latn 10.5 20.1 27.4 gsw_Latn 49.1 81.0 82.4 hbo_Hebr 40.3 50.0 49.5 heb_Hebr 67.5 68.3 67.7 hin_Deva 73.2 71.2 75.7 hrv_Latn 85.2 85.4 83.6 hsb_Latn 72.1 84.0 83.7 hun_Latn 82.3 81.1 81.3 hye_Armn 84.7 83.9 84.4 hyw_Armn 79.0 81.7 82.7 ind_Latn 83.7 83.3 83.2 isl_Latn 84.4 82.8 83.7 ita_Latn 87.4 89.2 89.1 jav_Latn 73.4 73.7 74.3 jpn_Jpan 14.8 34.9 23.7 kaz_Cyrl 77.2 75.2 76.4 kmr_Latn 73.5 75.4 77.7 kor_Hang 53.6 52.5 52.6 lat_Latn 75.6 70.6 71.4 lav_Latn 85.8 82.6 83.9 lij_Latn 47.0 77.3 77.4 lit_Latn 84.2 80.2 81.8 lzh_Hani 14.5 19.4 7.4 mal_Mlym 86.3 84.6 84.9 mar_Deva 82.5 83.3 84.7 mlt_Latn 21.5 80.1 81.5 myv_Cyrl 39.2 64.3 68.3 nap_Latn 58.8 66.7 88.9 nds_Latn 57.3 76.5 77.8 nld_Latn 88.6 88.3 88.3 nor_Latn 88.3 87.5 87.4 pcm_Latn 46.7 57.9 58.2 pol_Latn 83.1 83.0 81.3 por_Latn 88.3 88.5 88.8 quc_Latn 28.7 62.0 64.3 ron_Latn 83.6 80.2 79.7 rus_Cyrl 89.0 88.8 88.0 sah_Cyrl 22.3 77.6 77.5 san_Deva 19.1 24.8 21.9 sin_Sinh 58.5 55.9 55.5 slk_Latn 84.1 84.6 83.5 slv_Latn 78.1 75.8 75.3 sme_Latn 29.8 73.9 74.6 spa_Latn 88.2 88.9 88.3 sqi_Latn 78.5 77.4 77.0 srp_Latn 85.8 84.8 83.0 swe_Latn 93.4 92.2 92.4 tam_Taml 75.6 75.0 74.4 tat_Cyrl 45.6 69.5 69.8 tel_Telu 85.7 82.2 82.4 tgl_Latn 73.3 75.0 77.1 tha_Thai 44.3 56.5 49.9 tur_Latn 73.0 70.4 72.4 uig_Arab 68.3 68.1 68.9 ukr_Cyrl 85.5 84.6 83.5 urd_Arab 59.6 65.8 70.2 vie_Latn 70.4 66.8 66.1 wol_Latn 25.6 57.1 61.0 xav_Latn 6.2 17.6 17.0 yor_Latn 22.7 63.2 64.2 yue_Hani 27.7 42.7 23.9 zho_Hani 24.6 44.5 23.4 Table 17: F1 scores of baselines and FURINA on POS. |L| Glot500-m is better Furina is better
SR-B 369 33 336 SR-T 98 43 55 Taxi1500 351 46 305 NER 164 55 109 POS 91 40 51
Table 18
Those who are led by the Spirit of God are children of God.
Chinese original 那些被神的 引的人都是神的儿子。transliteration naxie bei shen de ling yindao de ren dou shi shen de rzi.
Arabic original . é<Ë @ Z A JK. @ Ñë é<Ë @ hðP ÑëXñ ®K áK YË @ ½ JËð @
transliteration awlyek althyn yqwdhm rwh allh hm abna’ allh.
Ukrainian original Котрi бо Духом Божим водять ся, тi сини Божi.transliteration Kotri bo Dukhom Bozhim vodyat sya, ti sini Bozhi.
Slovak original Tí, ktorých vedie Boží Duch, sú Božími synmi.transliteration Ti, ktorych vedie Bozi Duch, su Bozimi synmi.
Table 1: Examples of Uroman transliteration. We select sentences (translations of the sentence “Those who are led by the Spirit of God are children of God.”) from four languages that uses different scripts and transliterate them using Uroman. We notice some important characteristics of Uroman: tones (for Hani script) are not included and diacritics (for Latin script) are removed.
Text Classification. Taxi1500 (Ma et al., 2023), a multilingual 6-class text classification dataset available in more than 1,500 languages, is used. We select a subset of language-scripts supported by the model for evaluation. We report the zero-shot crosslingual performance (in macro F1 scores) using the English train set for fine-tuning and selecting the best model on the English dev set.
Sequence Labeling. Two types of tasks are considered: named entity recognition (NER) and PartOf-Speech (POS) tagging. We use WikiANN (Pan et al., 2017) for NER and Universal Dependencies (de Marneffe et al., 2021), version v2.11, for POS. We report the zero-shot performance (in macro F1 scores) for both tasks.

heading: 4.3 Results and Discussion
text: To better illustrate how the proposed TRANSLICO framework can influence the performance of different scripts, we group all language-scripts by their scripts and report the average performance for each group. The results of XLM-R, Glot500-m, and FURINA are shown in Table 2. We see an overall improvement for FURINA compared with Glot500-m in each task except for SR-T. We conjecture that the sub-optimal performance on SR-T can be related to the domain shift and the small set of languages supported by Tatoeba. Our fine-tuning data consists of sentences of many low-resource languages that come from genre-specific corpora such as the Bible, which can be quite different from Tatoeba, which is more modern in terms of the genre and mostly only supports high-resource languages (70 out 98 languages are high-resource languages).
FURINA performs surprisingly well on ST-B. An overall improvement of 10.9 (from 47.2 to 58.1) is achieved. We also see a consistently large increase for each script group of languages: e.g.,
12.3 for Latin script languages and 8.7 for Cyrillic languages. However, for Hani script (Chinese characters) languages, we see a sudden drop in performance, which can also be seen in other tasks such as SR-T, NER, and POS. We hypothesize that Uroman is suboptimal for Hani script because a great deal of important information is lost in the transliteration: both due to the removal of tones and due to the conflation of semantically different characters that are pronounced identically (e.g., 氦 (helium) v.s 害 (harm), both pronounced hài in Mandarin). In addition, Chinese characters are logograms. Even if the transliteration contains correct tones, the transliterated words potentially lose semantic or contextual nuances and are more prone to ambiguity (Amrhein and Sennrich, 2020), thus resulting in a performance drop. However, note that Hani languages are high-resource languages, so this result does not diminish our hypothesis that transliteration helps low-resource languages.
For other types of tasks, we also see a consistent improvement. In both NER and POS, FURINA achieves better performance than Glot500-m in 4 out of 5 script groups (except for Hani as discussed above). Compared with token-level classification (NER and POS), we see a more prominent in-
crease in sequence-level classification (Taxi1500): the overall F1 score is increased by 6.7 (more than 10%) compared with Glot500-m, and FURINA achieves substantially better performance for each script group. The results indicate that the proposed contrastive learning framework TRANSLICO boosts crosslingual transfer learning, especially for sequence-level tasks, e.g., sentence retrieval and sentence classification. We also evaluate both Glot500m and FURINA under the common script scenario, i.e., transliterating the evaluation dataset to Latin script. See the evaluation in §D.

heading: 5 Analysis

heading: 5.1 Ablation Study
text: We conduct ablation experiments on all five tasks, using the same hyperparameters and Glot500-m as the source model. Specifically, we explore the influence of MLM and TCM objectives in TRANSLICO. In addition, we investigate the importance of incorporating data from Latin script languages and classify the model variants into two groups (with Latin script languages or without). In our preliminary experiments, we also explore the weight of the TCM loss (e.g., 0.1, 0.5, and 1) and find the
influence is very small as the TCM loss quickly goes to a small value for different weights, possibly due to the simplicity of the contrastive task (the initial magnitude is close though). We report the best result for each model variant in Table 3.
The model generally performs worse when any one of the training objectives is missing. This holds for each group. For example, the overall accuracy in SR-B decreases by 8.6 and 13.3 for FURINA-Latn when TCM or MLM is missing. The only exception is the accuracy for “Other” in SR-T. We notice when TCM is missing, the result increases by 5.8 (FURINA-Latn) and 6.2 (FURINA+Latn). The possible reason is as follows. The sentences from languages using different scripts in SR-T (Tatoeba sentences are quite simple) are already aligned pretty well. Additionally fine-tuning the model with TCM on a small portion of data (for many low-resource languages the data is related to the Bible) whose domain is different from the domain of SR-T hurts the performance of underrepresented languages.
Interestingly, we find the introduction of TCM and MLM objectives has a more prominent impact on sequence-level (SR-B, SR-T, Taxi1500) than token-level tasks (NER, POS). For example, for FURINA-Latn, all three variants achieve competitive overall performance on token-level tasks: around 60 and 70 F1 scores on NER and POS respectively. This might suggest that TRANSLICO has a relatively small effect on individual token representations as we use sentence-level contrastive learning. In addition, in sequence labeling, the model may be able to transfer prevalent classes such as verb and noun (ImaniGooghari et al., 2023; Liu et al., 2023) to some extent even without the explicit crosslingual constraint imposed by TCM and MLM.
We also see a consistent improvement when
the Latin script data is incorporated into the finetuning data. Although there is an occasional small decrease for languages using other scripts (e.g., on SR-T and POS) when comparing FURINA+Latn and FURINA-Latn, the increase for Latin script languages is prominent for each task. This is expected, as FURINA-Latn can catastrophically forget the knowledge gained in its pretraining phase (Kirkpatrick et al., 2017). By incorporating the Latin script data and their Uroman transliteration, we can further increase lexical overlap and make the model more robust, since Uroman has a unified mechanism for romanization and removes all diacritics. For example, the word “salón” in Czech will be the same as the French word “salon” after Uroman removing the diacritics on “ó”.

heading: 5.2 Representation Visualization
text: To explore how TRANSLICO manipulates the representation space, we visualize the sentence representations from languages that use different scripts. Specifically, we feed Glot500-m and FURINA with 500 sentences from the SR-B task. To facilitate comparison, we only select 10 high-resource languages. Each language uses one of the 10 dominant scripts in the vocabulary of the models (we use GlotScript (Kargaran et al., 2024) to detect the script of the tokens). The languages are English (Latin), Russian ( Cyrl), Chinese (Hani), Arabic (Arab), Hindi (Deva), Greek (Grek), Korean (Hang), Malayalam (Mlym), and Hebrew (Hebrew). We obtain the sentence representations by meanpooling the contextualized token embeddings. We visualize the representations of the 8th layer (also used for SR-B and SR-T) by projecting them to two dimensions with principal component analysis (PCA) in Figure 3 (1st and 2nd subfigures). We
also compute the pairwise normalized cosine similarity between the centroid of each script group (3rd and 4th subfigures). Additionally, we visualize representations from every layer in Appendix E.
It can be seen that the representations from each script roughly form an individual single cluster for Glot500-m. In contrast, representations only form two major clusters for FURINA, where each cluster has certain related scripts. This is evidence that Glot500-m encodes script-sensitive information and distinct but related scripts are not wellaligned, whereas FURINA has learned better scriptneutrality for the representations of related scripts. The pairwise similarity also supports our argument: e.g., FURINA has higher similarity scores among (1) Latn, Cyrl, and Deva, and (2) Hani, Thai and Hang, where any script in each group is related to the rest of the scripts. This phenomenon indicates that TRANSLICO is effective in addressing the script barrier in the representation space of mPLMs by improving the similarity of the related scripts. Interestingly, we observe linguistic features and geographical proximity can also be related to the representation subspaces. Hani, Thai and Hang (Hangul) are in the same cluster, which might be explained by the fact that Thai, Korean, and Chinese are spoken in adjacent areas and their vocabularies are mutually influenced. With TRANSLICO, such similarity is further exploited and thus their representations are encouraged to be closer.

heading: 5.3 Case Study: Indic Group
text: To further explore TRANSLICO’s performance, we conduct a case study on 12 languages that exhibit areal features such as shared vocabulary. These
languages are mostly from the Indo-Aryan group and are mutually influenced with each other linguistically, historically, and phonologically, but use different scripts (Moosa et al., 2023).
Similar to the previous settings, we use Glot500m as our source mPLM with the training data as the only difference. Specifically, we randomly sample 10% of sentences from Glot500-c (ImaniGooghari et al., 2023) for each of the 12 languages. The details of the resulting fine-tuning dataset are shown in Table 5. We then use Uroman to transliterate these sentences into the Latin script. Subsequently, we create positive paired data using these sentences and their Latin script transliterations. This results in our training data consisting of 16.5M sentence pairs. We use TRANSLICO to fine-tune Glot500-m on the data and refer to the final model as FURINAIndic. Following the similar setting employed by Moosa et al. (2023), we evaluate FURINAIndic with three downstream tasks from IndicGLUE (Kakwani et al., 2020): Wikipedia Sec-
tion Title Prediction (WSTP), Cloze Style Question Answering (CSQA)) and a Named Entity Recognition (NER) task. One major difference between our evaluation and Moosa et al. (2023) is that we always evaluate the languages in their original scripts, whereas Moosa et al. (2023) evaluate on the transliterated data in a unified script (Latin). The details for hyperparameters of each task are reported in §A.4. To test the crosslingual transfer ability of FURINAIndic on the Indic group languages, for WSTP and NER, we fine-tune the model on all languages at once and then evaluate the model on the test set for each language. We use CSQA to test the zero-shot capability of FURINAIndic. We evaluate on F1 for NER and on accuracy for WSTP and CSQA. The results are shown in Table 4.
FURINAIndic outperforms Glot500-m on 8 out of 11 languages on the NER task, with a 0.1 slightly higher average score than Glot500-m. We see a similar small improvement on WSTP, where FURINAIndic beats Glot500-m on most of the languages except for Panjabi. We assume the reason for the small improvement in these two tasks is that the model is fine-tuned on the train set of all languages (not zero-shot crosslingual transfer), which could overshadow the benefits from TRANSLICO. Nevertheless, the general consistent improvement shows the effectiveness of TRANSLICO. For the zero-shot task CSQA, we see a large increase. FURINAIndic improves the average performance by more than 12 points (from 34.0 to 46.4). Although another competitive model, ALBERTUS (Moosa et al., 2023), beats Glot500-m (source model of FURINAIndic), FURINAIndic outperforms ALBERTUS consistently in all languages. This indicates that TRANSLICO enjoys the largest improvements in zero-shot scenarios, which is exactly the desideratum for most low-resource languages. Overall, the consistent improvement demonstrates TRANSLICO not only works well “globally” on all languages but also “locally” on a small group of languages that have lexical overlap but use different scripts.

heading: 6 Conclusion
text: In this work, we propose a novel framework TRANSLICO to fine-tune an mPLM only using a small portion of sentences in its pretraining corpus and their Latin transliteration. The framework contrasts the sentences in their original script and their transliterations to tackle the script barrier problem. Using Glot500-m as our source model, we fine-
tune it using the proposed framework and present the resulting model: FURINA. Through extensive experiments, we show FURINA better aligns the representations from different scripts into a common space and therefore outperforms Glot500-m on a wide range of crosslingual transfer tasks. In addition, we conduct a case study on Indic group languages that are known to be mutually influenced by each other but use different scripts. We show TRANSLICO can also boost the performance for the Indic group languages. We hope this framework can inspire more future work leveraging transliteration to improve crosslinguality of mPLMs.
Limitations
We propose a simple contrastive learning framework TRANSLICO that aims to address the script barrier. We show the effectiveness by using Glot500-m as the source model and fine-tuning it on a small portion (5%) of its pretraining data. We would assume the proposed framework can also be used directly for pretraining, through which the model might benefit further from seeing more data. However, due to a limited computation budget, we aren’t able to pretrain a model from scratch or continued pretrain a model using the full Glot500-c corpus. We would leave out how the proposed framework can be integrated into efficient pretraining or continued pretraining for future work.
We see the proposed framework TRANSLICO works “globally” when fine-tuning the model on data of all languages scripts, and also “locally” for a case study when only fine-tuning on the Indicgroup languages that are mutually influenced and have extensive lexical overlap. Unfortunately, we didn’t validate the framework further by trying more language groups, which could further demonstrate the usage of TRANSLICO. However, this is beyond the scope of this paper. Nevertheless, we hope our framework can inspire more work that applies a similar framework and focus “locally” on groups of languages of interest for future research.

heading: Acknowledgements
text: This research was supported by DFG (grant SCHU 2246/14-1) and The European Research Council (NonSequeToR, grant #740516). We appreciate Ayyoob Imani’s suggestions for designing a case study and Lixi Liu’s suggestions for the pairwise similarity graph. We also thank the anonymous reviewers for their constructive feedback.

heading: A Hyperparameters
text: A.1 Fine-tuning on Glot500-c
We fine-tune Glot500-m (ImaniGooghari et al., 2023) on a small portion of its training data, Glot500-c. Specifically, from each language-script, we randomly select 5% sentences. Note that we also consider languages that use Latin scripts when constructing our training data (another variant is only considering languages that do not use Latin scripts, which we do for the ablation study described in §5.1). Then we construct paired data by generating the Latin transliterations for each sentence using Uroman (Hermjakob et al., 2018). The paired data is then used to fine-tune the model using the proposed contrastive learning framework. For the MLM objective, we use the standard mask rate of 15% for both sentences in their original scripts and their Uroman transliterations. The weights for all objectives are set to 1 by default. We use Adam optimizer (Kingma and Ba, 2015) with (β1, β2) = (0.9, 0.999) and ϵ = 1e-6. The initial learning rate is set to 1e-5. The effective batch size is set to 768. Each batch contains sentence pairs (one in its original script and one in Latin transliteration) randomly picked from all languages. We set the per-GPU batch to 24, the gradient accumulation to 8, and train on four RTX A6000 GPUs ( 24× 8× 4 = 768). We use FP16 training (mixed precision (Micikevicius et al., 2018)) by default. We store checkpoints for the model every 2K steps and apply early stopping with the best average performance on downstream tasks. The model is finetuned for a maximum of 3 days (roughly 1 epoch).
A.2 Fine-tuning on Downstream tasks
The basic information of each downstream task dataset is shown in Table 7. The number of languages in language families and script groups for each downstream task is shown in Table 6 and 8 respectively. We introduce the detailed hyperparameters settings in the following.
For sequence-level retrieval tasks, i.e., SR-B and SR-T, we use English-aligned sentences (up to 500 and 1000 for SR-B and SR-T respectively) from languages that are supported by the model. Different from SR-T, most of the languages in SR-B are low-resource languages. In addition, many languages in SR-B use non-Latin scripts. The retrieval task is performed without any training: we directly use the model to encode all sentences, where each sentence is represented as the average of the contextual embedding at the 8th layer. We then compute the top-10 accuracy for each pair.
For sequence-level classification tasks, i.e., Taxi1500, we fine-tune the model with a 6-classes classification head on the English train set and select the best checkpoint using the English development set. We train a model using Adam optimizer for 40 epochs with early stopping. The learning
rate is set to 1e-5 and the effective batch size is set to 16 (batch size of 8 and gradient accumulation of 2). The training is done on a single GTX 1080 Ti GPU. We then evaluate the performance in a zeroshot transfer setting by evaluating the fine-tuned model on the test sets of all other languages. The Macro F1 score is reported for each language.
For token-level classification tasks, i.e., NER and POS, we fine-tune the model with a suitable classification head (7 for NER and 18 for POS) on the English train set and select the best checkpoint using the English development set. We train each model using Adam optimizer for a maximum of 10 epochs with early stopping. The learning rate is set to 2e-5 and the effective batch size is set to 32 (batch size of 8 and gradient accumulation of 4). The training is done on a single GTX 1080 Ti GPU. We then evaluate the performance in a zero-shot transfer setting by evaluating the fine-tuned model on the test sets of all other languages. The Macro F1 score is reported for each language.
A.3 Fine-tuning on Indic Group
We fine-tune Glot500-m (ImaniGooghari et al., 2023) on a part of indic group languages. Specifically, we randomly sample 10% of sentences from Glot500-c (ImaniGooghari et al., 2023) for 12 indic languages (shown in table 5). Then we create paired data by transliterating each sentence into Latin using Uroman (Hermjakob et al., 2018). The other settings are the same in Appendix A.1.
A.4 Evaluation on Indic Group
The information of three downstream tasks WSTP, NER and CSQA is represented in Table 10. The three tasks are following the same setting of Moosa et al. (2023). For the sentence classification task WSTP, we fine-tune the model with 4-class head on 11 indic languages all at once. We train the model using Adam optimizer for 20 epochs. The learning rate is set to 2e-5 and the effective batch size is set to 256 (batch size 0f 64 and gradient accumulation of 4). The training is done on four GTX 1080 Ti GPUs. We then evaluate the performance in a
crosslingual transfer setting by evaluating the finetuned model on the test set of 11 indic languages. The accuracy is reported for each language.
For token level task NER, we fine-tune the model with a 7 classification head on 11 indic languages all at once with 20 epochs. The learning rate is set to 2e-5 and the effective batch size is set to 32 (batch size 0f 8 and gradient accumulation of 4). The training is done on a single GTX 1080 Ti GPU. We then evaluate the performance in a crosslingual transfer setting by evaluating the fine-tuned model on the test set of 11 indic languages. The Macro F1 score is reported for each language.

heading: B Futher Fine-grained Analysis
text: To further analyze how TRANSLICO can influence the crosslinguality of the multilingual model, we additionally report the aggregated results for each language family in Table 9 and the number of languages that benefit from TRANSLICO in Table 11. We see similar improvement as we observe for different script groups.

heading: C Complete Results
text: We report the complete results for all tasks and language-scripts in Table 17, 18 (SR-B), Table 19 (SR-T), Table 20, 21 (Taxi1500), Table 22 (NER), and Table 23 (POS).

heading: D Evaluation on Transliterated Data
text: We evaluate both Glot500m and FURINA under the common script scenario. Specifically, we transliterate all the data (including data of language written in Latin script, which is consistent with how we fine-tune the model with TRANSLICO) from downstream tasks to Latin script. Per-task performance for each script group is shown in Table 12 (SR-B), Table 13 (SR-T), Table 14 (Taxi1500), Table 15 (NER) and Table 16 (POS).
The results indicate that the models consistently perform better when the languages are in their original script instead of in Latin (the common script). We hypothesize the major reason is that the models are not being manipulated with vocabulary extension for transliterated data. Nevertheless, we observe from the results that TRANSLICO-uni generally outperforms Glot500m-uni across all tasks. This indicates our TRANSLICO framework is also effective for common script scenarios.

heading: E Representation Visualization
text: We visualized sentence representations from all layers of two models using PCA. Figure 5 and Figure 4 present the sentence representations of Glot500-m and FURINA respectively.

heading: Future Work
text: We hope this framework can inspire more future work leveraging transliteration to improve crosslinguality of mPLMs. Limitations
We propose a simple contrastive learning framework TRANSLICO that aims to address the script barrier. We show the effectiveness by using Glot500-m as the source model and fine-tuning it on a small portion (5%) of its pretraining data. We would assume the proposed framework can also be used directly for pretraining, through which the model might benefit further from seeing more data. However, due to a limited computation budget, we aren’t able to pretrain a model from scratch or continued pretrain a model using the full Glot500-c corpus. We would leave out how the proposed framework can be integrated into efficient pretraining or continued pretraining for future work. We see the proposed framework TRANSLICO works “globally” when fine-tuning the model on data of all languages scripts, and also “locally” for a case study when only fine-tuning on the Indicgroup languages that are mutually influenced and have extensive lexical overlap. Unfortunately, we didn’t validate the framework further by trying more language groups, which could further demonstrate the usage of TRANSLICO. However, this is beyond the scope of this paper. Nevertheless, we hope our framework can inspire more work that applies a similar framework and focus “locally” on groups of languages of interest for future research