Romanization-based Large-scale Adaptation of Multilingual Language Models
Authors: Sukannya Purkayastha
Sebastian Ruder
Jonas Pfeiffer
Iryna Gurevych
Ivan Vulić
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilinguality
Parameter Efficiency
Transliteration
TL;DR: Transliteration is all you need for adapting language models to unseen languages specifically with unseen scripts

Abstract: Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. However, their large-scale deployment to many languages, besides pretraining data scarcity, is also hindered by the increase in vocabulary size and limitations in their parameter budget. In order to boost the capacity of mPLMs to deal with low-resource and unseen languages, we explore the potential of leveraging transliteration on a massive scale. In particular, we explore the UROMAN transliteration tool, which provides mappings from UTF-8 to Latin characters for all the writing systems, enabling inexpensive romanization for virtually any language. We first focus on establishing how UROMAN compares against other language-specific and manually curated transliterators for adapting multilingual PLMs. We then study and compare a plethora of data- and parameter-efficient strategies for adapting the mPLMs to romanized and non-romanized corpora of 14 diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance for many languages, with particular gains achieved in the most challenging setups: on languages with unseen scripts and with limited training data without any vocabulary augmentation. Further analyses reveal that an improved tokenizer based on romanized data can even outperform non-transliteration-based methods in the majority of languages.

Romanization-based Large-scale Adaptation of Multilingual Language Models
Sukannya Purkayastha Sebastian Ruder?, Jonas Pfeiffer", Iryna Gurevych', Ivan Vulic
Ubiquitous Knowledge Processing Lab, Department of Computer Science and Hessian Center for AI (hessian.Al),
Technical University of Darmstadt
2 Google DeepMind
Language Technology Lab, University of Cambridge
www www.ukp... taapp.tu--aanae www.uup.ttu-drmmstacee. raw.tadd..tteee
Abstract
Language
Original Text
Romanized Text
Translation
Bhojpuri
jorjiyan bhaasan
Georgian language
Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. How- ever, their large-scale deployment many lan- guages, besides pretraining data scarcity, is also hindered by the increase in vocabulary size and limitations in their parameter budget. In der to boost the capacity of mPLMs to deal with low- wew-resoure and unseen languages, we ex- plore the potential of leveraging trunsliteration on a massive scale. In particular, we explore the UROMAN transliteration tool, which pro- vides mappings from UTF-8 to Latin characters for all writing systems, enabling inexpen- sive romanization for virtually any language. We first focus on establishing how UROMAN compares against other language-specific and manually curated transliterators for adapting multilingual PLMs. We then study and com- pare plethora of data- and parameter-efficient strategies for adapting the mPLMs to roman- ized and non-romanized corpora of diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance many languages, with partic- ular gains achieved in the most challenging setups: on languages with unseen scripts and with limited training data without any vocabu- lary augmentation. Further analyses reveal that an improved tokenizer based on romanized data can even outperform non-transliteraioon-aae methods in the majority of languages.
Sinhala
dón@sde رل tnggn.
grabalooka ayran sedithakioca
planets
Sindhi
Iran
Khmer
economy
Figure I: Romanization across different languages.
size, as the vocabulary is bound to increase with the number of languages added if per-language perfor- mance is to be maintained (Hu et al., 2020; Artetxe et al., 2020; Pfeiffer et al., 2022). Second, pretrain- ing mPLMs with fixed model capacity improves cross-lingual performance up point after which it starts to decrease; this is the phenomenon termed the curse of multilinguality (Conneau et al., 2020). Transliteration refers to the process of convert- ing language represented in one writing system to another (Wellisch et al., 1978). Latin script- centered transliteration or romanization is the most common form of transliteration (Lin et al., 201 Amrhein and Sennrich, 2020; Demirsahin et al., 2022) the Latin/Roman script is by far the most widely adopted writing script in the world (Daniels and Bright, 1996; van Esch al., 2022).! 1 Adapt- ing mPLMs via transliteration can address the two aforementioned critical issues. 1) Since the Latin script covers dominant portion of the mPLM's vocabulary (c.g., 77% in case of mBERT, see Ács), 'romanizing" the remaining part of the vocabulary might mitigate the vocabulary size issue and boost vocabulary sharing. 2) Since new tokens are added during the romanization process, reusing pretrained embeddings from the mPLM's embed- ding matrix helps reuse the information already present within the mPLM, thereby allocating the model's parameter budget more efficiently.
1 Introduction
Massively multilingual language models (mPLMs) such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have become the driving force for varicty of applications in multilingual NLP (Ponti et al., 2020; Hu et al., 2020; Moghe al., 2023). However, guaranteeing and main- taining strong performance for wide spectrum low-resource languages is difficult due to two crucial problems. The first issue is the vocabulary
However, the main drawback of transliteration seems to be the expensive process of creating ef- fective language-specific transliterators. they
'According to Encyclopedia Britannica, up to 70% of the world population is employing the Latin script.

---

typically require language expertise curate dic- tionaries that map tokens from one language and script to another. Therefore, previous attempts at mPLM adaptation to unseen languages via translit- eration (Muller et al., 2021; Chau and Smith, 2021; Dhamecha et al., 2021; Moosa et al., 2023) were constrained to a handful of languages due to the limited availability of language-specine transliter- ators, were applied only to languages that have 'language siblings' with developed transliterators. In this work, unlike previous work, we propose to use and then evaluate the usefulness of a uni- versal romanization tool, UROMAN (Hermjakob et al., 2018), for quick, large-scale and effective adaptation of mPLMs to low-resource languages. The UROMAN tool disposes of language-specific curated dictionaries and maps any UTF-8 charac- ter to the Latin script, increasing the portability of romanization, with some examples in Figure 1.
any UTF-8 character the Latin script, without the use of any external language-specific dictionaries (see Hermjakob et al. (2018) for technical details).
Cross-Lingual Transfer to Low-Resource Lan- guages. Parameter--fffiient and modular fine- tuning methods (Pfeiffer et al., 2023) such as adapters (Houlsby et al., 2019; Pfeiffer et al.. 2020b) have been used for cross-lingual trans- fer, putting a particular focus on enabling trans- fer to low-resource languages and scenarios, in- cluding languages with scripts 'unseen' by the base mPLM (Pfeiffer et al., 2021). Adapters are small lightweight components stitched into the base mPLM, and then trained for particular languages and tasks while keeping the parameters of the orig- inal mPLM frozen. This circumvents the issues of catastrophic forgetting and interference (Mc- Closkey and Cohen, 1989) within the mPLM, and allows for extending its reach also to unseen lan- guages (Pfeiffer et al., 2021; Ansell et al.. 2021). For our main empirical analyses, we adopt a state-of-the-arr art modular method for cross-lingual transfer: MAD-X (Pfeiffer et al., 2020b). In short, MAD-X is based on language adapters (LA), task adapters (TA), and invertible adapters (INV). While LAs are trained for specific languages relying on masked language modeling, TAs are trained with high-resource languages relying on task-annotated data and task-specific objectives. At inference, the source LA replaced with the target LA while the TA is kept. In order to do parameter-efffiieet learning for the token-level embeddings across dif- ferent languages and to deal with the vocabulary mismatch between source and target languages, Pfeiffer et al. (2020b) also propose INV adapters: they are placed on top of the embedding layer and their inverses precede the output embedding layer?? We adopt the better-performing MAD-X 2.0 setup (Pfeiffer al., 2021) where the adapters in the last Transformer layer are dropped at inference."
We analyze language adaptation on a massive scale via TROMAN-baseh romanization 14 diverse low-resource languages. We conduct ex- periments within the standard parameter-efficieet adapter-based cross-lingual transfer setup on two tasks: Named Entity Recognition (NER) on the WikiANN dataset (Pan et al., 2017; Rahimi et al., 2019), and Dependency Parsing (DP) with Univer- sal Dependencies v2.7 (Nivre et al., 2020). Our key results suggest that UROMAN-based transliter- ation can offer strong performance on par or even outperforming adaptation with language-specific transliterators, setting up the basis for wider use of transliteration-basee based mPLM adaptation techniques in future work. The gains with romanization- bascd adaptation over standard adaptation baselines are particularly pronounced for languages with unseen scripts (~-8-22 performance points) without any vocabulary augmentation.
set of
2 Background
Why UROMAN-Based Romanization? UROMAN- based romanization is not always fully reversible, and its usage for transliteration has thus been lim- ited in the literature. However, due its high portability, UROMAN can help scale the process of transliteration massively and as such benefit low- resource scenarios and wider adaptation of mPLMs. The main idea, as hinted in $1, is to (lear to) map
Experiments and Results
As the main means of analyzing the impact of transliteration in general and UROMAN-based ro- manization in particular, we train different variants of language adapters within the MAD-X frame- work, based on transliterated and non non-transliterated
'They are trained together with the LAs while the rest of the mPLM is kept frozen.
Cooee and data available at https: //githb.. com Com/UKPT.ab h.tp/.ab//mn resarrmmniieee zat on _as_b_de____ adaptation
"We refer the reader to the original papers for further tech- nical details regarding the MAD-X framework.

---

Language
Family
Script
# Sentences
3.1 Experimental Setup
Bhojpuri (bh)
Indo-Puropeun
Devanagari
35,983
Buryat (borr)
Mongolic
Cyrillic
41,692
Data, Languages and Tasks. Following Pfeiffer
Erzya (myv)
Uralic
Cyrllic
42.575
et al. (2021), we select mBERT as our hase mPLM.
Meadow Mari (inhe)
Uralic
Cytilicc
144,529
Min Dong (odo)
Sino-Tibetan
Chinese
33,978
We experiment with 14 typologically diverse low- resource languages that are not part of mBERT's
Mingrelian (xmff)
Kartvelian
Georgion
63,032
Sindhi (sd)
Indo European
Arabic
86,176
Sorani Kurdish (ckb)
Indo European
Arabic
459,475
pretraining corpora, with 5/14 14 languages written in distinct scripts (see Table for details). For LA training, we use Wikipedia dumps for the target languages, which we also transliterate (using dif- ferent transliterators). Evaluation is conducted on two standard cross- cross-lingual transfer tasks in zero- shot setups: 1) the WikiAnn NER dataset (Pan el al., 2017) with the train, dev, and test splits from (Rahimi et al., 2019); 2) for dependency parsing, rely the UD Dataset v2.7 (Nivre et al., 2020).
Uyghur (ug)
Turkic
Arabic
149,813
Amhäric (am)
Atro Asianc"
Ge oz
88,520
Diveli
Balo-Eurupean
Thaana
34.779
Khuner (km)
Austrsssiatic
Khmer
139,704
Sinhala (si)
Indo-European
Sinhola
219,866
Tihetan (bo)
Sino Tibetan
Tibetan
131,362
Table I: Languages with their ISO 639-3 codes used in our evaluation, along with their seript, language family, and number of sentences available for pretraining. The dashed line separates languages with unseen scripts, placed in the bottom part of the table.
LAs and TAs. English is the source language in all experiments, and is used for training TAs. The En- glish LA is obtained directly from Adapterhub.ml (Pfeiffer et al., 2020a), LAs and embeddings (when needed) are only trained for target languages.
versions of target language data, outlined here.
Variants with Ton-Transliterated Data. For the Non-TransLA+NN variant, we train LAs and INV adapters together. This variant serves to examine the extent to which mPLMs can adapt to unseen languages without any vocabulary extension. We compare this to Non-TransLA-Embb.e which trains new tokenizer for the target language (Pfeiffer al., 2021): the so-called 'lexically overlapping' tokens are initialized with mPLM's trained embed- dings, while the remaining embeddings are initial- ized randomly. All these embeddings (EmbLex) are fine-tuned along with LAs.
Finally, for the Non- Non-Transj,n Trans variant, we train a WordPiece tokenizer on the target language data with vocabulary size of 10K.
Training of Language and Task Adapters. We train all the language adapters for 50 epochs or ~ 50K update steps based on the corpus size. The batch size is set 64 and the learning rate is 1e-4.
We train English task adapters following the setup from (Preiffer et al., 2020b). For NER, we di- rectly obtain the task adapter from Adapterhub.ni which is trained with a learning rate of le 1 for 10 epochs. For DP, we train a Transformer-bass (Glavaš and Vulić, 2021) biaffine attention depen- dency parser (Dozat and Manning, 2017). We use learning rate of 5e- 1 and train for 10 epochs in (Pfeiffer et al., 2021).
Variants with Transliterated Data. We evaluate a TransLA variant, which uses the same setup as Non- TransLA but now with transliterated data. We again note that in this efficient setup, do not extend the vocabulary size, and use the fewest train- able parameters. In the TransL.A<PLLM variant, we train LAs along with fine-tuning the pretrained embeddings of mPLM (mPLMn). This further en- hances the model capacity by fine-tuning the em- bedding layer instead of using invertible adapters. 6 For both variants, transliterated data can be pro- duced via different transliterators: (i) language- specific ones; (ii) the ones from 'language siblings' (e.g., using a Georgian transliterator for Mingre- lian), or (ii) UROMAN.
All the reported results in both tasks (NER and DP) are reported as averages over 6 random seeds. All the models have been trained on A100 or V100 GPUs. None of the training methods consumed more than 36 hours. As the main means of analyz- ing the impact of transliteration in general and URO- MAN-based romanization in particular, different variants of language adapters within the MAD-X framework, based on transliterated and non con-transliterated versions of target language data, outlined here.
train
"Since LAs without INV typically perform worse than with INV (Pfeiffer et al., 2020b), also confirmed in our preliminary experiments, we do not ablate to the setup without INV.
"We do not have this setup for non-trans trterttteeeeedd data since, for languages with unseen scripts, most of the tokens are re- placed hy the generic 'UNK' token, and fine-tuning embed- dings hardly henefit downstream performance.
3.2 Results and Discussion
UROMAN versus Other Transliterators and Transliteration Strategies. In order to estab-

---

Task
Transliterator
am
ar
ka
TM
hi
sd
ayg
NER (Macro FI)
UROMAN
25.6
24.8
61.4
66.5
48.6
35.3
43.7
Other
57.3
56.7
35,9
43.8
UROMAN
3..176.6
3
47.37
32.4
33.8/1788
37.5/19.1
UD (UAS/ / LAS)
Other
29.9/5.4
32.6/ 19.9
45.0/
19.9
33.2 / 17.9
35.2 15.8
Table 2: Comparison of UROMAN with language-specift transliterators.
Seen Script
Unseen Script
Method
bh
cdo
ckh
mhr
sd
xmn
am
ho
dv
km
si
UROMAN BORROW
32.59
27.34
67.73
64.68
35.33
28,10
52.58
25.69
35,95
29.99
41.76
31.83
26.89
53.42
(ar)
45. 86
16.79
12.85
24.77
25.42
19.51
53.55
27.20
25.18
35.82
18.95
21.19
32.75
20.01
21.59
Table 3: Comparison of various transliteration strategies on the NER task (Macro-Fi) --1)
Seen Script
Unseen Script
Variant
bh
cdo
xml
mhr
ckb
bo dv
km
avg
Non- A-INV
55.14
24.19
31.31
49.74
70.31
45.54
33.53
3.26
19.86
18.72
13.81
23.14
18.39
TransLa ADDY
32.59
27.34
35.33
$2.58
64.68
67.73
28.10
25.69
35.95
29.99
41.76
31.83
26.89
Non- Trans_AvEalu.
60.04F
2831
42.47
$1399
61.05
79.12
50.42
47.60
40.96
31.21
53.94
35.89
"49.0I
TramsLa mPLMg
73.92
33.43
37.09
52.39
45.24
47.44
Table 4: Results (Macro-FI scores) WikiAnn NER averaged over 6 random seeds.
Seen Script
Unseen Script
Variant
bh
myv
ug
bxr
am
avg
Non- Moon-Trans A-NN
28.46 / 11.53
45.28/ / 26.27
33.44/ / 15.28
39.75/ 19.77
19.08/
1,85
33 33.20/ 10.81 35.43 12.41
TransLa + INY
25.12/ 10.17
45.74/
26.64
32.30
37.92
17.23
36.07
7.58
Kon- Transs Auiiinie
26.66/10.10
38.347
25.34
0.20
2231
39.5T/76.02
36,477
8.39
"3..0//12:200 39.01 14.68
TransiA
28.04 11.13
41.97 / 20.29
50.89/ / 16.56
35.03 / 20.29
39.10/ 9.00
Table 5: Results (UAS/ / LAS scores) in the DP task with UD, averaged over random seeds.
lish the utility of UROMAN as viable translit- erator, especially for low-resource languages, we compare its performance with transliteration op- tions using the TransLA+ setup as the most ef- ficient scenario. First, we compare UROMAN with language-spectio transliterators available for se- lected languages: amseg (Yimam et al., 2021) for Amharic, aiebhaarat-ranssliteaaan transliteaatiioon (Madhani et al., 2022) for Hindi and Sindhi, lang-irans for Ara- bic, and transliterate for Russian and Georgian". The transliterators used in this work are outlined in Table 6. The results are provided in Table 2. On average, UROMAN performs better or compa- rable to the language-specific transliterators. This provides justification to use UROMAN for massive transliteration scale.
refers a random setting where we associate any non-ASCII character with any ASCII character, giving us an estimate of whether we actually need knowledge of the language to build transliterators. The results are provided in Table 3: UROMAN is largely and consistently outperforming both BOR- ROW and RAND, where the single exception is BORROW (from Hindi to Bhojpuri). Surprisingly, RAND also yields reasonable performance and on average even outperforms the Non- NOn-TranSLA+IN variant with monnttransliteumm non-transliterated data (21.59 vs 18.39 in Table later). This provides further evidence towards the utility of transliteration in general and UROMAN-based romanization in particular to assist and improve language adaptation.
Performance on Low-Resource Languages is summarized in Table 4 and Table 5.9 We note that TransLA+INY outperforms TranTrAnNLA for all the languages with unseen scripts, and achieves that with huge margins (~ 8-22 points for NER and ~ 17 points in UAS scores). We observe similar trends for some the languages with seen scripts such as Min Dong (cdo), Sindhi (sd), Mingrelian
Second, we compare UROMAN to two other transliteration strategies. (i) BORROW refers to borrowing transliterators from languages within the same language family and written in the same script. 8 Since building transliterators are costly, this gives an estimate of whether it is possible to rely on the related transliterators when we do not have a language-specific one at hand. (ii) RAND
WWe also compare the performance of these methods with the standard cross-lingual transfer setup for finetuning an out- of-the-box mBERT for languages with unseen scripts in Ap- pendix B. All the adapter-based methods massively outper- form an out-of-the-oox mBERT in this scenario.
Forr reproducibility, the links to the langunge-specific transliterators are available in Appendix "F.g.. Hindi transliterator can he horrowed for Rhojpuri since the two are related and written in Devanagari.

---

4 Conclusion
30
40-
In this work, we have systematically analyzed and confirmed the potential of romanization, imple- mented via the UROMAN tool, to help with adap- tation of multilingual pretrained language models. Given (i) its broad applicability and (ii) strong per- formance overall and for languages with unseen scripts, we hope our study will inspire more work on transliteration-bass adaptation.
Variant
30-
Trans
Non Trand
201
Script
All Unseen
100
1000
10000
Ful
Number of sentences for MLM
Figure 2: Sample efficiency in the NER task.
Limitations
UAS
In this paper, work with UROMAN (Hermjakob et al., 2018) which is an unsupervised romaniza- tion tool. While it is an effective tool for roman- ization at scale, it still has potential drawbacks. Since it is only based on lexical substitution, its transliterations may not semantically or phoneti- cally align with the source content and may dif- fer from transliterations preferred by native speak- ers. Moreover, UROMAN is not invertible-as have highlighted-and may thus be less ap- pealing when text in the original script needs be exactly reproduced. Our proposed method, while it is parameter-efficient and effective- particularly for low-resource languages still underperforms language-specific tokenizer-based non- non-transliteration methods. Future work may fo- cus on developing an improved and more efficient tokenizer for transliteration-bait methods we highlight in the Appendix.
Non- Non-Transz.a Eincer
100
1000
10000
Full
Number of sentences for MLM
Figure 3: Sample efficiency in the DP task.
(xmf) on NER tasks and Erzya (myv) on DP. The less efficient T as expected, further improves the performance for all the languages except for Tibetan (bo). 10 Mon-TranSLA+Embh however, now outperforms UROMAN-based meth- ods for majority of the languages. This ob- servation can be attributed to various factors re- lated to mBERT's tokenizer, and we provide an in-depth analysis later in Appendix C. Nonetheless, we observe strong and competitive performance of TransLA + mPLM, in both tasks, again indicating that more attention should be put on transliteration- based language adaptation in future work.
While there now growing body of available evaluation resources for low-resource languages (Ebrahimi et al., 2022; Mhaske et al., 2023; Winata et al., 2023, among others), our final selection of tasks, resources and languages has been driven and constrained by the specific concrete goal of our short paper: studying and evaluating if and how translieentioo/ronanizziooo can help with adapta- tion of languages with scripts unseen by the pre- trained multilingual language model. We thus closely follow the experimental setup of Pfeiffer et al. (2021) which used the same set of tasks and languages with unseen scripts.
Sample Efficiency. Finally, we simulate a few-shot setup to study the effectiveness of using transliter- ated versus non- non-transliterated data in data-scarce scenarios. For NER, we evaluate performance on all the languages and on languages with unseen scripts; for DP, we evaluate on all the languages. Figure 2 indicates that TranSLA+INV forms better than all the other methods at sample sizes 100 (i.e., 100 sentences in the target language) and 1, 000. However, from 10, 000 sentences on- ward, Trn-TrrassA-Emmuu takes the lead. We ob- serve similar trends in the DP task (see Fig 3).
average per-
Finally, romanization can be seen as step to- wards providing more universal, or rather language- agnostic, input text representation. fullefledged comparisons against other approaches that aim to strike language independence at the input or feature level, such as byte-level models (e.g., ByT5 (Xue et al., 2022)) and pixel-based mod- els (e.g., PIXEL (Rust et al., 2023)) go beyond
This establishes the utility of transliteration for (ex- tremely) low-resource scenarios.
"For Tibetan, longer words are composed using shorter words separated by tsek (".") which is not a valid space de- limiter for the mBERT tokenizer; the number of produced subwords is thus much higher than for the other languages.

---

the scope of this particular work, but we point out this as a very interesting future research av- enue. Moreover, the integration of these language- agnostic representations with 'romanization' -bssed approaches might yield additional benefits, and should also be attested in future research.
Peter Daniels and William Bright. 1996. The world's writing systems. Oxford University Press.
Isin Demirsahin, Cibu Johny, Alexander Gutkin, and Brian Roark. 2022. Criteria for useful automatic Romanization in South Asian languages. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6662-6673, Marseille, France. European Language Resources Association.
Acknowledgements
This work has been funded by the German Re- search Foundation (DFG) as part of the Research Training Group KRITIS No. GRK 2222. The work of Ivan Vulié has been supported by a personal Royal Society University Research Fellowship (no 221 13;; 2022-))
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
We thank Indraneil Paul, Yongxin Huang, Ivan Habernal, and Massimo Nicosia for their valuable feedback and suggestions on a draft of this paper.
Tejas Dhamecha, Rudra Murthy, Samarth Bharad- waj. Karthik Sankaranarayanan, and Pushpak Bhat- tacharyya. 2021. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8584-8595, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
References
Chantal Amrhein and Rico Sennrich. 2020. On Roman- ization for model transfer between scripts in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2461-2469, Online. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Se- bastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021. MAD-G: Multilingual adapter gen- eration for efficient cross- poss-lingual transfer. In Find- ings of the Association for Computational Linguis- tics: EMNLP 202], pages 4762-4781, I, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resous resource languages. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 6279-6299, Dublin, Ireland. Association for Compu- tational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transfcrability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online, Association for Computational Linguistics.
Ethan C. Chau and Noah A. Smith. 2021. Specializing multilingual language models: An empirical study. In Proceedings of the 1st Workshop on Mulrilingual Representation Learning, pages 51 51-61, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Goran Glavaš and Ivan Vulié. 2021. Is supervised syn- tactic parsing beneficial for language understanding tasks? an empirical investigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3090-3104, Online, Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised pross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Ulf Hermjakob, Jonathan May, and Kevin Knight. 2018. Out-of-the-box universal Romanization tool uroman. In Proceedings of ACL 2018, System Demonstrations, pages 13-18, Melbourne, Australia, Association for Computational Linguistics.

---

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Benjamin Muller, Antonios Anastasopoulos, Benoft Sagot, and Djamé Seddah. 2021. When being un- seen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 442. Online. Association for Computa- tional Linguisties.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sehastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Xiaoman Pan, Boliang Zhang. Jonathan May. Joel Noth- man, Kevin Knight, and Heng Ji. 2017. name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics ( (Volume I: Long Papers), pages 1946-1958 Vancouver, Canada. As- sociation for Computational Linguistics.
Ying Lin, Cash Costello, Boliang Zhang, Di Lu, Heng Ji, James Mayfield, and Paul McNamee. 2018. Plat- forms for non non-speakers annotating names in any lan- guage. In Proceedings of ACL 2018, System Demon- strations. pages 1-6, Melbourne, Australia. Associa- tion for Computational Linguisties.
Jonas Pfeiffer. Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational I Linguistics: Human Language Technologies. pages 3479-344400 -3995, Seattle, United States. Association for Computational Lin- guistics.
Yash Madhani, Sushane Parthan, Priyanka A. Bedekar, Ruchi Khapra, Vivek Seshadri, Anoop Kunchukut- tan, Pratyush Kumar, and Mitesh M. Khapra. 2022. Aksharantar: Towards building open transliteration tools for the next billion users. arXiv preprint.
Michael McCloskey and Neal J. Cohen. 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychol- ogy of Learning and Motivation, pages (09-165. Aca- demic Press.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sehastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A framework for adapting transformers. Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Arnav Mhaske, Harshit Kedia, Sumanth Doddapaneni, Mitesh M. Khapra, Pratyush Kumar, Rudra Murthy, and Anoop Kunchukuttan. 2023. Naamapadam: A large-scale named entity annotated data for Indic lan- guages. In Proceedings of the 61st Annual Meeting of the Association for Computational I. Linguistics (Vol- ume 1: Long Papers), pages 1044)-10456, Toronto, Canada, Association for Computational Linguistics.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023. Modular deep learning. arXiv preprint.
Nikita Moghe, Evgeniia Razumovskaia, Liane Guillou, Ivan Vulié, Anna Korhonen, and Alexandra Birch. 2023. Multi3NLU++: A multilingual, multi-intent, multi-domain dataset for natural language under- standing in task-oriented dialogue. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3732-3755, Toronto, Canada. Associa- tion for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020b. MAD-X: An Adapter-Based Framework for Multi-Task Lrogs-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNNPP), pages 7654-7673, Online, Association for Computa- tional Linguistics.
Jonas Pfeiffer. Ivan Vulié, Iryna Gurevych, and Sebas- tian Ruder. 2021. UNKs everywhere: Adapting mul- tilingual language models to new scripts. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 101 10186-10203, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Ibraheem Mubammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. 2023. Does transliteration help multilingual language modeling? In Findings of the Association for Computational Linguistics: EACL 2023, pages 670-685, Dubrovnik, Croatia. As- sociation for Computational Linguistics.

---

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLPP), pages 2362-2376, Online. As- sociation for Computational Linguisties.
Experimentation and evaluation with multiple tasks and datasets. Future Internet, 13(11).
for amharic:
Judit Ács. Exploring bert's vocabulary. Blog Post.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Mas- sively multilingual transfer for NER. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy, Association for Computational Linguisties.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Eliz- abeth Salesky, Miryam Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? the monolingual performance of multilin- gual language models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118-118100 18-3135, Online. Association for Computational Linguistics.
Daan van Esch, Tamar Lucassen, Sehastian Ruder, Isaac Caswell, and Clara Rivera. 2022. Writing system and speaker metadata for 2,800+ language varieties. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5035-5046, Mar- seille, France. European Language Resources Asso- ciation.
Hans H Wellisch, Lee Breuer, Richard Foreman, and Robert Wilson. 1978. The conversion of scripts, its nature, history, and utilization. New York; Toronto: Wiley.
Genta Indra Winata, Alham Fikri Aji. Samuel Cahyawi- jaya, Rahmad Mahendra, Fajri Koto, Ade Romad- hony, Kemal Kurniawan, David Moeljadi, Radi- tyo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau. Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual parallel sentiment dataset for Indonesian local languages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 815-834, Dubrovnik, Croatia. Association for Com- putational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami AI- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByTS: Towards token-free future with pre-trained byte-to-byte models. Transac- fions of the Association for Computational Linguis- tics, 10:291-306.
Seid Muhie Yimam, Abinew Ali Ayele, Gopalakrish- nan Venkatesh. Ibrahim Gashaw, and Chris Bie- mann. 2021. Introducing various semantic models

---

A Transliterators in Evaluation
on the transliterated data and initialize lexically overlapping embeddings with mBERT's pretrained embeddings.
Besides UROMAN, also employ various language-specific transliterators which are publicly available. We list them in Table 6.
We plot the performance in Figure 5. The new method, Trans now outperforms the non- transliteration Draebberrationbbeeed variant on 8/12 languages and also average. Consequently, this validates our hypothesis and is in line with the previous work (Moosa et al., 2023). However, in performance in the case of mhr (-10.71) and cdo (-10.14) when compared Trans mPL Ma These drops may be attributed to the lower degree of lexical overlap with mBERT's vocabulary, and consequently a higher number of randomly initial- ized embeddings for those target languages.
B Performance comparison of mBERT
We adapt the standard cross-lingual transfer setup for mBERT. The model is finetuned on the task data for source language (high-resource) and is used perform inference on the target language (low-resource). We report the performance com- parison of the standard looss-lnngual transfer setup for mBERT on the NER task for languages with unseen scripts with the adapter- based methods in Table 7. We observe that the adapter-bast based meth- ods outperform mBERT by huge margins.
found a drop
C Further Analyses
Following previous work (Ács; Rust et al., 2021; Moosa et al., 2023), we further analyze tokeniza- tion quality of the mBERT tokenizer using the fol- lowing established metrics: 1) % of "UNK'' mea- sures the % of "UNK" tokens produced by the tokenizer, and aim is to compare their rate before and after transliteration; 2) fertility mea- sures the number of subwords that are produced per tokenized word; 3) proportion of continued subwords measures the proportion of words for which the tokenized word is split across at least two subwords (denoted by the symbol ##).
From the results summarized in Figure 4, it is apparent that transliteration drastically reduces % of UNKs. However, mBERT's tokenizer under- performs as compared to monolingual tokenizers based on fertility and the proportion of continued subwords (Rust et al., 2021). Transliteration per- forms better for some languages where the quality of the mBERT tokenizer is similar to the monolin- gual tokenizer such as for dv, km, and cdo. On the other hand, transliteration methods perform worse on languages where the quality of the underlying mBERT tokenizer is relatively poor.
In order to test the hypothesis that the tokenizer quality might be the principal reason for the perfor- mance gap for the transliteration-base methods in comparison to the non-transliteration based meth- ods, carried an additional experiment. For the experiment, we adapt the Non-T Ton-TransLAAmml to operate on transliterated data, and call this vari- ant Trans A+Embus Here, we train a new tokenizer

---

Transliterator
Used for languages
Available
UROMAN
All
github. cowp:/si-n-llp//emnn
eppp..org/prrjject/ameeee//
umseg
transliterate
ka
hidoharra-transstternnnt
hi,
sppp..rt//pooocc////alalaaaaaaaa interatioon
lang-trans
Table Transliterators used this work.
Now Tand
Fontle
Pwg Cons
Pron
(a)
(b)
(C)
Figure 4: Tokenizer quality analysis. % of UNKs before and after transliteration, b) Fertility, and c) Proportion of continued subwords for mBERT vs monolingual tokenizer.
Unseen Scripts
Variant
om
bo
dv
km
si
mBERT
0.91
17.40
1.30
10.71
2,50
Non-Transj, A-INV
3.26
Transla INV
25.69
41.76
Non- Non-TransLAvBuuke
47.60
40.96
31.21
53.94
45.89
Transla 1PPMM
37.09
33.82
40.40
52.39
45.24
Table 7: Performance Comparison of mBERT with
adapter-based finetuning methods for unseen scripts on the NER task.
Trans
Non: TrunscaeCoter
Figure S: Comparison of Non-TransLAwEmmaa Embba with Transt on NER (left) and DP (right).


Reviews:
[{"review_id": "qpDV0AY2r3", "reviewer": "Reviewer_SUm1", "summary": "", "questions": "- What are the similar languages used for each language for BORROW strategy?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their useful feedback and suggestions. We address the concerns below:\n# Reasons \n   #### $R1$. **“The evaluation is done on only two tasks. Both these are tasks that are to language-dependent to a fairly large extent. The good performance of NER due to romanization could possibly be also attributed to named entities in English having similar romanization. It would be intersting to understand the performance of universal romanization on semantic tasks like text classification, NLI or sentiment analysis.”**\n\n**$Ans.$** We selected the WikiAnn and Dependency Parsing benchmarks due to their relevance in assessing model performance on languages and scripts that were not present in the pretraining data. These datasets have been used in prior research for similar reasons, as demonstrated by works such as [1] and [2].\nWe acknowledge the importance of evaluating UROMAN's effectiveness on a variety of tasks, however, it's worth noting that identifying datasets that meet the criteria of featuring unseen languages or scripts while spanning multiple semantic tasks can be a challenging endeavor. To the best of our knowledge, there are no alternative datasets and additional tasks that align with our evaluation objectives.\n\n$R2$. **“The novelty is the work is limited. Some of findings confirm what is know in literature: the use of transliteration and vocabulary adaptation in low-resource scenarios is beneficial. The benefit of universal romanization with UROMAN for NLU is a cross-lingual setup is new, but UROMAN has been explored for multilingual NMT previously.”**\n\n**$Ans$.** While it's true that UROMAN has been explored previously in multilingual neural machine translation (NMT) tasks, our paper takes a fresh perspective on its application in the context of recent advancements in language adaptation methods. The field of NLP has seen significant advancements since the previous UROMAN explorations, including methods that involve extending the vocabulary or replacing the embedding layer. Our study investigates the compatibility and effectiveness of UROMAN within this modern landscape, demonstrating its relevance and applicability.\n\nFurthermore, one of our main contributions lies in comparing UROMAN, a universal transliterator that can be applied to any script, with labor-intensive and expensive human-curated transliteration tools. While previous research may have examined UROMAN in multilingual NMT, our focus is on the benefits and trade-offs of UROMAN in comparison to specialized transliterators in the context of large-scale language model adaptation. This comparison offers insights into the practicality and efficiency of UROMAN across diverse languages, including low-resource settings. Our paper's primary objective is to validate the performance of UROMAN within the framework of recent language adaptation methods, rather than merely replicating previous findings. Through rigorous experimentation, we establish that UROMAN-based transliteration can indeed achieve competitive performance, even when compared to carefully curated and expensive transliteration tools. These results emphasize the robustness and effectiveness of UROMAN in contemporary cross-lingual setups.\n\n# Questions\n#### $Q1$. **“What are the similar languages used for each language for BORROW strategy?”**\n**$Ans$.** Table 2 second row, outlines the languages used for the BORROW strategy in the parentheses. For Bhojpuri(bh), we used Hindi(hi) transliterator, Sorani Kurdish (ckb) - Arabic (ar), Meadow Mari (mhr) - Russian (ru), Sindhi (sd) - Arabic (ar), Uyghur (ug) - Arabic (ar) and Mingrelian (xmf) - Russian (ru). We will add a detailed table for these borrowed language mappings to the paper following your question. \n\n### **References**  \n[1] “UNKs Everywhere: Adapting Multilingual Language Models to New Scripts\" (Pfeiffer et al., EMNLP 2021)  \n[2] \"When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models\" (Muller et al., NAACL 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows that a universal romanization scheme is good for adapting PLMs to unseen languages, and is competitive with vocabulary adaptation and use of language specific romanization.\n- Further, vocabulary adaptation using romanized data seems to further improve the model performance - which is an interesting result. This result is reported in the appendix, but it find it one of the interesting contributions of the paper and it is important enough to report in the main paper. Additionally, you could report results \n- The paper is well-written, experiments are well-described and well-motivated.", "reasons_to_reject": "- The evaluation is done on only two tasks. Both these are tasks that are to language-dependent to a fairly large extent. The good performance of NER due to romanization could possibly be also attributed to named entities in English having similar romanization. It would be intersting to understand the performance of universal romanization on semantic tasks like text classification, NLI or sentiment analysis. \n\n- The novelty is the work is limited.  Some of findings confirm what is know in literature: the use of transliteration and vocabulary adaptation in low-resource scenarios is beneficial. The benefit of universal romanization with UROMAN for NLU is a cross-lingual setup is new, but UROMAN has been explored for multilingual NMT previously.", "questions_to_authors": "- What are the similar languages used for each language for BORROW strategy?", "paper_topic_and_main_contributions": "In this paper, the authors explore the use of universal romanization for extending pre-trained languages to unseen languages (for seen and unseen scripts). They use the UROMAN romanization scheme based on UNICODE standard documentation for the same. They evaluate 2 tasks (NER and DP) in a zero-shot cross-lingual setup. Through the experiments and studies, the paper concludes that UROMAN romanization is good or better than language-specific romanization schemes and other baselines.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6VIBbO9TkT", "reviewer": "Reviewer_oGZa", "summary": "", "questions": "A: When comparing monolingual transliteration tools borrowed from languages written with the same script, UROMAN, and a random setup, the random setup achieves a surprisingly high performance. Do the authors have any idea why this might be the case?\n\nB: Do the authors think that high fertility values or a high proportion of continued subwords necessarily indicate that the tokenizer performs worse for the target language? Would this be the case even for morphologically complex languages?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their useful and detailed feedback. Below are our justifications for their concerns.\n\n# **Reasons**\n\n#### $R1$. **\"The authors make no reference to byte-based (ByT5; Xue et al., 2021) and/or pixel-based (PIXEL; Rust et al., 2022) tokenization methods, or other universal transliteration tools. The paper would benefit from a reference to at least byte-based and pixel-based methods and some justification why romanisation might be more appropriate to use. This justification could also include some performance comparison.”**\n\n$Ans$. Byte-based tokenization, such as ByT5 [1], and pixel-based tokenization, such as PIXEL [2], can mitigate the issue of unknown-token fallbacks for unseen characters. However, the representations of these unseen tokens remain uninformative, which may limit the model's ability to generalize effectively to new scripts and languages. In contrast, transliteration provides a means of converting unseen characters into familiar Latin characters and subword, enabling the model to leverage existing knowledge about Latin script characters and subwords. This inherently offers more informative zero-shot information, as it allows the model to draw on the relationships between scripts and languages while handling unseen tokens. We will make these connections and (dis)similarities clearer in the paper by adding a discussion on the advantages of romanization.\n\n$R2$. **“The authors do not compare their results with a baseline of using an out-of-the-box mBERT model without any language adapters. The inclusion of this (which would likely underperform all the other methods presented) would make their results that much more robust.”**  \n\n$Ans$.  A significant portion of the languages we evaluated feature scripts that are entirely unseen during mBERT's pretraining. Consequently, employing an out-of-the-box mBERT model without any language adapters or additional adaptation is not a viable option for accurately representing these scripts. The inherent inability of mBERT to handle these scripts has been demonstrated in prior research, as highlighted in [3] and [4]. Our preliminary experiments align with the aforementioned research findings, showing that full finetuning of mBERT without specialized adaptation performs poorly on languages with unseen scripts. For instance, our initial experiments validated this trend, with Amharic achieving $0.9$ F1 on WikiAnn, Tibetan achieving $17.4$, and Khmer achieving $10.7$. We will incorporate the performance metrics achieved by an out-of-the-box mBERT model without any language adapters as part of our main results. \n\n$R3.$ **“There are no details on the training data sizes the language adapters were trained on.”**  \n\n$Ans$. We apologize for the oversight in not providing the training data sizes for the language adapters in the initial version of our paper. The number of sentences used for each language adapter is as follows:\n\n* Amharic ($88320$)\n* Bhojpuri ($35983$)\n* Tibetan ($131362$)\n* MinDong ($33978$)\n* Divehi ($34779$)\n* Khmer ($139704$)\n* Sindhi ($86176$)\n* Sinhala ($219866$)\n* Mingrelian ($63032$)\n* Meadow Mari ($144529$)\n* Sorani Kurdish ($459475$)\n* Buryat ($41692$)\n* Uyghur ($149813$)  \n\nWe will include a section that explicitly presents the training data sizes for each language adapter, ensuring that this information is readily available to readers.\n\n# **Questions**\n#### $Q1$. **“When comparing monolingual transliteration tools borrowed from languages written with the same script, UROMAN, and a random setup, the random setup achieves a surprisingly high performance. Do the authors have any idea why this might be the case?”**\n\n$Ans$. One crucial factor contributing to the observed behavior is the composition of mBERT's vocabulary. A substantial portion (77%) of mBERT's vocabulary consists of Latin characters, which are common across various languages. Consequently, when characters in a non-Latin script are transformed to Latin characters in the random setup, the number of subwords generated tends to decrease. This reduction in subword count can lead to better performance due to a closer match between the tokenization of the training data and the pretrained vocabulary. However, the situation differs when dealing with non-Latin scripts that are entirely unseen during pretraining. In these cases, the transliteration process introduces a large number of subwords due to the unique characteristics of these scripts. The abundance of subwords generated in the process of transliteration can lead to suboptimal performance as the subword-to-vocabulary mapping becomes less informative.\n\n#### $Q2$. **\"Do the authors think that high fertility values or a high proportion of continued subwords necessarily indicate that the tokenizer performs worse for the target language? Would this be the case even for morphologically complex languages?\"**\n\n$Ans$. Subword fertility has been shown to have a strong correlation with the performance of multilingual models. This phenomenon has been explored and established in prior research, most notably in [5]. While subword fertility's impact is potentially more pronounced in languages with simpler morphologies, it is still a relevant consideration for morphologically complex languages. These languages can exhibit a wide range of word forms, affixes, and inflections, which may result in higher subword fertility. In such cases, the subword segmentation process becomes crucial for capturing the intricate morphological structure accurately.\n\n# **Typos**\n#### $T1$. **\"It might be better to have the languages used listed in the main body of the paper and not just in the appendix.\"**\n\n$Ans$. Due to page limitations of a short paper, we were unable to include this information in the main body - plenty of prior work also listed the languages as part of the appendix. However, provided we are granted an additional page, we will add it to the main paper.\n\n### **References**\n[1] “ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models” (Xue et. al, TACL 2022)  \n[2] “Language Modelling With Pixels” (Rust et. al, ICLR 2023)  \n[3] “UNKs Everywhere: Adapting Multilingual Language Models to New Scripts\" (Pfeiffer et al., EMNLP 2021)  \n[4] \"When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models\" (Muller et al., NAACL 2021)  \n[5] “How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models” (Rust et al., ACL 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is addressing an important and pervasive issue in multilingual NLP.\n* There is some robust analysis demonstrating the sample efficiency of the method as well as explaining the cases where it might underperform.\n* The paper compares a large number of setups and the inclusion of these seems well-motivated.\n* The authors achieve definite improvements over other methods. Their results are made more robust by the fact that they represent the average over 6 random seeds.", "reasons_to_reject": "* The authors make no reference to byte-based (ByT5; Xue et al., 2021) and/or pixel-based (PIXEL; Rust et al., 2022) tokenization methods, or other universal transliteration tools. The paper would benefit from a reference to at least byte-based and pixel-based methods and some justification why romanisation might be more appropriate to use. This justification could also include some performance comparison.\n* The authors do not compare their results with a baseline of using an out-of-the-box mBERT model without any language adapters. The inclusion of this (which would likely underperform all the other methods presented) would make their results that much more robust.\n* Since the UROMAN transliteration is often irreversible, there is a problem of how useful this method might be for real-world use. However, the authors are aware of this and mention it in their limitations section.\n* There are no details on the training data sizes the language adapters were trained on.", "questions_to_authors": "A: When comparing monolingual transliteration tools borrowed from languages written with the same script, UROMAN, and a random setup, the random setup achieves a surprisingly high performance. Do the authors have any idea why this might be the case?\n\nB: Do the authors think that high fertility values or a high proportion of continued subwords necessarily indicate that the tokenizer performs worse for the target language? Would this be the case even for morphologically complex languages?", "paper_topic_and_main_contributions": "In this paper, the authors aim to utilise transliteration to improve cross-lingual transfer of languages not written with the Latin script, especially targetting low-resource and unseen languages. This romanisation avoids the extension of the original model vocabulary and can help to better leverage existing model information thanks to overlapping vocabulary items. The authors employ **UROMAN**, a universal romanisation tool and demonstrate in many cases marked performance benefits on a diverse set of 14 low-resource languages on NER and dependency parsing. Additionally, they demonstrate the sample efficiency of their methodology, which is promising for achieving performance gains over low-resource languages. \n\nDespite gaps in the literature review and the lack of comparison with other methods, such as other universal romanisation frameworks or byte-based tokenization, the paper is informative all-in-all.\n\nSome minor notes:\n* The only BORROW setup that outperforms both UROMAN and RAND when it comes to transliteration is when transliterating Bhojpuri using a Hindi tool. It might be nice to mention that this is probably because the two languages are closely related unlike for instance Meadow Mari and Russian.\n* When describing why performance on Tibetan drops when using their Trans_LA+INV setup as compared to Non-Trans_LA+INV, the authors give a nice explanation related to the word separator character in Tibetan. They also make the claim that the number of produced subwords becomes higher for Tibetan than for other languages because of this. It might be nice to show some graph that demonstrates this.\n* When mentioning the correlation in Appendix E between the performance reached via transliteration and the quality of the mBERT tokenizer, it would be nice if a figure could demonstrate this for the reader as well.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* It might be better to have the languages used listed in the main body of the paper and not just in the appendix."}, "extra_text": {}}, {"review_id": "gsMhQjssTV", "reviewer": "Reviewer_PTYq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their useful feedback and suggestions.\n\n# **Reasons**\n#### $R1$. **“All transliteration-based systems which perform well modify the embedding table. Despite using adapters, the original mBERT system is therefore not preserved. Hence there is no reason not to finetune all mBERT parameters and results for such a finetuning baseline should be presented as well.\"**\n\n$Ans$. Our decision to finetune the embedding layer while keeping the transformer weights frozen is motivated by the goal of preserving the transferability of transformer blocks across languages. Full finetuning of the transformer blocks on source or target tasks can lead to a drop in transfer performance, as indicated in previous research, such as the \"MAD-X\" study [1]. This drop in transferability occurs because the specificity introduced through full finetuning hampers the model's ability to generalize effectively across diverse languages and linguistic characteristics.\n\nBy focusing on finetuning the embedding layer, we aim to achieve a balance between language specificity and cross-lingual transfer. This approach aligns with the principles demonstrated in the \"XQuAD\" paper [2], where finetuning the embedding layer allowed monolingual models to become multilingual, providing enhanced language coverage with manageable adjustments.\n\nFrom a parameter-efficiency perspective, finetuning only the embedding layer offers significant advantages. Making the embedding layer language-specific maintains the benefits of a shared transformer architecture while adapting the model to language-specific linguistic features.\n\nThe concept of posthoc analysis to identify changed embeddings and then map them to respective target languages further enhances the efficiency of the proposed setup. This approach minimizes the computational cost associated with updating embeddings, making it practical for production-level deployment.\n\n#### $R2$. **“The performance difference between Non-Trans LA+Emb_Lex and Trans LA + mPLM_ft is small on average. At the same time, the results for individual languages vary greatly. This suggests that the differences might be insignificant. The authors should therefore do a significance test. It was not clear to me whether the results have been obtained from single runs or whether they are averages over several runs. The latter could reduce the above-mentioned variance in the results.”**\n\n$Ans$. All the reported results in our paper have been obtained by averaging over six random seeds, as indicated in the captions of Table 3 and Table 4. We will highlight this information more prominently in the paper following your suggestion.\n\n# **Typos**\n#### $T1$. **“Figure 2: The image legend hides a part of the graph.”**\n\n$Ans$. We will improve the figure, apologies for the oversight. \n\n#### $T2$. **\"unclear: \"RAND refers to a random setting where we associate any non-ASCII character with any ASCII character\" Did you randomly select a single ASCII character for each non-ASCII character?\"**\n\n$Ans$. We considered all the ASCII characters and randomly assigned any ASCII character to any Non-ASCII character. It is a many-to-many mapping. \n\n#### $T3$. **unclear definition: \"proportion of continued subwords measures the proportion of words for which the tokenized word is split across at least two subwords\"**\n\n$Ans$. The metric termed “Proportion of continued words” measures proportion of words in a particular language which gets split into more than two subwords. In case of mBERT, these subwords are preceded by ## symbol. For more information we refer the reader to [3]. We will clarify this in the paper.\n\n### **References**\n\n[1] “MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer” (Pfeiffer et. al, EMNLP 2020)  \n[2] “On the Cross-lingual Transferability of Monolingual Representations” (Artetxe et. al, ACL 2020)  \n[3] “How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models” (Rust et al., ACL 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper shows that UROMAN is a good alternative to language-specific transliteration systems for NLP purposes.\nIt also shows that transliteration-based handling of foreign scripts gives good results.", "reasons_to_reject": "All transliteration-based systems which perform well modify the embedding table. Despite using adapters, the original mBERT system is therefore not preserved. Hence there is no reason not to finetune _all_ mBERT parameters and results for such a finetuning baseline should be presented as well.\n\nThe performance difference between Non-Trans LA+Emb_Lex and Trans LA + mPLM_ft is small on average. At the same time, the results for individual languages vary greatly. This suggests that the differences might be insignificant. The authors should therefore do a significance test.\nIt was not clear to me whether the results have been obtained from single runs or whether they are averages over several runs. The latter could reduce the above-mentioned variance in the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper compares the zero-shot performance of MAD-X with mBERT on NER and UD parsing with and without transliterating non-ASCII characters to ASCII characters. MAD-X uses language and task adapter layers to adapt mBERT to different languages and tasks.\n\nThe experimental results indicate that \n\n- the simple UROMAN transliteration method performs as good as language-specific transliterators on NER and better on UD parsing\n\n- the UROMAN-based system outperforms regular systems whose vocabulary is not extended.\n\n- the UROMAN-based system performs similar to regular systems whose vocabulary _is_ extended.\n\n- the UROMAN-based system with extended vocabulary outperforms regular systems with vocabulary extension.\n\n-  the UROMAN-based system is particularly useful when low-resource training data is very scarce.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2: The image legend hides a part of the graph.\n\nunclear: \"RAND refers to a random setting where we associate any non-ASCII character with any ASCII character\"\nDid you randomly select a single ASCII character for each non-ASCII character?\n\nunclear definition: \"proportion of continued subwords measures the proportion of words for which the tokenized word is split across at least two subwords\""}}]

Author rebuttal:

Meta-review: This work proposes UROMAN to adapt multilingual pretrained language models (PLMs) through transliterating UTF-8 characters to Latin scripts. All authors agree that the task of adapting PLMs to unseen scripts is important and well-motivated. The strength of UROMAN is its generalizability to any new language with unseen scripts. The authors show that UROMAN adapts PLMs to new languages with unseen scripts as good as or even better than language-specific transliteration, and the choice of languages in the experimental setup is diverse enough (with runs over 6 different seeds).

The paper lacks some baseline results, such as the finetuning of the entire model for language adaptation. Note that in this current climate, finetuning the entire small models like mBERT is an extremely reasonable practice; adapters are particularly useful for models with billions of parameters, so the authors' argument against not finetuning the whole model as baseline comparison due to computational cost doesn't really hold in my opinion. There is also work that shows that finetuning the whole model is better than language adapters at mBERT model sizes [1, 2].  While it's mentioned in the rebuttal, the authors should also include the results for out-of-the-box mBERT results as well in the paper and the training details such as number of tokens for adaptation per language. 

[1]: Ebrahimi, A., & Kann, K. (2021). How to Adapt Your Pretrained Multilingual Model to 1600 Languages. In ACL 2021.
[2]: Yong, Zheng-Xin, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M. Saiful Bari et al. (2023) "Bloom+1: Adding language support to bloom for zero-shot prompting." In ACL 2023.
Decision: Accept-Findings