Transliteration: A Simple Technique For Improving Multilingual Language Modeling 

1 Introduction . In the last few years , we have seen impressive advances in many NLP tasks . These advances have been primarily led by the availability of large representative corpora and improvement in the architecture of large language models . While improving model architectures , training methods , regularization techniques , etc. , can help advance the state of NLP in general , the unavailability of large , diverse corpora is the bottleneck for most languages ( Joshi et al. , 2020 ) . However , creating representative and inclusive corpora is an ongoing process and is not always possible for many low-resource languages . Thus to inclusively advance the state of NLP across languages , it is crucial to develop techniques for training MLLMs that can extract the most out of existing multilingual corpora . Here we focus on the issue of diverse writing scripts used by closely related languages that may prevent MLLMs from learning good cross-lingual representations . For example , the Indo-Aryan family of languages uses at least six different scripts among them . Previous papers ( Pfeiffer et al. , 2021 ) have noted that low-resource languages that use unique scripts tend to have very few tokens representing them at the tokenizer . As a result , these languages tend to have more UNKnown tokens , and the words in these languages tend to be more split up by sub-word tokenizers . While this issue is evident for Indian languages , there are other examples of closely related languages separated by different scripts such as Serbian and Croatian . Often we can easily transliterate from one script to another using rule-based systems . For example , there are established standards that can be used to transliterate Greek ( ISO 843 ) , Cyrillic ( ISO 9 ) , Indic scripts ( ISO 15919 ) , and Thai ( ISO 11940 ) to the Latin script . In this paper , We focus on the Indo-Aryan language family and empirically measure the effect of transliteration on the downstream performance of MLLMs . The linguistic relatedness in the Indo-Aryan language family is discussed in appendix A.1 . First , we pretrain two ALBERT base ( Lan et al. , 2020 ) models from scratch on the IndoAryan languages of the OSCAR corpus ( Ortiz Su ’ arez et al. , 2019 ) , where one is pretrained with original scripts and the other after transliterating to a common script . Then , we evaluate the models on four diverse downstream tasks from the IndicGLUE benchmark dataset , which are Wikipedia Section Title Prediction ( WSTP ) , News Category Classification ( NCC ) , Named Entity Recognition ( NER ) , and Cloze-Style Question Answering ( CSQA ) . Additionally , we evaluate the models on five different public datasets spanning three different tasks , Article Genre Classification ( AGC ) , Sentiment Analysis ( SA ) , and Discourse Mode Classification ( DMC ) . We use nine random seeds on all downstream finetuning tasks . To rigorously compare the two models , we perform the Mann-Whitney U test ( MWU ) between the uniscript model ( group 1 ) and the multi-script model ( group 2 ) . MWU test lets us verify whether the performance differences of the two models are significant or not . Apart from statistical significance , we also report three different effect sizes to conclusively determine whether the magnitude of improvement due to transliteration is useful in practice . Using the MWU test , we conclude that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages . We also measure the tokenizer quality and the cross-lingual representation similarity ( CLRS ) to understand why the uni-script model performs better than the multi-script model . To measure tokenization quality , we use subword fertility and unbroken ratio ( Ács , 2019 ; Rust et al. , 2021 ) . To measure the CLRS , we use the centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ) similarity score . We measure the CKA similarity score between the hidden representations of the models on the parallel sentences of eight Indo-Aryan languages from the FLORES-101 dataset ( Goyal et al. , 2021 ) . We find that , compared to the multi-script model , the uni-script model achieves a higher CKA score and it is more stable throughout the hidden layers of the uni-script model . Based on this , we conclude that the uni-script model learns better cross-lingual representation than the multi-script model . In summary , our contributions are primarily three-fold : 1 . We find that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages . 2 . We establish this finding through rigorous experiments and show the statistical significance along with the effect size of transliteration using the Mann-Whitney U test . 3 . Using CKA on the FLORES-101 dataset , we show that transliteration helps MLLMs learn better cross-lingual representation . 2 Motivation and Background . In their study , Joshi et al . ( 2020 ) showed the resource disparity between low-resource and high-resource languages , and ( Ruder , 2020 ) discussed the necessity of working with lowresource languages . A large body of work suggests that language-relatedness can help MLLMs achieve better performance on low-resource languages by leveraging related highresource languages . For instance , ( Pires et al. , 2019 ) found that lexical overlap improved mBERT ’ s multilingual representation capability even though it learned to capture multilingual representations with zero lexical overlaps . ( Dabre et al. , 2017 ) showed that transfer learning in the same or linguistically similar language family gives the best performance for NMT . ( Lauscher et al. , 2020 ) found that language relatedness is crucial for POS-tagging and dependency parsing tasks . Although , corpus size is much more important for tasks like NLI and Question Answering . ( Wu & Dredze , 2020 ) demonstrated bilingual BERT can outperform monolingual BERT on low-resource languages when the bilingual languages are linguistically closely related . Nevertheless , mBERT outperforms bilingual BERT when it comes to low-resource languages . However , one of the major challenges in leveraging transfer between high-resource and low-resource languages is overcoming the script barrier . Script barrier exists when multiple closely related languages use different scripts . ( Anastasopoulos & Neubig , 2019 ) found that for morphological inflection , script barrier between closely related languages impedes cross-lingual learning . However , transliteration and phonemebased techniques have been proposed to solve this issue . For example , ( Murikinati et al. , 2020 ) expanded upon ( Anastasopoulos & Neubig , 2019 ) and showed that both transliteration and grapheme to phoneme ( g2p ) conversion removes script barrier and improves cross-lingual morphological inflection and ( Rijhwani et al. , 2019 ) showed that pivoting lowresource languages to their closely related high-resource languages results in better zero shot entity linking capacity and used phoneme-based pivoting to overcome the script barrier . ( Bharadwaj et al. , 2016 ) showed that phoneme representation outperformed orthographic representations for NER . ( Chaudhary et al. , 2018 ) also used phoneme representation to resolve script barrier and adapt word embeddings to low-resource languages . ( Goyal et al. , 2020 ) and ( Song et al. , 2020 ) both utilized transliteration and showed that language relatedness was required for improving performance on NMT . ( Amrhein & Sennrich , 2020 ) studied how transliteration improved NMT and came to the conclusion that transliteration offered significant improvement for low-resource languages with different scripts . ( Khemchandani et al. , 2021 ) showed on Indo-Aryan languages that language relatedness could be exploited through transliteration along with bilingual lexicon-based pseudo-translation and aligned loss to incorporate low-resource languages into pretrained mBERT . ( Muller et al. , 2021a ) showed that for unseen languages , script barrier hindered transfer between low-resource and high-resource languages for MLLMs and transliteration removed this barrier . They showed that transliterating Uyghur , Buryat , Erzya , Sorani , Meadow Mari , and Mingrelian to Latin script and finetuning mBERT on the respective corpus with masked language modeling objective improved their downstream POS performance significantly . ( Dhamecha et al. , 2021 ) showed that transliterating Indo-Aryan languages improved multilingual language model performance . In contrast , ( K et al. , 2020 ) and ( Artetxe et al. , 2020 ) seem to show that multilingual BERT can learn cross-lingual representations without any lexical overlap , a shared vocabulary , or joint training . However , these works focus on zero shot cross-lingual transfer learning where the models are finetuned on one language and evaluated on another . From the literature , it can be seen that apart from g2p , many in the community believe transliteration to be a potential solution for script barriers . However , most of the work shows the benefits of transliteration for NMT . Nevertheless , there is no solid empirical analysis of the effects of transliteration for MLLMs apart from ( Dhamecha et al. , 2021 ; Muller et al. , 2021a ) , which are our contemporary studies . Hence , the motivation behind this paper is to provide a solid empirical analysis of the effect of transliteration for MLLMs with statistical analysis and determine if it helps models learn better cross-lingual representation . Several techniques have recently been used to study the hidden representations of multilingual language models . ( Kudugunta et al. , 2019 ) study CLRS of NMT models using SVCCA ( Raghu et al. , 2017 ) . ( Singh et al. , 2019 ) used PWCCA ( Morcos et al. , 2018 ) to study the CLRS of mBERT and found that it drastically falls with depth . ( Wu et al. , 2020 ) have used CKA to study the CLRS of bilingual BERT models . They found that similarity is highest in the first few layers and drops moderately with depth . ( Muller et al. , 2021b ) used CKA to study CLRS of mBERT before and after finetuning on downstream tasks . They found in all cases that CLRS increases steadily in the first five layers , then it decreases in the later layers . From this , they conclude that mBERT learns multilingual alignment in the early layers and preserves it throughout finetuning . A contemporary work ( Del & Fishel , 2021 ) has applied various similarity measures to understand CLRS of various multilingual masked language models . Their results also show that CLRS increases in the first half of the models , while in the later layers , this similarity steadily falls . 3 Experiment and Results . 3.1 Dataset . We pretrained two ALBERT base models from scratch on a subset of the OSCAR corpus . OSCAR corpus is a large multilingual corpus extracted from Common Crawl that contains hundreds of different languages . We use the unshuffled deduplicated version of OSCAR corpus available via Huggingface datasets library ( Lhoest et al. , 2021 ) . We pretrain on Panjabi , Hindi , Bangla , Oriya , Assamese , Gujarati , Marathi , Sinhala , Nepali , Sanskrit , Goan Konkani , Maithili , Bihari , and Bishnupriya portion of the OSCAR corpus . The pretraining dataset details are given in appendix A.5 . We evaluate the models on four downstream tasks from IndicGLUE , which are NCC , WSTP , CSQA , and NER ( Kakwani et al. , 2020 ) . In addition , we evaluate the models on other publicly available datasets . Specifically , on BBC Hindi News Classification , Soham Bangla News Classification , iNLTK Headlines Classification , IITP Movie , and Product Review Sentiment Analysis ( Akhtar et al. , 2016 ) , and MIDAS Discourse Mode Classification ( Dhanwal et al. , 2020 ) datasets .

Summary: In this paper, the authors show explore the following question: is it beneficial to represent different languages in the same script for training pre-training LMs. They explore this question for related languages which share similar scripts (making it easy to transliterate between scripts). The study has been performed on Indo-Aryan languages, which is an important representative of this scenario. The comparison between single-script and multi-script ALBERT language models show the single script model outperforms a multiscript model. The analysis shows that the single-script model vocabularies have a higher level of subword sharing.