Transliteration: A Simple Technique For Improving Multilingual Language Modeling 

1 Introduction . In the last few years , we have seen impressive advances in many NLP tasks . These advances have been primarily led by the availability of large representative corpora and improvement in the architecture of large language models . While improving model architectures , training methods , regularization techniques , etc. , can help advance the state of NLP in general , the unavailability of large , diverse corpora is the bottleneck for most languages ( Joshi et al. , 2020 ) . However , creating representative and inclusive corpora is an ongoing process and is not always possible for many low-resource languages . Thus to inclusively advance the state of NLP across languages , it is crucial to develop techniques for training MLLMs that can extract the most out of existing multilingual corpora . Here we focus on the issue of diverse writing scripts used by closely related languages that may prevent MLLMs from learning good cross-lingual representations . For example , the Indo-Aryan family of languages uses at least six different scripts among them . Previous papers ( Pfeiffer et al. , 2021 ) have noted that low-resource languages that use unique scripts tend to have very few tokens representing them at the tokenizer . As a result , these languages tend to have more UNKnown tokens , and the words in these languages tend to be more split up by sub-word tokenizers . While this issue is evident for Indian languages , there are other examples of closely related languages separated by different scripts such as Serbian and Croatian . Often we can easily transliterate from one script to another using rule-based systems . For example , there are established standards that can be used to transliterate Greek ( ISO 843 ) , Cyrillic ( ISO 9 ) , Indic scripts ( ISO 15919 ) , and Thai ( ISO 11940 ) to the Latin script . In this paper , We focus on the Indo-Aryan language family and empirically measure the effect of transliteration on the downstream performance of MLLMs . The linguistic relatedness in the Indo-Aryan language family is discussed in appendix A.1 . First , we pretrain two ALBERT base ( Lan et al. , 2020 ) models from scratch on the IndoAryan languages of the OSCAR corpus ( Ortiz Su ’ arez et al. , 2019 ) , where one is pretrained with original scripts and the other after transliterating to a common script . Then , we evaluate the models on four diverse downstream tasks from the IndicGLUE benchmark dataset , which are Wikipedia Section Title Prediction ( WSTP ) , News Category Classification ( NCC ) , Named Entity Recognition ( NER ) , and Cloze-Style Question Answering ( CSQA ) . Additionally , we evaluate the models on five different public datasets spanning three different tasks , Article Genre Classification ( AGC ) , Sentiment Analysis ( SA ) , and Discourse Mode Classification ( DMC ) . We use nine random seeds on all downstream finetuning tasks . To rigorously compare the two models , we perform the Mann-Whitney U test ( MWU ) between the uniscript model ( group 1 ) and the multi-script model ( group 2 ) . MWU test lets us verify whether the performance differences of the two models are significant or not . Apart from statistical significance , we also report three different effect sizes to conclusively determine whether the magnitude of improvement due to transliteration is useful in practice . Using the MWU test , we conclude that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages . We also measure the tokenizer quality and the cross-lingual representation similarity ( CLRS ) to understand why the uni-script model performs better than the multi-script model . To measure tokenization quality , we use subword fertility and unbroken ratio ( Ács , 2019 ; Rust et al. , 2021 ) . To measure the CLRS , we use the centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ) similarity score . We measure the CKA similarity score between the hidden representations of the models on the parallel sentences of eight Indo-Aryan languages from the FLORES-101 dataset ( Goyal et al. , 2021 ) . We find that , compared to the multi-script model , the uni-script model achieves a higher CKA score and it is more stable throughout the hidden layers of the uni-script model . Based on this , we conclude that the uni-script model learns better cross-lingual representation than the multi-script model . In summary , our contributions are primarily three-fold : 1 . We find that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages . 2 . We establish this finding through rigorous experiments and show the statistical significance along with the effect size of transliteration using the Mann-Whitney U test . 3 . Using CKA on the FLORES-101 dataset , we show that transliteration helps MLLMs learn better cross-lingual representation . 2 Motivation and Background . In their study , Joshi et al . ( 2020 ) showed the resource disparity between low-resource and high-resource languages , and ( Ruder , 2020 ) discussed the necessity of working with lowresource languages . A large body of work suggests that language-relatedness can help MLLMs achieve better performance on low-resource languages by leveraging related highresource languages . For instance , ( Pires et al. , 2019 ) found that lexical overlap improved mBERT ’ s multilingual representation capability even though it learned to capture multilingual representations with zero lexical overlaps . ( Dabre et al. , 2017 ) showed that transfer learning in the same or linguistically similar language family gives the best performance for NMT . ( Lauscher et al. , 2020 ) found that language relatedness is crucial for POS-tagging and dependency parsing tasks . Although , corpus size is much more important for tasks like NLI and Question Answering . ( Wu & Dredze , 2020 ) demonstrated bilingual BERT can outperform monolingual BERT on low-resource languages when the bilingual languages are linguistically closely related . Nevertheless , mBERT outperforms bilingual BERT when it comes to low-resource languages . However , one of the major challenges in leveraging transfer between high-resource and low-resource languages is overcoming the script barrier . Script barrier exists when multiple closely related languages use different scripts . ( Anastasopoulos & Neubig , 2019 ) found that for morphological inflection , script barrier between closely related languages impedes cross-lingual learning . However , transliteration and phonemebased techniques have been proposed to solve this issue . For example , ( Murikinati et al. , 2020 ) expanded upon ( Anastasopoulos & Neubig , 2019 ) and showed that both transliteration and grapheme to phoneme ( g2p ) conversion removes script barrier and improves cross-lingual morphological inflection and ( Rijhwani et al. , 2019 ) showed that pivoting lowresource languages to their closely related high-resource languages results in better zero shot entity linking capacity and used phoneme-based pivoting to overcome the script barrier . ( Bharadwaj et al. , 2016 ) showed that phoneme representation outperformed orthographic representations for NER . ( Chaudhary et al. , 2018 ) also used phoneme representation to resolve script barrier and adapt word embeddings to low-resource languages . ( Goyal et al. , 2020 ) and ( Song et al. , 2020 ) both utilized transliteration and showed that language relatedness was required for improving performance on NMT . ( Amrhein & Sennrich , 2020 ) studied how transliteration improved NMT and came to the conclusion that transliteration offered significant improvement for low-resource languages with different scripts . ( Khemchandani et al. , 2021 ) showed on Indo-Aryan languages that language relatedness could be exploited through transliteration along with bilingual lexicon-based pseudo-translation and aligned loss to incorporate low-resource languages into pretrained mBERT . ( Muller et al. , 2021a ) showed that for unseen languages , script barrier hindered transfer between low-resource and high-resource languages for MLLMs and transliteration removed this barrier . They showed that transliterating Uyghur , Buryat , Erzya , Sorani , Meadow Mari , and Mingrelian to Latin script and finetuning mBERT on the respective corpus with masked language modeling objective improved their downstream POS performance significantly . ( Dhamecha et al. , 2021 ) showed that transliterating Indo-Aryan languages improved multilingual language model performance . In contrast , ( K et al. , 2020 ) and ( Artetxe et al. , 2020 ) seem to show that multilingual BERT can learn cross-lingual representations without any lexical overlap , a shared vocabulary , or joint training . However , these works focus on zero shot cross-lingual transfer learning where the models are finetuned on one language and evaluated on another . From the literature , it can be seen that apart from g2p , many in the community believe transliteration to be a potential solution for script barriers . However , most of the work shows the benefits of transliteration for NMT . Nevertheless , there is no solid empirical analysis of the effects of transliteration for MLLMs apart from ( Dhamecha et al. , 2021 ; Muller et al. , 2021a ) , which are our contemporary studies . Hence , the motivation behind this paper is to provide a solid empirical analysis of the effect of transliteration for MLLMs with statistical analysis and determine if it helps models learn better cross-lingual representation . Several techniques have recently been used to study the hidden representations of multilingual language models . ( Kudugunta et al. , 2019 ) study CLRS of NMT models using SVCCA ( Raghu et al. , 2017 ) . ( Singh et al. , 2019 ) used PWCCA ( Morcos et al. , 2018 ) to study the CLRS of mBERT and found that it drastically falls with depth . ( Wu et al. , 2020 ) have used CKA to study the CLRS of bilingual BERT models . They found that similarity is highest in the first few layers and drops moderately with depth . ( Muller et al. , 2021b ) used CKA to study CLRS of mBERT before and after finetuning on downstream tasks . They found in all cases that CLRS increases steadily in the first five layers , then it decreases in the later layers . From this , they conclude that mBERT learns multilingual alignment in the early layers and preserves it throughout finetuning . A contemporary work ( Del & Fishel , 2021 ) has applied various similarity measures to understand CLRS of various multilingual masked language models . Their results also show that CLRS increases in the first half of the models , while in the later layers , this similarity steadily falls . 3 Experiment and Results . 3.1 Dataset . We pretrained two ALBERT base models from scratch on a subset of the OSCAR corpus . OSCAR corpus is a large multilingual corpus extracted from Common Crawl that contains hundreds of different languages . We use the unshuffled deduplicated version of OSCAR corpus available via Huggingface datasets library ( Lhoest et al. , 2021 ) . We pretrain on Panjabi , Hindi , Bangla , Oriya , Assamese , Gujarati , Marathi , Sinhala , Nepali , Sanskrit , Goan Konkani , Maithili , Bihari , and Bishnupriya portion of the OSCAR corpus . The pretraining dataset details are given in appendix A.5 . We evaluate the models on four downstream tasks from IndicGLUE , which are NCC , WSTP , CSQA , and NER ( Kakwani et al. , 2020 ) . In addition , we evaluate the models on other publicly available datasets . Specifically , on BBC Hindi News Classification , Soham Bangla News Classification , iNLTK Headlines Classification , IITP Movie , and Product Review Sentiment Analysis ( Akhtar et al. , 2016 ) , and MIDAS Discourse Mode Classification ( Dhanwal et al. , 2020 ) datasets .

Summary: This paper uses transliteration to build better multilingual language models. This is particularly useful for building downstream NLP models for low resource languages. The idea behind this is that script divergences between languages avoid pretrained models to pool resources across language, leading to poor performances on these scripted low-resource languages. Transliteration, i.e. using a common script for all the languages can help solve this issue.   For experiments, the authors focus on Indo-Aryan languages, where they first transliterate all the documents to a common script. Then, the ALBERT model is pre-trained on the transliterated corpora of these languages extracted from the OSCAR corpus. For downstream tasks (section title prediction, news category classification, NER and genre/sentiment classification), they perform fine tuning for each tasks independently, in a multilingual manner when possible. The transliterated models outperform the baseline systems (with no transliteration) on most tasks.