Transliteration: A Simple Technique For Improving Multilingual Language Modeling 
Authors: Ibraheem Muhammad Moosa
Mahmud Elahi Akhter
Ashfia Binte Habib
Venue: ICLR 2022 Submitted (2022)
Keywords: Multilingual Language Model
Natural Language Processing
Transliteration
Underrepresented Language Modeling

Abstract: While impressive performance in natural language processing tasks has been achieved for many languages by transfer learning from large pretrained multilingual language models, it is limited by the unavailability of large corpora for most languages and the barrier of different scripts. Script difference forces the tokens of two languages to be separated at the input. Thus we hypothesize that transliterating all the languages to the same script can improve the performance of language models. Languages of South Asia and Southeast Asia present a unique opportunity of testing this hypothesis as almost all of the major languages in this region have their own script. Nevertheless, it is possible to transliterate them to a single representation easily. We validate our hypothesis empirically by pretraining ALBERT models on the Indo-Aryan languages available on the OSCAR corpus and measuring the model's performance on the Indo-Aryan subset of the IndicGLUE benchmark. Compared to the non-transliteration-based model, the transliteration-based model (termed XLM-Indic) shows significant improvement on almost all tasks of IndicGLUE. For example, XLM-Indic performed better on News Classification (0.41%), Multiple Choice QA (4.62%), NER (6.66%), and Cloze-Style QA (3.32%). In addition, XLM-Indic establishes new SOTA results for most tasks the on IndicGLUE benchmark while being competitive at the rest. Across the tasks of IndicGLUE, the most underrepresented languages seem to gain the most improvement. For instance, for the NER, XLM-Indic achieves 10%, 35%, and 58.5% better F1-scores on Gujarati, Panjabi, and Oriya languages compared to the current SOTA. 

Under review as a conference paper at ICLR 2022
TRANSLITERATION: A SIMPLE TECHNIQUE FOR IM- PROVING CROSS-LINGES LANGUAGE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Impressive performance in natural language processing tasks has been achieved for many languages by transfer learning from large pretrained Multilingual Language Models (MLLM). However, unavailability of large corpora. for most languages currently poses significant hurdle. Thus it is important for MLLMs to extract the most out of existing corpora. In this regard, script diversity presents challenge to MLLMs by reducing lexical overlap between closely related languages. Therefore, transliterat- ing closely related languages that use different writing scripts to common script may improve the downstream task performance of MLLMs. In this paper, we empirically mcasure the effect of translitcration on the perfor- mance of MLLMs by locusing on the Indo-Aryan language family, which has the highest script diversity. We pretrain two ALBERT models from scratch, where one is pretrained with original scripts and the other after transliterating to a common script. Afterward, we evaluate the models on the IndieGLUE benchmark. We perform Mann-W Whitney U test on the per- formance metrics of the models to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages. We also measure the cross-lingual representation similarity (CLRS) of the models using centered kernel alignment (CKA) on parallel sentences of eight languages from the FLORES- 101 dataset. We find that the hidden representations of the transliteration based model have higher and more stable CLRS scores.
INTRODUCTION
In the last few years, we have seen impressive advances in many NLP tasks. These advances have been primarily led by the availability of large representative corpora and improvement in the architecture of large language models. While improving model architectures, training methods, regularization techniques, etc., can help advance the state of NLP in general, the unavailability of large, diverse corpora is the bottleneck for most languages (Joshi et al 2020). However, creating representative and inclusive corpora is an ongoing process and is not always possible for many low-resource languages. Thus to inclusively advance the state of NLP across languages, it is crucial to develop techniques for training MLLMs that can extract the most out of existing multilingual corpora.
Here we focus on the issue of diverse writing scripls used by closely related languages that may prevent MLLMs from learning good cross-lingual representations. For example, the Indo-Aryan family of langnages uses at least six different scripts among them. Previous papers (Pfeiffer et al,, 2021) have noted that low-resource languages that use unique scripts tend to have very few tokens representing them at the tokenizer. As result, these lan- guages tend to have more UNKnown tokens, and the words in these languages tend to be more split up by sub-word tokenizers. While this issue is evident for Indian languages, there are other examples of closely related languages separated by different scripts such as Serbian and Croatian. Often
easily transliterate from script to another us- ing rule-based systems. For example, there are established standards that can be used to transliterate Greek (ISO 843), Cyrillic (TSO 9), Indic scripts (TSO 15919), and Thai (ISO

---

Under review as a conference paper at ICLR 2022
11940) to the Latin script. In this paper, We focus the Indo-Aryan language family and empirically measure the effect of transliteration on the downstream performance of MLLMs. The linguistic relatedness in the Indo-Aryan language family is discussed in appendix A.1.
First, we pretrain two ALBERT base (Lan et al 2020) models from scratch on the Indo- Aryan languages of the OSCAR corpus (Ortiz Su an'arez al., 2019), where one is pretrained with original scripts and the other after transliterating to a common script. Then, we oval- uate the models on four diverse downstream tasks from the IndicGLUE benchmark dataset, which are Wikipedia Section Title Prediction (WSTP), News Category Classification (NCC), Named Entity Recognition (NER), and Cloze-Style Question Answering (CSQA). Addition- ally, we evaluate the models on five different public datasets spanning three different tasks, Article Genre Classification (AGC), Sentiment Analysis (SA), and Discourse Mode Classifi- cation (DMC). We use nine random seeds on all downstream finetuning tasks. To rigorously compare the two models, we perform the Mann-Whitney U test. (MWU) between the uni- script model (group 1) and the multi-script model (group 2). MWU test lets verify whether the performance differences of the two models are significant or not. Apart from statistical significance, whether the magnitude of improvement due to transliteration is useful in practice. Using the MWU lest, we conclude that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages.
also report three different effect sizes to conclusively determine
We also measure the tokenizer quality and the cross-lingual representation similarity (CLRS) to understand why the uni-script model performs better than the multi-script model To
measure tokenization quality, we use subword fertility and unbroken ratio (Acs, 2019 Rust
20210 To measure the CLRS, we use the centered kernel alignment. (CKA) (Kornblith eLall
et
2019) similarity score. We measure the CKA similarity score between the hidden
representations of the models the parallel sentences of eight Anyo-Aryan languages from the FLORES-100 101 dataset (Goyal et al., 2021). We find that, compared to the multi-script model, the uni-script model achieves a higher CKA score and it is more stable throughout the hidden layers of the uni-script model. Based on this, we conclude that the uni-script model learns better cross-lingual representation than the multi-script model.
In summary, our contributions are primarily three-fold:
1. We find that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages.
2. We establish this finding through rigorous experiments and show the statistical significance along with the effect size of transliteration using the Mann-Whitng U
test.
3. Using CKA the FLORES- 101 dataset, we show that transliteration helps MLLMs learn better cross-lingual representation.
2 MOTIVATION AND BACKGROUND
In their study, Joshi et al (202)) showed the resource disparity between low-rcsource and high-resouree languages, and (Ruder, 2020) discussed the necessity of working with low- resource languages. A large body of work suggests that language-relatednees can help MLLMs achieve better performance_on low-resource, resource languages. For instance, (Pires el al., 2019) found that lexical overlap improved mBERT's multilingual representation capability eyen thongh it learned to capture multi- lingual representations with zero lexical overlaps. (Dabre et all, 2017) showed that transfer learning the same or linguistically similar language family gives the best performance for NMT. (Lauscher et al., 2020) found that language relatedness is crucial for POS-tagging and dependency parsing tasks. Although, corpus size is much more important for tasks like NLI and Question Answering. (Wu & Dredze, 2020) demonstrated bilingual BERT can outperform monolingual BERT on low-resource languages when the bilingual languages are linguistically closely related. Nevertheless, mBERT outperforms bilingual BERT when it comes to low-resource languages. However, one of the major challenges in leveraging transfer between high-resource and low-resource languages is overcoming the script barrier. Script
ooreerannno..ooooooosorrereeeeee languages by leveraging related high-
2

---

Under review as a conference paper at ICLR 2022
barrier exists when multiple closely related languages use different scripts. (Anastasopou- los & Neubig, 2019) found that for morphological inflection, script barrier between closely related languages impedes cross-lingual learning. However, transliteration and phoneme- based techniques have proposed to solve this issue For example, (Murikinati et al, 2020) expanded upon (Anastasopoulos & Neubig 2019) and showed that both translit- eration and grapheme to phoneme (g2p) conversion removes script barrier and improves cross-lingual morphological inflection and (Rijjhaani et al, 2019) showed that pivoting low- resource languages to their closely related high-resource languages results in better zero shot entity linking capacity and used phoneme-basce pivoting to overcome the script barrier. (Bharadwaj et all, 2016) showed that phoneme representalion oulperformed orthographic representations for NER. (Chaudhary et al. 2018) also used phoneme represenlation to resolve ve scripi barrier adlpt word embeddings to low-resource languages. (Goyal et al 2020) and (Song et al., 2020) both utilized transliteration and showed that language related- ness was required for improving performance on NMT. (Amrhein & Sennrich. 2020) studied how transliteration improved NMT and came to the conclusion that transliteration offered significant improvement inmprvveettttteeeemennt for low-rcsource languages with different scripts. (Khemchandaní et al,, 2021) showed on Indo-Aryan languages that language relatedness could be exploited through transliteration along with bilingual lexicon-based pseudo-translatiin and aligned loss to incorporate low-resource languages into pretrained mBERT. (Muller et all 202 showed that for unseen languages, script barrier hindered transfer between low-resource and high-resource languages for MLLMs and transliteration removed this barrier. They showed that transliterating Uyghur, Buryat, Frzya, Sorani, Meadow Mari, and Mingrelian to Latin script and finetuning mBERT on the respective corpus with masked language modeling ob- jective improved their downstream POS performance significantly. (Dhamecha et al, 2021) showed that transliterating Indo-Aryan Aaao--ayaa_aannuuaeeeea languages_iipppnnneeeeeeeeee inilingaalm multiiinguull language model performance. In contrast, (K et al. 2020) and (Artetxe et al., 2020) seem to show that multilingual BERT can learn cross-lingual representations without any lexical overlap, shared vocabulary, or joint training. However, these works focus on zero shot cross- lingual transfer learning where the models are finetuned one language and evaluated on another.
From the literature, it can be seen that apart from g2p, many in the community believe transliteration to be a potential solution for script barriers. However, most of the work shows the benefits of transliteration for NMT. Nevertheless, there is no solid, empirical analysis of the_effects of transliteration for MILMs apart from (Dhamecha et ale 2021; Muller al, 2021a), which are our contemporary studies. Hence, the motivation behind this paper is to provide a solid empirical analysis of the effect of transliteration for MLLMs with statistical analysis and determine if it helps models learn better cross-lingual representation.
Several techniques recentlyy been used to study the hidden representations of multilin- gual langnage models. (Kudugunta pt al. 2019) study CLRS of NMT models using SVCCA (Raghu et al., 2017). (Singh et al. 2019) used PWCCA (Morcos et al. 2015) to study the CLRS of mBERT and found that it drastically falls with depth. (Wu et al., 2020) have used CKA to study the CLRS of bilingual BERT models. They found that. similarity is highest in the first few layers and drops moderately with depth. (Muller et al, 2021b) used CKA to study CLRS of mBERT before and after finetuning on downstream tasks. They found in all cases that CLRS increases steadily in the first five layers, then it decreases in the later layers. From this, they conclude that mBERT learns multilingual alignment in the early layers and preserves it throughout finetuning. A contemporary work (Del & Fishel, 2021) has applied various similarity measures to understand CLRS of various multilingual masked language models. Their results also show that CLRS increases in the first half of the models, while in the later layers, this similarity steadily falls.
3 EXPERIMENT AND RESULTS
3.1 DATASET
We pretrained two ALBERT base models from scratch on subset of the OSCAR corpus. OSCAR corpus is a large multilingual corpus extracted from Common Crawl that contains hundreds of different languages. We use the unshuffled deduplicated version of OSCAR
3

---

Under review as a conference paper at ICLR 2022
corpus available via Huggingface datasets library (Lhoest et al., 2021). We pretrain on Pan- jabi, Hindi, Bangla, Oriya, Assamese, Gujarati, Marathi, Sinhala, Nepali, Sanskrit, Goan Konkani, Maithili, Bihari, and Bishnupriva portion of the OSCAR corpus. The pretraining dataset details are given in appendix A.5.
We evaluate the models four downstrean tasks from IndicGLUE, which NCC, WSTP, CSQA, and NER (Kakwani et al. 2020). In addition, we evaluate the models on other pub- licly available datasets. Specifically, on BBC Hindi News Classification, Soham Bangla News Classification, iNLTK Headlines Classification, IITP Movie, and Product Review Sentiment Maalysis(AAhtan) et al. 2016), and MIDAS Discourse Mode Classification (Dhanwal (pasiiia al 2020) datasets.
3.2 PRETRAINING
Corpus Preparation: Since the OSCAR corpus contains raw text from the Web, we apply a few filtering and normalization. First, we discard entries where the dominant script does not match the language tag--tt-geeteee provided by the OSCAR corpus. Then we use the
IndicNLP normalizer (Kunhhuuutt 2020) to normalize the raw text. For XLM-Indic, we then transliterate all the text to 180-15919 format using the Aksharamukhal RRaaan 2015
library.
Next, we train two SentencePiece tokenizers (Kudo & Richardson 2018) on the transliterated and the non-transliterated corpus with a vocabulary size of 50,000. Then we use the trained tokenizer and the sentence-splitter from the IndicNL. library to split long entries from the corpus at sentence boundaries that no entry may have more than 512 tokens. Finally, we discard short entries (<512 characters) to improve the training efficiency.
Tokenizer Training:
ALBERT Model Training: We first pretrained an ALBERT base model from scratch on the non-transliteraaeed corpus as our baseline. Afterward, we pretrained another ALBERT base from scratch on the transliterated corpus. We chose the ALBERT base model due to computing constraints. We trained the models were trained using the same hyperparameters. We followed the hyperparameters used in (Lan et al. 2020) except for batch size and learning rate. The pretraining objective is also the same ase (Lan et al., 2020). The hyperparameter values and dotails are presented in the appendix A. Each model took about 7.5 days to train. We use the ALBERT implementation from the Huggingface Transformers Library (Wolf et al., 2020). Henceforth, unless mentioned otherwise, the non-transliierationn or multi-script model will be called the baseline model, and the transliteration or uni-script model will be called ALM-Indic.
single TPUv3 VM. Both models
3.3 STATISTICAL ANALYSIS
We perform statistical analysis to determine if the performance differences between base- line and XLMM-ndic are significant. In short, the statistical analysis tells us the effect of transliteration on model performance For this purpose, we perform Mann Whitney U test (MWU) (Mann& & Whitney, 1947: Wilcoxon, 1945). MWU is a non-parametrie hypothesis test between two groups/ populations. MWU chosen because it has weak assumptions. The only assumptions of MWU are that the samples of the two groups are independent of each other, and the samples are ordinal. Under the MWU, our null hypothesis or ho is that the performances of XLM--nnic (group 1) and baseline (group 2) are equivalent, and the alternative hypothesis or h2 that the performances (groups) are non-equal. We set our confidence interval at 0.05 and reject the ho for p-values three test. statistics as p-valne only give statistical significance, which can be misleading at times (Sulivvan & Feinn, 2012).
We also report
The test statistics are three different effect sizes that convey three different information. These test statistics are absolute effect size (0), common language effect size (p ), and standardized effect size. The absolute effect size is the difference between the mean of the models' performance metric, which is given as,
4

---

Under review as a conference paper at ICLR 2022
8 /4BMalidde
for any given task and language. For any given task, if the ho is rejected, positive ò indicates XLM-Indic is better, and a negative ō indicates the baseline is better. gives us the probability of superiority of one group being better performing between given two groups. That is the probability that a random performance sample of XLM-Indic is greater than a random performance sample of the bascline. The last test statistics standardized effect size which indicates the magnitude of difference between the performance values of XLM- Indic (group 1) and baseline (group 2). Standardized effect size shows us how realistically significant the performance differences are between models even if the performance difference statistically signíficant. It gives us a value between to 1. and its ranges are: small effect ( (0 S std. effect I 0.3) medium effect ( 0.3 < std. effect 0.5) and large effect (0.5 < std. effect). We show the standardized effect size only on public datasets. We performed MWU all downstream tasks CSQAp On C CSQA,wo only report the 8. The MWU is performed using the SciPy library (Virtanen et al., 2020), and the results are further validated using (2017, 2017
3.4 DOWNSTREAM FINETUNING
We finetune the models on each downstream task independentlyy The hyperparameters are selected based on the recommendations in (Mosbach al 2021). The specific hyperpa- rameters used for each task are reported in the appendix A.4 On all tasks, we finetune with nine random seeds and report the average and standard deviation of the metrics. In Table I and Table g. we report the performances on IndicGLUF benchmark tasks and in Table 2 on other publicly available datasets. Here we discuss each of the tasks and com- pare the performance of M-Idic with the baseline baseeelie model. We show the results from IndicBERT (Kakwani (Kkkwai et al. 2020) and (Dhamecha et al., 2021) as independent baselines.
On some tasks, the performance of XLM-Indic is observably better than the baseline. For WSTP and NER, XLM-Indic shows improvement by several percentage points over the baseline all languages. Conversely, on other tasks, the difference is muanced, and it is not clear whether XLM-Indic is actually performing better or not. An example of this can be seen in the iNT.KK Gujarati News Classification task shown in Table 2. We discuss these nuanced results in-depth in the latter part of this section under our statistical analysis.
Wikipedia Section Title Prediction This is a multiple-choice question-answer task from IndicGLUE, On this task, we predict the title of a Wikipedia article section from four choices. However, we Irain a sequence classification model instead of a multiple-choice question-answer model for simplicity. This lask the same across all languages. Thus we simultaneously finetune the model on all these languages. In Table 1 we report the test set accuracy of different models on this task. Here, XLM-Indic performs significantly better than the baseline. A p value of for all languages indicates that XLM- Indic will always perform better than the basclinc. The 8 value of Oriya (7.12), Gujarati (6.02), and Assamese (5.04) indicates that translitcration helps comparatively low-resource languages more compared to Hindi (4.06) and Bengali (3.02).
News Category Classification: This is a collection of news article classification tasks from the IndicGLUE benchmark on five languages. The dataset sizes vary widely. For example, the Oriya dataset is about ten times the Gujarati dataset. Thus this task helps us test the finetuning capability of the models on classification tasks with different dataset sizes. In Table D. we report the test set accuracy of different models on this task. We see our first failure to reject the null hypothesis case for the Bengali and Marathi classification tasks. Even though the 8 is negative, we can not reject the null hypothesis for Bengali and Marathi as the p-values are 0.181 and 0.1683, respectively. The p for these two tasks show outperform the baseline only 31% of the time. For Panjabi (8 of 1.07) and Oriya (8 of 0.68), the p. indicates that XLM-Indic can always outperform baseline. However, for Gujarati (8 of 0.60), XLM-Indic can outperform the baseline 80% of the time.
that XLd-Innic
5

---

Under review as a conference paper at ICLR 2022
Table Results on Classification Tasks from IndicGLUE Benchmark
Model
bi
bnn
Wikipedia Title Prodiction
Independent Bascliue.
mBRRT (lanmcha
80.81
78.08
84.38
71.90
XLM-R Kakwank
70.29
76.32
80.31
68.25
1/G-316
27.29
77.44
6G544
IndieBER base Ealeram at
2021
67.59
74.02
80.11
57.14
65.82
68.79
72.56
60.60
IndieDERT large Kakkwaii al DTA
77.54
77.80
82.008
88.25
$6.99
52.22
77.44
70.41
Oura
Boseline
14.33:00.85
78:18:0033
$1.18+0:28
76.70:+0.55
76.97++0.59
79.10+t0.84
X1.M-fndic
77.55++0.61
92.26-+0.18
$4.35+0.29
$1,47+0.000
81.74+0.82
82.39++0.27
82.74/0.52
81.78
Test statisties
0.22
4.06
3.20
7.12
5.04
6.02
3.64
4.61
proalon
0.0034
0.0004
0.0004
0.0004
0.0004
0.0004
0.0004
Category Classification Independent Baselbup
XLM-R Kakwani et al Qu2u
94.87
519. 221
977.07
966.15
965.67
9H6.61
mBERT Kakwani at al 2020
94.87
97.71
60.55
6-1.02
96.07
68.64
IndieEERT base Chllouan IET large Knkwwai
97.44
97.14
97.33
100.00
96.07
97.72
94.87
97.71
97.60
73.08
95.00
01.65
96.83+0-119
9S.
05.80+0044
89.58+00225
XIdM-Indic
07.00110.17
07,00.10.22
08.77.10.12
00.00.10.56
00.47.10.21
08.70
Test statlstics
1.07
-0.35
0.68
0.60
-0.18
0.40
0,00XS
0.0004
NER (Fi-Score)
Independent Boscline
mBRRT Dhanocha
50.00
86.56
011.81
19.05
02.31
68.04
91.22
71.29
XLMRR Rakwinn ot Maly
17.86
89.82
912.95
25.00
66.07
05.32
87.
62.11
IndieBERt base Salewam
2021
21.45
90.30
93.39
8.80
$1.87
56.74
88.71
56.00
IidieMEEK large Nakwaai
86.81
Oure
Baseline
76 6.:
01.80:10.2 42
06.30:10.1 10
84.181118
75.45:1:.8
60 10.10:12..0
88.72.10.00
83.10
MIM--nnde
85.42-113 1.0
02.03-0.21
07.3 07.31-+0.22
99. 09.54++0.58
90.06+22.2
80.16+00115
90.56-+0-44
89.85
Test statistics
6.08
0.0004096
0.0004096
0.0003963
0.0004038
0.000401
0.0004096
0.0004005
oonnge indicates baseline and XLM XLd--Indie are equal and blue indicates XLM Indic is better
Named Entity Recognition: We use the balanced Wikiann datasct from Rahimi et al. (2019) for this task. Similar to WSTP, we simultancously finetune the modols on all lan- guages. In Table 1 report the Fl-scores of different models. As per the p-value, XLM- Indic's s improvement over baseline is statistically significant. We specifically see that as per XLM-Indic makes significant improvements NER. The p for all languages indicate that XLM-Indic can always outperform the baseline on all Indo-Aryan family of languages. Similar to WSTP, the å values show that transliter- ation helps low-resource languages more. For instance, the 5 for Panjabi, Oriya, Assamese, and Gujarati are 8.73, 9.36, 13.61, and 11.06, respectively. We can also see improvement on Hindi (1.13), Bengali (0.92), and Marathi (1.84) NER.
Panjabi, Oriya, Assamese, and Gujarati
Table 2: Results on Public Datascts
Langunge
Datasee
Iwvi:BERT-hee
Basellon
XLM--nnie
Tea Statistio
Aucoray
Zlasurt
Arcinay
E5-Scre
peralue
Stasctedied dac Fleenre
Artidl Gesro Classification
Newo
T430
Solare Artidd Chosifrotion
(05681
ISLII Finditee
12.01
141 11.0120.00
0.32
U5587
0.59
6.13
londlize
94.30
212112.11
01.2155134
#3J0111L75
-0.14
01920
Sentiment Anabyste
HTT Poofuct Rasiows
70 50:50-444
710050000
74.3240666
Ic
HET Monce Bistews
20L0
33.11222
10.3022.1
12 BL344003
10. 0L.773.5
0.61
015457
0.50
0.5
Discouree Mode Classfiicttin
MIDAS Descum
76
27.
16.54
C00004
cesapr XLM--astt aquad Indiato NLMi-ladi better

---

Under review as a conference paper at ICLR 2022
Article Genre, Sentiment & Discourse Mode Classification: We evaluate the models on publicly available sequence classification datasets. Most of these classification tasks are highly skewed. Thus report F1 scores in addition to accuracy on these tasks. We encourage others to do so in the future. As stated earlier, apart from á and A we also report standardized effect size these datasets their difference as the IndieGLUE tasks. These tasks show one of the major reasons why multi- seeded results along statistical analysis are imperative. We present our results in Table b. We only show the test statistics for the Fi-score here. The test statistics for accuracy are provided in appendix A.6. For BBC News classification, we can see that even though the 8 for Hindi is 2.12, there is no stalistically significant difference in the models' performance. The null hypothesis can not be rejected due to the p-value being 0.4363, and the standardized effect size is small (0.19). Both of these are due to the standard deviation of the baseline model, which is (#4.6). However, p indicates that XLM-Indic would outperform the baseline 62% of the time. The Soham Articles classification, a Bengali news classification task with six classes, is marginally rejected due to the p-value being 0.05031. Nevertheless, the standardized effect size is modium (0.46), and per p, XLM-Indic outperforms the baseline 78% of the time. For INLTK Headline classification task, we see that both models are equivalent for Gujarati and Marathi with p-value of 0.5457 and 0.6665, respectively. Both of them have small effect size (Gujarati 0.15 and Marathi 0.10). The p indicates that Indic outperforms the baseline 59% of times for Gujarati and 43% of times for Marathi. On sentiment analysis tasks, we see a similar trend. Both models are equivalent performance wise with a p-value of 0.3865 for ITTP Product Review and 0.5457 for ITTP Movie review. The standardized effect size for the tasks is 0.21 and 0.15, respectively, which indicates that the standardized effect is small for both tasks. Apart from that, as per p. XLM-Inde Indic outperforms the baseline 63% of times for IITP Product review and 59% of the time for ITTP Movic review. Lastly, for Discourse Mode Classification, we see that Indic outperforms the baseline with a d value of 16.54 and p-value of 0.00004. The standardized effect is large (0.83) and indicates that XLM-Indic will always outperform the baseline.
not pronounced
Effect of Transliteration: In general, we can see that the effect of transliteration is mostly positive. However, it does not hurt the model's performance either, as we can see that the models performed cquivalently in all the tasks where the null hypothesis could not be rejected. Transliteration mostly helps the comparatively low-resource languages (Panjabi, Oriya, Assamese), while high-resource languages also (Hindi, Bengali) see good improvements seen in WSTP and NER tasks. As for transliteration not hurting model performance, we can see that in Table & excluding Discourse Mode Classification, in all other tasks, XLMi-nndic performance Classification had a statistically significant performance difference.
equivalent to the baseline. While Discourse Mode
3.5 ZERO SHOT CAPABILITY TESTING
We use the CSQA task to test the zero shot capability of the models as we can use the pretrained masked language models without fineluning on this task. On CSQA, we are given four entities to choose from to fill in a masked entity in a sentence. This task is designed to test whether language models can be used as knowledge bases (Petroni et al., 2019). In Table g we report the results. We see that XLM-Indic performs better on all languages. The 8 is particularly impressive in Gujarati (10.36), Assamese (3.44), and Marathi (3.39). This indicates that XLM-Indic is better at representing entity knowledge than the baseline model. This is not surprising many entity names have the same spelling regardless of the script used. XLM-Indic maps these to the same token, which helps it to construct better knowledge representation. We found some issue with IndieBERT evaluation code for this task. We discuss this issue in appendix A.2. We evaluate IndicBERT with our code and report. the results in Table
7

---

Under review as a conference paper at ICLR 2022
Table 3: Test accuracy on CSQA
Model
pa
bn
gu
avg
Cloze-style QA (Zero Shot)
Independent Baseline
IndicBERT base (Our Evaluation)
29.33
30.76
28.45
30.38
29.98
81.50
29.71
37.16
Ours
Baseline
31.04
36.72
35.19
34.63
33.92
59.86
36.
38.21
XL.M-Indic
32.77
38.52
36.38
36.00
37.36
70.22
39.53
41.54
5
1.73
1.8
1.19
1.37
3.44
10.36
3.39
3.33
orange indicates baseline and XLM-Indic equal and blue indicates Xndi-Indi is better than baseline
4
WHY TRANSLITERATION WORKS
In this section, analyze why XLM-Indic performs better than the baseline model from two perspectives, tokenization quality, and CLRS.
4.1 TOKENIZER QUALITY ANALYSIS
As discussed and 2 we expect the transliteration model to exploit better tokenization across the languages. Whereas the same word written in six different scripts gets mapped to six different tokens in the baseline model, these get mapped to the same token in the translitcration model. Thus we expect the tokenizer to more efficiently tokenize a given text. in the case of the transliteration model.. Following Acs 2019) and (Rust et al., 2021).
Seselre
Seaeline
69
(b) Unbroken Ratio. Figure 1: Subword fertility (lower is better) and unbroken ratio (higher is better)
(a) Subword Fertility.
we mcasure the subword fortility (average mymber of tokens per word) and the ratio of words unbroken by the tokenizer. From figure ! we can that transliteration reduces the splitting of words. This indicates that many words that were represented by different tokens in the baseline model are represented by a single token in the transliteration model.
4.2 CROSS-LINGUAL REPRESENTATION SIMILARITY
Following (Muller al. 2021b) (Wu et al 2020) and (Del & Fishel 2021 we pepapply CKA to
measure CLRS. We use the CKA implementation from the Ecco library (Alammar 2021
We use parallar sentences on eight Indo-Aryan languages from the FL.RRF--10 101 Goyal
et al. 2021) dataset. This is dataset of professionally translated parallel sentences 101 languages. We only consider Panjabi, Hindi, Bengali, Oriya, Assamese, Gujarati, Marathi, and Nepali sentences from the FLORES- 101- datasct these are the only ones present in the pretraining corpus.
First, extract the hidden state representations of the models on these parallel sentences.
Then for each language pair, we calculate the CKA similarity score between the hidden state representations of the corresponding layers. For example, we calculate the CKA similarity
8

---

Under review as a conference paper at ICLR 2022
6.1
045
(b) XLM- Figure 2: CKA Similarity Score for the bascline model and XLM-Indic
(a) Baseline
score between layer eight hidden state representations of Bengali and Hindi parallel tences. Then for each, language, we average its CKA similarity scores with other languages per layer. In Figure 2 we plot this average CKA similarity for each layer of the models for each language. We see that XLM-Indic retains high CLRS score throughout the model. On the other hand, the CLRS score drops in the middle and end layers of the bascline model. Overall the CLRS score of XLM-Indic is more stable. This indicates that XLM-Indic has learned better gross-lingual representations. We show the individual similarity figures all language pairs in appendix A.9.
5 CONCLUSION AND FUTURE WORK
In this paper, we found that transliterating closely related languages to a common script improves MLLM performance and leads to learning better CLRS. We established this by conducting rigorous empirical and statistical analysis to quantify the statistical significance and effect size of transliteration on MLLM for Indo-Aryan languages which have the largest script diversity. We found that transliteration especially improved the performance of com- paratively low-resource languages and did not hurt the performance high-resource lan- guages_ This findings are in agreement with contemporary literature (Dhamecha et al, 2021; Muller et al.. 2021a). Our results indicate that other scenarios where closcly re- lated languages from language family use different scripts, transliteration can be used to improve MILM performance. For example, the Dravidian, South Slavic, and Turkic Family of languages present. similar scenarios. We would also like to implore peers to include more hypothesis tests their studies against a strong baseline. Non-parametrie tests like MWU are easy use and have a high level of interpretability, In future, we would like to measure the impact of transliteration on larger models and more languages, Also, character- based models hold more potential to exploit the lcxical overlap enabled by translitcration. Efficient Transformer models that can efficiently handle long sequences can be used to train these character-level models.
REPRODUCIBILITY STATEMENT
We have taken crea ensure our results reproducible. The dataset details are pro- vided, in subsection 3.1 and the data processing sleps and andtraiini details are described in 3.2. The MWU implementation details are provided in 3.3. Finetuning hyperparameters are provided in appendix A.4 We have also provided our code as supplementary material,
REFERENCES
Statistics Kingdom 2017. Mann whitney u test calculator.
com/170median mann_ mann_whitned www.weiit.....m..mmm htmml 2017.
9

---

Under review as a conference paper at ICLR 2022
Judit Ács. Exploring BERT's Vocabulary. Blog Post. 2019. URL http:///udiaacs..iiiiatoa 1o/2019/02/19/bert ottp:///eerttttttiiiiioiooooo--o on--sttssssssssss stats.html,
Md. Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. Aspect based sentiment anal- ysis: Category detection and sentiment classification for hindi. In proceedings of the 17th International Conference on Intelligent Tezt Processing and Computational Linguistics (CICLING 2016), April 3-9, 2016, Konya, Turkey, 2016.
J Alammar. Ecco: An open source library for the explainability of transformer lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing: System Demonstrations, pp. 249 257, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18658/v1/2021. acl-mmm..3 URL attps://aclanthhlogg. org/2021. http//22..ccl-daeee sell-dmmo... 30.
Chantal Amrhein and Rico Sennrich. On Romanization for model transfer between scripts in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2461-2469, Online, November 2020. Association for Computational Linguistics. doi: 10. 18.18 vv (2220..indiin... emn.p..23. URL https://aclanthoology
org/2020. org/2020.findings findingsemnlll 223.
Antonios Anastasopoulos and Graham Neubig. Pushing the limits of low-resource morpho- logical inflection. In EMNLP, 2019.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transfer- ability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623 4637, Online, July 2020. As- URL
sociation for Computational Linguistics. doi: 10.18653 acl-000.cclmmaain httpss://ccaanthoo..g org/2020. pocl-main.40
Akash Bharadwaj, David Mortensen, Chris Dyer, and Jaime Carbonell. Phonologically aware neural model for named entity recognition in low resource transfer settings. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1462 1472, Austin, Texas, November 2016. Association for Computational Linguistics. wtw.eyooog...---- 1153.
doi: 10. 10.18653/ //1//vv///// v1/D16-1158 URL
Aditi Chaudhary, Chunting Zhou, Lori S. Levin, Graham Neubig, David R. Mortensen, and Jaime G. Carbonell. Adapting word embeddings to new languages with morphological and phonological subword representations. In EMNLP, 2018.
R.A.E. Coningham, F.R. Allchin, C.M. Batt, and D. Lucy. Passage to india? anuradha- pura and the early use of the brahmi script. Cambridge Archaeological Journal 6(1): 73 97. April 1996. doi:
URL https://dde org/ /tt//11.11077 1017/
$0959774300001608.
Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. An empirical study of language related- ness for transfer learning in neural machine translation. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, pp. 282-286. The National University (Phillippines), November 2017. URL http:///ccaanntoooooorr/////////
1038
Maksym Del and Mark Fishel. Establishing interlingua in multilingual language models. ArXio, 22102219901207,, 2021.
Tejas Dhamecha, Rudra Murthy, Samarth Bharadwaj, Karthik Sankaranarayanan, and Pushpak Bhattacharyya. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages. In Proceedings of the 2021 Conference on Empérical Methods in Natural Language Processing, pp. $584-8594 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational 675.
Linguistics. URL https://acllnthollsse wrw.//22/222202 wwwp:20111...22..eeee2222222.pm1
Swapmil Dhanwal, Hritwik Dutta, Hitesh Nankani, Nilay Shrivastava, Y. Kumar, Junyi Jessy Li, Debanjan Mahata, Rakesh Gosangi, Haimin Zhang, R. R. Shah, and Amanda Stent. An annotated dataset of discourse modes in hindi stories. In LREC, 2020.
10

---

Under review as a conference paper at ICLR 2022
Naman Goyal, Cynthia Gao, Vishrav Chauchary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjan Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 evaluation benchmark for low-resource and multilinuual machine transla- tion. ArXio, abs/2106.031993, 2021.
Vikrant Goyal, Sourav Kumar, and Dipti Misra Sharma. Efficient neural machine trans- lation for low-resource languages via exploiting related languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 162-168, Online, July 2020. Association for Computational Linguistics, doi: 10.18653/ v1 w
URL https://acllnnthologgy.ooo www./000......ooooo. acl-srw.
Charles F. Hockett, Peter T. Daniels, and William Bright. The world's writing systems. Language, 73(2):379, June 1997. doi: 10.2307/416027, URL https://doi.oon orgo 2307/ 416027.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Lönguistics, pp. 6282 6293, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 acl-main.560. CRL
org/2020 meee-mmin...m 560.
Karthikeyan K. Zihan Wang, Stephen Mayhow, and Dan Roth. Cross-lingual ability of multilingual BERT: an empirical study. In 8th International Conference on Learning Represendations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nu 2020. URL https: https://openneviieet mett/forumid--tJeTTe
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. IndieNI.Puite: Monolingual corpora, eval- uation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4948 4961, Online, November 2020. Association for Computational Linguistics. 220200.indings-eeee 445
Linguutticssiii doi: 10.18653, www.fffi-ninnennnnn
URL
Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1312 1323, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ve //2//22222c2a//// 105. URL https://aclanthhll
wwwporggg...///.../ 2021 acl -ong..lo..... 105.
Diederik P. Kingma and Jimmn Ba. Adam: A method for stochastic optimization. CoRR, abs/ 1412.690,, 2015.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ArXiv, hts/1905.00414, 2019.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. Investigating multilin- gual NMT representations at scalc. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on pp. 1565 1575, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18633/11D119-1177
Natural Language Processing (EMNLP-IJCNI)
URL httpl//aaabbb////see///////aaaaa 2020.
KinchnaattaaanTTeeeennneelPPlaal http://hwwww..aapo////otttota/aa wttp://ccooaooooiiiie comm/anooppumccuuuuttaaa indic andic_nlp_1s
11

---

Under review as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942 2020.
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online, November 2020, Association for Computational Lin- guistics. doi: 10.18653/v v1 /ttp:///0202000mmmmmmmm 2020 emnlp-main.
(EMNLP). pp.
URL bttpp://acclatthologg org/
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexan- der Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic. November 2021. Association for Computational Linguisties. URL https://aclanthollogy org/2021. 1022222..1mmpp...mmmmmmmm mmmlp-demo.21
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Associalion for Compulational Linguistics: Volume 2, Short Papers, Pp. 8 14, Valencia, Spain, April 2017. Association for Computational Linguisties. URL https: //aclanthology. 200/
H. B. Mann and D. R. Whitney. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1): 50 - 000 1947. doi: 2512144aoms/1111 1177730491
URI. https://dod. /ttp:::///oooor/10../ 1214/ //aoms//
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. In NeurIPS, 2018
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine- misconceptions, explanations, and strong baselines. In 9th International Conference on Learning Represenlations. ICLR 2021. Virlual Evend, Austria, May 3-7, 2021. OpenReview.nn net, 2021. URL https://openrevierr met/forumm/ wiw.:mdddddnnndddnn
luning BERT:
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. When being unseen from mBERTT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 448 462, Online, June 2021a. Association for Computational Linguistics. doi: 10, 18653 11/2021.naacl-maan
URL https:///ccaatthooogg org/2021. maacl-mmin.33.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. First align, then predict: Understanding the cross-lingual ability of multilingual bert. In EACT, 2021b.
Nikitha Murikinati, Antonios Anastasopoulos, and Graham Neubig. Transliteration for cross-lingual morphological inflection. In Proceedings of the 17th SIGMORPHON Work- shop on Compulational Research in Phonelics, Phonology, and Morphology, pp. 189 197, Online, July 2020. Association for Computational Linguistics. doi: 10. 18.18653 /v1 /v11/22200.0. /2020. sigmorphon- 1.22. URL https: attpps:/acclattholla
org/ 20gg/2220..oo sigmorphon-1.... 1.22
Pedro Javier Ortiz Su' li'a Benoit Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Piotr Bañski, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz,
12

---

Under review as a conference paper at ICLR 2022
Harald L"ungen, and Caroline Iliadi (eds.), Proceedings of the Workshop Challenges in the Management of Large Corpora (CMLC-7) 2019, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July
2019, pp. 9 9- 16, Mannheim, 2019. Leibniz-Institut f"ur Deutsche Sprache. doi: 10.14618 ids-pub-9021. URL. http://nnn-resool wwwrriing..ddddeeeee /ururrr:///// :d:::::::::::::::::: :e:::d::bb:a:::e matass::m39-99220
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint. Conference on Natural Language Processing (EMNIP-IJCNNLP),, pp- 2463 2473, Hong Kong, China, November 2019. Association for Computational Linguis- tics. doi: 10.18653/ 20118533vv/Bhl 11/ 1250. URL
Jonas Pfeiffer, Ivan Vuli'e, Iryna Gurevych, and Sebastian Ruder. Unks everywhere: Adapt- ing multilingual language models to new scripts. In EMNLP, 2021.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Associalion for Compulational Linguistics, pp. 4996 5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: URL ftttpa//acllamtoooog..oo/// org//19-149/
10.18653/v1/P11 1493.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NIPS, 2017.
Afshin Rahimi, Yuan Li, and Trevor Cohn. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Associalion for Compulational Linguistics, pp. 151 164, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://wwwg hctps://www.allvebb.o aclweb. org/ http://r.rrrrrattooooo/////-0000
Vinodh Rajan, Aksharammkka transliteration tool. http:://ggittu..cooau/rrrttaaall aksharamukha wwtpaaauuuaaaaahoo.r.... python, 2015. Accessod: 2021-10-04.
Shruti Rijhwani, Jiateng Xie, Graham Neubig, and Jaime G. Carbonell. Zero-shot neural
transfer for cross-lingual entity linking. In AAAI, 2019.
Sebastian Ruder, Why You Should Do NLP Beyond English.
http://ruder.it io/
nlp-beyond seeeyondd-nggiith asg.ish. 2020.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Långuisties and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3118 3135, Online, August 2021. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2000 2a2.0021.clloooo lcl--oong.245 URL https:
//aclanthology 202g//2221110 acl-long. .43.
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. Bert is not an inter- lingna and the bias of tokenization. In EMNLP, 2019.
Haiyue Song, Raj Dabre, Zhuoyuan Mao, Fei Cheng, Sadao Kurohashi, and Eiichiro Sumita. Pre-training via leveraging assisting languages for neural machine transla- tion. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Lânguistics: Student Research Workshop, pp. 279-285, Online, July 2020, Asso- ciation for Computational Linguistics. doi: 10.18653/w1/2020.aac /2020..cl-ss..33.. URL https: //aclanthology. , wrtpe200....1..........
G. M. Sullivan and R. Feinn. Using Effect Size-or Why the P Value Is Not Enough. J Grad Med Educ, 4(3):279 282, Sep 2012.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, (CML'1,, pp. 3319 3328. JMR.Rorrg 2017.
13

---

Under review as a conference paper at ICLR 2022
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Sté- fan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay May- orov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Fric Larson, C Carey, Ilhan Polat, Yu Feng, Fric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimr- man, lan Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261- 272, 2020. doi: 10.1038/s41592-00 -19--099----9-----2-
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics ((): December 1945. doi: 01.2307/300196. URL https://doi. https:://oi..or/11..200000 2307/3092237
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistios. doi: 10.18653/v 10t///1120000 12020. emnlp-demos.66 6. CRL bttps://aclanthan
org/2020. wet20200.mmmmmemm.m... emnlp-demos.6
Shijie Wu and Mark Dredze. Are all languages created equal in multilingual BERT? In Pro- ceedings of the 5th Workshop Represenlation Learning for NLP, pp. 120- 130, Online, July 2020, Association for Computational Linguistics. doi: 10.18653/v1 //200///1//000rrp///lpll 16. URL attps://accantthologg/ org/ /0200/2200222 rep1411p-1.16
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zeltlemoyer, and Veselin Stoyanov. Emerging cross- lonsullingual structure in pretrained language models. In ACL, 2020.
A APPENDIX
A.1 LINGUISTIC RELATEDNESS IN INDO-ARYAN FAMILY
Many South Asian and Southeast Asian languages are intimately connected linguistically, historically, phonologically (Lttelll et al, 2017) and phylogenetically. However, due to dif- ferent scripts, it is difficult for MLLMs to fully exploit. this shared information. Among the languages we considered in this study we encounter six different scripts. These are shown in Table Nevertheless, Lhese these scripis have shared ancestry from ancient Brahmic script (Hockett et al., 1997: Coningham et al,, 1996) and have similar structures that we can easily use to transliterate them to a common script. The 1SO-15919 standard has been designed to perform this transliteration. Also, many of these languages heavily borrow from Sanskrit, and due to influence, many words are shared among these languages. Therefore, due to their relatedness and highly diverse script barrier, this family of languages presents unique opportunity to analyze the effects of transliteration on MLLMs.
A.2 CLOZE STYLE QA INDICBERT EVALUATION ISSUE
Since a word can be tokenized to multiple tokens by the subword tokenizer, correctly eval- uating the model on this task requires special care. Specifically, we have to use the same number mask tokens as the number of subword tokens that a word gets split into. Then calculate the probability for the word by multiplying the probability of the subword to- kens predicted by the masked language model. We found that on the IndicBERT evaluation code, only single mask token was used irrespective of the number of subword tokens that word gets split into. We do not think this is the correct way to evaluate a masked language model on this task.
14

---

Under review as a conference paper at ICLR 2022
A.3 PRETRAINING HYPERPARAMETER:
We used batch size of 256, which is the highest that fits into TPU memory, whereas the ALBERT paper used a batch size of 4096. As our batch size is 1/16th the ALBERT paper,
learning rate of 1e-3/8, which approximately 1/16th of the learning rate used in the ALBERT paper (1.76e-2). Additionally, we use the Adam Optimizer(KKngnan & Ba 2015) instead of the LAMB optimizer. The of the hyperparameters were the same as the ALBERT paper. Specifically, we use sequence length of 512 with absolute positional encoding, weight docay of 1c-2, warmup steps of 5000, max gradient norm of 1.0, and Adam epsilon of 1c-6. The models were trained for 1M steps.
A.4 DOWNSTREAM HYPERPARAMETERS
All of our hyperparameters for downstream tasks presented in Table 4 The batch size was chosen to be the maximum that fits in memory. This was done so that each batch contains approximately the same number tokens. Otherwise the hyperparameters were chosen following the recommendations of (Mosbach et al, 2021). On the highly ITTP Movie Review, ITTP Product Review and MIDAS Discourse found that this default setting resulted in worse performance compared to the independent baselines. So we finetuned the learning rate and classifier dropout on the validation set of these tasks.
Table 4: Hyperparameters for all tasks
Task
TPU
Batch Size
Learning Rate
Weight Decay
Dropout
Epochs
Warmup Ratio
Category Classiliation
Fala:
Wilcpodia Section- Title Prediction
True
256
26-5
0.01
0.1
0.10
Named Kntity Revognition
Thse
612
26-5
0.01
0.1
20
0.10
BBC Hindi News ClassiBaation
Falae:
26-5
0.01
0.3
0.10
Solam Bongln News Chossfication
False
Headlines Classification
ITT Movie Review
False
0.01
0.25
ITT Product Review
Fabse
5e-5
0.01
0.5
20
0.10
MIDAS Discourse Mode
Falon
26-5
0.01
0.5
0.10
A.5 DATASET DETAILS
Here we provide the corpus size details.
Table 5: Pretraining dataset details
Language
Pretraining corpus size GB
Language Sub-family
Script
hi
8.9
Central Indo-Aryan
Devanagari
bn
5.8
Eastern Indo-Aryan
Bengali-Assanese
mr
1.4
Southern Indo-Aryan
Devanagari
1.2
Northern Indo-Aryan
Devanagari
0.783
Insular Indo-Aryan
Sinhala.
gu
0.705
Western Indo-Aryan
Gujarati
pa
0.449
Northwestern Indo-Aryan
Gurmukhi
or
0.18
Eastern Ardo-Aryan
Oriya
0.069
Eastern Indo-Aryan
Bengali-Assanese
0.036
Sanskrit
Devanagari
bpy
0.0017
Eastern Indo-Aryan
Bengali-Assamess
gom
0.0017
Southern Indo-Aryan
Devanagari
bh
0.000034
Eastern Indo-Aryan
Devanagari
mai
0.000011
Eastern Indo-Aryan
Devanagari
15

---

Under review as a conference paper at ICLR 2022
A.6 PUBLIC DATASETS ACCURACY TEST STATISTICS
The MWU p-values and test statistics for public datasets accuracy is given in Table B We can see that for BBC News (p-value 0.0088) and Soham Articles Classification (p-value 0.0090) XLM-Indic is better than the baseline with a 6 of 1.86 and 0.67, respectively. Both lasks have large standardized effect size (0.62 for both). However, as per P3 XLM--ndic outperforms the baseline 87% of the time. Whereas, the p for Soham News Article Classifi- cation is 0.57. As for INLTK Headlines, XLM-Indic and the baseline perform equivalently. On INLTK Headlines, the p-value for Gujarati (6 of 0.32) is 0.6249 and Marathi (ō of -0.17) 0.3503. On ITTP Product Reviews, XLM-Indic outperforms the basclinc with a of 0.85, p-value of 0.4099 and p of However, the standardized effect size is medium (0.48) for the task. In contrast, on ITTP Movie Reviews, both models are equivalent performance wise with 8 of 0.15, p-value of 0.8941, P of 0.52 and small (0.031) standardized effect. Finally, we can see that both models performing equally on Discourse Mode Classification. The is 0.15 with p-value of 0. 0.61 and small (0.073) standardized effect size. However, per the p, XL.M-Indic outperforms the baseline 45% of the time.
Table 6: Public datasets test statistics accuracy
Language
Dataset.
Test Statistics
8
p-value
Standardized effect size.
Article Genre Classification
BBC News
1.86
0.0088
0.62
bn
Soham News Article Classification
0.67
0.0090
0.57
0.62
gu
INLTK Hcadlines
0.32
0.6249
0.57
0.12
mi
INLTK Headlines
-0.17
0.3503
0.36
0.22
Sentiment Analysis
hi hi
ITTP Product Reviews
0.85
0.04099
0.79
0.48
HITP Movie Reviews
0.15
0.8941
0.52
0,031
Discourse Mode Classification MIDAS Discourse
hi
0.15
0.7561
0.45
0.073
orange indicates baseline and XLM-Indic are equal and blue indicates XLM-Indic is better
A.7 CASE STUDY ON SOHAM BANGLA NEWS CLASSIPICATION:
In order to better understand the contrast between classification decision of XLM- and baseline model, using the Layer Integrated Gradients method (Sundararajan et al,, 2017) as a local self-cxplaining method, we performed multiple out of distribution adversarial exam- ples to check the input features attribution difference between XLM-Indic and the bascline. The experiments were performed using Soham Bangla News Classification finetuned mod- els. Soham Bangla News Classification was chosen due to it being a multi-class classification lask. It has six classes which are 'entertainment' "international", "kolkata', 'national', 'sport' and 'state.. The adversarial examples were created by mixing class dependant words. For example, "They won the award for their acting performance." should be classified as enter- tainment. However, "They won the penalty by acting injured." should be classified as sports instead of ontertainment. In our experiments, XLM-Indic tended to perform better com- pared to baseline and capture more correct class dependant words instead of the adversarial ones. However, providing certain class dependant keyword (L.e. 'police', 'political, 'actor', 'footballer) let the baseline model fix its missclassificaiin. We provide one such example below.
We used the example, "Ma शाशा 9 आभालरर शरििस कालक रि रोड शान शामा लف लाক ক (") which translates to, "He used to act insane the side of the road; however, he would
16

---

Under review as a conference paper at ICLR 2022
Legend: II Negative D Neutral Positive
True Label
Predicted Label
Word Importance
2
LABEL_4 (0.98)
[CLS]
00
4130
9129
[SEP]
Attribution Label Attribution Score
LABEL_4
Figure 3: Feature importance for baseline
Legend: @ Negative D Neutral B Positive
True Label
Predicted Label
Word Importance
2
LABEL (0.41)
[CLS)
rastrra pi8
dēra
abtinaya
karato
kintu
räla
hale
suyign
péléi
dakin
larati [SE?]
Attribution Label Attribution Score
LABEL 2
0.29
Figure 4: Feature importance for XLMM-ndic
rob whenever he got the chance at night.". The true label for this example under Soham News Classification should be 'state'. Which is label 2 in the dataset. Here act is used as class dependant adversarial word to deviate the models towards 'entertainmentt. Whereas, would like the models to attribute 'rob', which is the the correct word Figure 4 shows the word attributions for the baseline model. We can see that the baseline classified the sentence "entertainment" (label 4) and put positive attribution पाकमसा, which means act. On the other hand it put negative attribution XLM-Innic correctly classified the sentence and puts the expected positive and negative attributions on 'rob' and 'act' respectively. Nevertheless, as stated earlier, allowing certain class dependant keywords fixes the baselines misclassification. For example, we found that adding 'Police report states that' in front of our provided example fixed the baseline models prediction to the correct class.
डलকल, which means rob. However,
A.8 PRETRAINING LOSS CURVES
Our pretraining loss curve provided in figure
Pretraining Loss
Baseline XLM-Indic
Loss
2
200
400
600
800
1000
Steps
Figure 5: Pretraining loss: Baseline vs Transliteration model
17

---

Under review as a conference paper at ICLR 2022
A.9 CROSS-LINGUAL SIMILARITY OF ALL LANGUAGE PAIRS
603
010
605
0.75
440
(a) Baseline PA-X
(b) XLM-Indie PA-X
013
0.02
600
(c) Baseline HI-X
(d) XLM-Indic HI-X
010
635
010
665
0411
(e) Baseline BN-X
(f) XLM-Indic BN-X
8x
311
6.0
010
0
050
(g) Baseline OR-X
(h) XLM-Indic OR-X
Figure 6: CKA of Baseline and MLMM-Tndi all language pairs for pa, hi,bn and
18

---

Under review as a conference paper at ICLR 2022
93
674
010
(a) Baseline AS-X
(b) XLM-Indic AS-X
gux
(c) Baseline GU-X
(d) XLM-Indi GU-X
633
010
605
0.41
Layers
(e) Baseline MR-X
XLMiIndic MR-X
213
0.02
3411
040
3
(g) Baseline NE-X
(h) XLM-Indic NE-X
Figure 7: CKA of Baseline and XLM-Indic on all language pairs
19


Reviews:
[{"review_id": "uUhsBQxGbeM", "reviewer": "Reviewer_XxGN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths - \n1) The paper is well written and easy to follow.\n2) Strong results with a thorough study on many languages and also statistical tests. \n3) A simple yet effective way to increase the sharing of information across languages.\n\nWeaknesses - \n1) Transliteration is not formally defined, which is leading to authors not making a connection with the works that do the same thing but by representing them into phonemes.\n2) Additionally, this leads to being restricted to only the Indo-Aryan languages and not expanding to a much larger set of languages around the world, by using various available g2p tools.\n3) The paper seems to have also missed a lot of research in this direction which do almost exactly the same thing, for cross-lingual downstream NLP tasks or infact multilingual language modeling itself. I have listed a few of them as part of the general remarks. \n4) The core-idea and the contributions made by this paper is not very novel. It's been used a lot in the field of speech and NLP. Although the authors apply this work to a new setup and on a new corpora, the key takeaways are not new or surprising.\n\nGeneral Remarks - \n1) In section 3.2 : “As our batch size is 1/16th of the ALBERT paper, we use a learning rate of 1e-3/8, which is approximately 1/16th of the learning rate used in the ALBERT paper (1.76e-2).”  You could instead be doing accumulation of gradients instead of change the learning rates. By doing that you can increase your effective batchsize with limited compute. \n2) From your code it seems that you used hugging face implementation for the model training and tuning and evaluation? It would be good to mention that in the paper.\n3) “However, we would like to mention that, in this paper our purpose is not to achieve the SOTA but to understand the impact of transliteration on the performance of language models.”   → the paper is mentioning SOTA and results everywhere (even half of the abstract) and is studying very quickly transliteration in the last section (section 4) in about 15 lines and one figure which is not very striking and not very commented\n4) The authors mention \"However, we would like to mention that, in this paper our purpose is not to achieve the SOTA but to understand the impact of transliteration on the performance of language models.”. But the paper does seem to be focused almost entirely on results with the final section on studying the transliteration. \n5) Expanding to this point, it would be good to look at unrelated languages and get a sense of the subword fertility in both related cluster and unrelated cluster of languages.\n6) In section 3.4 - the authors mentioned that they modified the evaluation code. I always wary on the side of caution about editing evaluations. It would be good to get it verified by the IndicBERT github codebase. \n7) It would probably be a good idea to make a relation with the works that use phoneme representations for language modeling and downstream speech and NLP tasks. I have listed a few that come to my mind below - \n\n\n[a] Phonologically Aware Neural Model for Named Entity Recognition in Low Resource Transfer Settings, EMNLP 2016\n\n[b] Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning, NAACL 2016\n\n[c] Zero-shot Neural Transfer for Cross-lingual Entity Linking, AAAI 2019\n\n[d] Phoneme Level Language Models for Sequence Based Low Resource ASR, ICASSP 2019\n\n[e] Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation, ACL 2019\n\n[f] On Romanization for Model Transfer Between Scripts in Neural Machine Translation, EMNLP 2020\n\n[g] Using Phoneme Representations to Build Predictive Models Robust to ASR Errors, ACM SIGIR 2020\n\n[h] Phonotactic Complexity and Its Trade-offs, TACL 2020\n", "summary_of_the_paper": "This paper uses transliteration to build better multilingual language models. This is particularly useful for building downstream NLP models for low resource languages. The idea behind this is that script divergences between languages avoid pretrained models to pool resources across language, leading to poor performances on these scripted low-resource languages. Transliteration, i.e. using a common script for all the languages can help solve this issue. \n\nFor experiments, the authors focus on Indo-Aryan languages, where they first transliterate all the documents to a common script. Then, the ALBERT model is pre-trained on the transliterated corpora of these languages extracted from the OSCAR corpus. For downstream tasks (section title prediction, news category classification, NER and genre/sentiment classification), they perform fine tuning for each tasks independently, in a multilingual manner when possible. The transliterated models outperform the baseline systems (with no transliteration) on most tasks. \n", "summary_of_the_review": "The authors perform extensive experimentations on many languages and perform statistical tests by training many models with different random seeds. The paper is also well written and easy to understand. However, there are several limitations in the scope and relation to prior work. Few major limitation in my opinion are - (1) Not relating transliteration to grapheme to phonemes. In fact this realization would have encouraged the authors to expand from Indo-Aryan languages to a much larger set of languages. Additionally, this would have also have led to nice set of relation to prior work. (2) Although it's important to consider internationalization of language technologies, the core-idea and the contributions made by this paper is not really novel. A lot of prior work have already shown the efficacy of this direction of research.\n\n**After the Author Response** -- \nThank you for the detailed response and the changes in the paper. The paper has indeed improved quite a bit from its previous shape. As the other reviewers also pointed out, I am still concerned about the novelty of the work so I am inclined to keep my score the same. However, with the new changes I think now the paper brings some insights that could be useful to some readers in the community. I would highly encourage the authors to submit this paper in a core NLP conference. Finally, please refrain from making significant edits between revisions as it really increases the burden on the reviewers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gDxkuElvrLj", "reviewer": "Reviewer_yNTm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThrough extensive experiments on a large number of tasks, the paper shows that a single script model is good for constrained scenarios when working with related languages: \n\n- Small vocabulary (50k)\n- high degree of parameter sharing (ALBERT)\n- trained on limited corpora (OSCAR corpora)\n\nA compact model trained on single script for related languages can be competitive/better than larger capacity models trained with multi-script data of a much larger magnitude. These findings are in line with concurrent work by Dhamecha et al (2021). \n\nTejas Indulal Dhamecha, Rudra Murthy V, Samarth Bharadwaj, Karthik Sankaranarayanan, Pushpak Bhattacharyya. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages. EMNLP. 2021.\n\n**Weaknesses**\n\n- Studies on the impact of vocabulary size, pre-training corpus size, and fine-tuning corpus size will be beneficial in understanding if transliteration is beneficial in constrained scenarios only. Some previous work indicates that large multilingual models can implicitly map-related languages into a common representation space even if the scripts are different. \n\n   - Kudugunta, S. R., Bapna, A., Caswell, I., Arivazhagan, N., & Firat, O. (2019). Investigating multilingual NMT representations at scale. EMNLP.\n   - Karthikeyan K, Zihan Wang, Stephen Mayhew, Dan Roth. Cross-Lingual Ability of Multilingual BERT: An Empirical Study. ICLR 2020. \n\n**Questions**\n\n- How are the hyperparameters selected? The same hyperparameters have been used for finetuning all models, is there a bias in the selection of these hyperparameters like early stopping, learning rate, etc for certain models? Kindly clarify.\n\n**Comments**\n\n- Please include citations and URLs for the IndicNLP and Aksharmukha library.\n   - Kunchukuttan, Anoop. \"The indicNLP library.\" Indian language NLP Library (2020). \n   - https://github.com/virtualvinodh/aksharamukha-python\n\n   ", "summary_of_the_paper": "In this paper, the authors show explore the following question: is it beneficial to represent different languages in the same script for training pre-training LMs. They explore this question for related languages which share similar scripts (making it easy to transliterate between scripts). The study has been performed on Indo-Aryan languages, which is an important representative of this scenario. The comparison between single-script and multi-script ALBERT language models show the single script model outperforms a multiscript model. The analysis shows that the single-script model vocabularies have a higher level of subword sharing.", "summary_of_the_review": "The main contribution of the paper - that transliteration can help multilingual language models is an important observation for related languages. Though many in the community have believed this hypothesis, there were no solid empirical data to prove this for NLU.  Some work exists to show the benefits of single script mapping for NMT. Hence, this work fills that gap. Concurrent work by Dhamecha et al (2021) also confirms the major finding of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f54X-3G6up1", "reviewer": "Reviewer_rME7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n* The paper suggests a simple existing transliteration method (ISO-15919 standard) to convert Indic native characters into latin-based format. The transliterated format of a word tends to be shared across different Indic characters of different languages. This may have helped the model learn these languages better.\n* The paper's proposal achieves good performance of multiple tasks in the South Asia languages.\n\nWeaknesses:\n* The paper considerably lacks novelty and does not meet the standard for ICLR. This method of transliteration already exists and have been adopted in other areas of NLP. The actual procedure of transliteration is also not invented by the authors, but taken from existing work. The paper simply suggests to convert text from one format to another by a known method, which I think is not novel.\n* The method may not work for other languages, such as Chinese, where a transliteration from native Chinese character to Pinyin (probably) may not work well.\n* Only Indic languages are explored. And I doubt this method would work in other languages with the same situation as South Asia languages, such as Chinese, Japanese, Korean; or other Asian languages like Thai, Malay, and Vietnamese.\n* The presentation of the paper is quite poor. The method section is confusing, too much text, and difficult to comprehend and imagine. A simple diagram to aid understanding would make the presentation much better.\n\n", "summary_of_the_paper": "The paper proposes to replace the native characters and words of some South Asia low-resource languages into latin-based transliteration using a rule-based converter and use these transliterated data to pre-train cross-lingual language model. The linguistic sharing of the transliterated format of these languages enable the model to alleviate out-of-vocabulary and limited data size to improve performance on the IndicGLUE benchmark, which is only focused on South Asia languages.", "summary_of_the_review": "Overall, I think the paper significantly lacks novelty as only suggest an adoption of an existing method in a straightforward way that anyone would have thought of. The performance contribution is limited to a subset of languages and there is limited prospect that it would work in other languages and application.\n\n**AFTER READING AUTHOR RESPONSE**\n\nI have read the author response and very appreciate the authors' effort in responding. Here are my afterthoughts:\n1. The novelty concern remained the same. The similar argument that \"method X is applied in task A but never in B, so it's novel\" has appeared frequently in past ICLR conferences. The general consensus is that this does not have sufficient technical novelty, as we value technical novelty in terms of machine learning techniques. It would have made a difference if the authors propose a novel technique Z based on X that makes things better for task B.\n2. The authors acknowledge that this may only work in Indic languages, so it has limited application.\n3. The paper, and the responses, show that the paper has lots of linguistic contribution. So I guess it will have better chance in core NLP venues, such as EMNLP.\n4. I change the score to 5 to appreciate the authors' effort in the response and revision. However, this still does not warrant an acceptance for me.\n5. The authors responded very late, on the last date of the deadline, which is not appreciative.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WQX3RjHabcD", "reviewer": "Reviewer_M3CH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of this paper is that it provides a simple solution to the script suffering in language modeling, and compares its performance with several multilingual pretrained models as well as the non-transliteration version of the model. The transliteration ALBERT shows better performance over the non-transliterated version. The main weakness is that it only compares the performance within several Indian languages. And its claims may not stand in a larger scope. For example, this work tends to claim that tokens with the same spellings in different languages carry the same meanings. This claim is hard to stand. For instance, ‘burro’ has unrelated meanings in Spanish and Italian respectively, which are close languages under the same language family. And the lexical mapping may not be one-to-one even for different spelling systems of the same language like Simplified and Traditional Chinese (https://aclanthology.org/2020.acl-main.648.pdf). Additionally, it is not clear to see that shared tokens collapse to the same subword from the subword fertility analysis in Section 4. Finally, the experiments are not sufficient because script conversion and distinguishing closeness of languages are important problems in conducting this method. However, they are not fully discussed because the authors only select a group of very closed languages, and therefore cannot resolve the concerns when this technique is applied. In fact, since many languages can be Romanized, languages like Greek and Russian can be considered to test the hypothesis in this work.\n", "summary_of_the_paper": "This paper talks about whether uniform script representation can alleviate the dominance of language-specific corpus size and therefore benefit cross-lingual language modeling. The authors train two ALBERTs on non-transliterated and transliterated (ISO-15919) corpus (filtered and normalized version of OSCAR). Those two models are further fine-tuned for four classification/sequence labeling tasks in IndicGLUE. The result shows that XLM-Indic achieves better or comparable performance in most settings. They also test the model’s zero shot capability in cloze style QA, and XLM-Indic also achieves better or comparable performance. By measuring lexical fertility, they claim that the success of shared scripts comes from that low-resource languages borrow better representation of shared lexical from other languages where those lexical are frequently used.", "summary_of_the_review": "In summary, this work studies an important problem and tries to provide an effective solution. The experiment is done on a small scale and limited setting, and many underlying hypotheses of this work are not widely applied based on the insufficient results. I would not recommend its acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HdOudkm054d", "reviewer": "Reviewer_vmSB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The idea of introducing transliteration into MLLM is interesting, where the ALBERT is pretrained secondarily on the OSCAR corpus.\n2. The authors empirically validate their XLM-Indic under different settings. Besides the performance of downstream tasks after finetuning, results of zero-shot capability testing are also provided. \n3. This paper also simply discusses why transliteration can improve the performance of MLLM by analyzing the subword fertility and unbroken ratio.\n\nCons:\n1. One main concern about this paper is that the contribution is not enough. The purpose of this paper is to use the transliteration technique to improve the performance of MLLMs. However, only ALBERT is selected for validation (Other representative MLLMs such as mBERT, XLM are not discussed.). And there is no significant novelty of improving ALBERT to XLM-Indic such as better model architecture or pretraining objective (at least the improvement is not well presented in the paper). \n2. The connection between transliteration technique and performance gain (on downstream tasks) is vague. Experimental results show that XLM-Indic (ALBERT size) significantly outperform existing MLLMs, while it is difficult for readers to get the intuition from the paper about how the performance gain is achieved, although Section 4 tries to provide some reasons (which are insufficient).\n3. Many necessary details are missing. For example, the objective of pretraining ALBERT to XLM-Indic is not provided (only the hyper-parameters are provided in the appendix). There is no introduction about the details of XLM-Indic model. In addition, there is limited discussion about existing work. Using translation-based models to improve the performance of MLLMs is a widely-used strategy, and it would be better if authors can discuss and compare their XLM-Indic with existing methods. \n\n \nMinor comments: \n* In Tables 2&3, it would be nice to see the performance of downstream tasks on some common languages such as English. That would help readers to figure out whether knowledge of common languages is forgotten after the secondary pretraining.\n* In Tables 5&6, the results about ablation study are a bit confusing. It would be better if a clearer description about the assumptions can be provided.\n", "summary_of_the_paper": "This paper hypothesizes that transliterating all the languages to the same script can improve the performance of multilingual language models (MLLM). Experiments are designed to validate this hypothesis by comparing the performance of ALBERT and transliteration-based ALBERT (“XLM-Indic”). The authors find that XLM-Indic outperforms ALBERT across many downstream tasks such as text classification and QA. The main contribution is that the authors empirically validate the effectiveness of introducing transliteration technique on improving the overall performance of MLLMs. The improvement is obvious especially for underrepresented languages.", "summary_of_the_review": "The paper does not provide a significant contribution and many necessary details are missing. It lacks the substance and depth to merit a strong recommendation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors show that it is possible to overcome the script barrier in MLLMs by using transliteration. In effect, they show that transliterating all text to a single script improves the performance for low-resource languages. They also provide additional analysis in the form of statistical tests and crosslingual representation analysis to substantiate their claims.

The main concerns raised by the reviewers are: 
(i) lack of novelty: the idea of using transliteration has been extensively studied in the context of NMT, Speech. It has also been studied in the context of MLLMs by some recent work (which can be considered to be contemporary). IMO, this is a concern.
(ii) focus on Indic languages: there are some concerns raised about the broader applicability of the techniques presented in the paper (personally, I disagree with this concern as Indic languages are important - for example, there are numerous papers which only report results on En-De, En-Ru translation)
(iii) limited evaluation: the technique is evaluated only using the ALBERT model and other configurations (such as ROBERTA, XLM, etc) are not considered. IMO, it would have helped if the authors presented results on these models also (at least we would know if transliteration only helps in the case of small/compact models or even in the case of large models)
(iv) missing references: there is a large body of related work on NMT, speech, etch which the authors had missed in their initial draft. This has been rectified in the updated version.

The reviewers did participate in the discussion with the meta-reviewer (not with the authors though) and even after looking at the revised draft mentioned that the novelty is limited. 

To summarise my views, I think the initial draft of the paper did need improvements and the final draft is a significantly improved version of the initial draft. However, I still feel the novelty is missing. Even the empirical novelty claimed by the authors is ;lacking due to the use of a single model (ALBERT).
Decision: Reject