Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper uses transliteration to build better multilingual language models. This is particularly useful for building downstream NLP models for low resource languages. The idea behind this is that script divergences between languages avoid pretrained models to pool resources across language, leading to poor performances on these scripted low-resource languages. Transliteration, i.e. using a common script for all the languages can help solve this issue. 

For experiments, the authors focus on Indo-Aryan languages, where they first transliterate all the documents to a common script. Then, the ALBERT model is pre-trained on the transliterated corpora of these languages extracted from the OSCAR corpus. For downstream tasks (section title prediction, news category classification, NER and genre/sentiment classification), they perform fine tuning for each tasks independently, in a multilingual manner when possible. The transliterated models outperform the baseline systems (with no transliteration) on most tasks. 

main_review: Strengths - 
1) The paper is well written and easy to follow.
2) Strong results with a thorough study on many languages and also statistical tests. 
3) A simple yet effective way to increase the sharing of information across languages.

Weaknesses - 
1) Transliteration is not formally defined, which is leading to authors not making a connection with the works that do the same thing but by representing them into phonemes.
2) Additionally, this leads to being restricted to only the Indo-Aryan languages and not expanding to a much larger set of languages around the world, by using various available g2p tools.
3) The paper seems to have also missed a lot of research in this direction which do almost exactly the same thing, for cross-lingual downstream NLP tasks or infact multilingual language modeling itself. I have listed a few of them as part of the general remarks. 
4) The core-idea and the contributions made by this paper is not very novel. It's been used a lot in the field of speech and NLP. Although the authors apply this work to a new setup and on a new corpora, the key takeaways are not new or surprising.

General Remarks - 
1) In section 3.2 : “As our batch size is 1/16th of the ALBERT paper, we use a learning rate of 1e-3/8, which is approximately 1/16th of the learning rate used in the ALBERT paper (1.76e-2).”  You could instead be doing accumulation of gradients instead of change the learning rates. By doing that you can increase your effective batchsize with limited compute. 
2) From your code it seems that you used hugging face implementation for the model training and tuning and evaluation? It would be good to mention that in the paper.
3) “However, we would like to mention that, in this paper our purpose is not to achieve the SOTA but to understand the impact of transliteration on the performance of language models.”   → the paper is mentioning SOTA and results everywhere (even half of the abstract) and is studying very quickly transliteration in the last section (section 4) in about 15 lines and one figure which is not very striking and not very commented
4) The authors mention "However, we would like to mention that, in this paper our purpose is not to achieve the SOTA but to understand the impact of transliteration on the performance of language models.”. But the paper does seem to be focused almost entirely on results with the final section on studying the transliteration. 
5) Expanding to this point, it would be good to look at unrelated languages and get a sense of the subword fertility in both related cluster and unrelated cluster of languages.
6) In section 3.4 - the authors mentioned that they modified the evaluation code. I always wary on the side of caution about editing evaluations. It would be good to get it verified by the IndicBERT github codebase. 
7) It would probably be a good idea to make a relation with the works that use phoneme representations for language modeling and downstream speech and NLP tasks. I have listed a few that come to my mind below - 


[a] Phonologically Aware Neural Model for Named Entity Recognition in Low Resource Transfer Settings, EMNLP 2016

[b] Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning, NAACL 2016

[c] Zero-shot Neural Transfer for Cross-lingual Entity Linking, AAAI 2019

[d] Phoneme Level Language Models for Sequence Based Low Resource ASR, ICASSP 2019

[e] Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation, ACL 2019

[f] On Romanization for Model Transfer Between Scripts in Neural Machine Translation, EMNLP 2020

[g] Using Phoneme Representations to Build Predictive Models Robust to ASR Errors, ACM SIGIR 2020

[h] Phonotactic Complexity and Its Trade-offs, TACL 2020

summary_of_the_review: The authors perform extensive experimentations on many languages and perform statistical tests by training many models with different random seeds. The paper is also well written and easy to understand. However, there are several limitations in the scope and relation to prior work. Few major limitation in my opinion are - (1) Not relating transliteration to grapheme to phonemes. In fact this realization would have encouraged the authors to expand from Indo-Aryan languages to a much larger set of languages. Additionally, this would have also have led to nice set of relation to prior work. (2) Although it's important to consider internationalization of language technologies, the core-idea and the contributions made by this paper is not really novel. A lot of prior work have already shown the efficacy of this direction of research.

**After the Author Response** -- 
Thank you for the detailed response and the changes in the paper. The paper has indeed improved quite a bit from its previous shape. As the other reviewers also pointed out, I am still concerned about the novelty of the work so I am inclined to keep my score the same. However, with the new changes I think now the paper brings some insights that could be useful to some readers in the community. I would highly encourage the authors to submit this paper in a core NLP conference. Finally, please refrain from making significant edits between revisions as it really increases the burden on the reviewers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors show explore the following question: is it beneficial to represent different languages in the same script for training pre-training LMs. They explore this question for related languages which share similar scripts (making it easy to transliterate between scripts). The study has been performed on Indo-Aryan languages, which is an important representative of this scenario. The comparison between single-script and multi-script ALBERT language models show the single script model outperforms a multiscript model. The analysis shows that the single-script model vocabularies have a higher level of subword sharing.
main_review: **Strengths**

Through extensive experiments on a large number of tasks, the paper shows that a single script model is good for constrained scenarios when working with related languages: 

- Small vocabulary (50k)
- high degree of parameter sharing (ALBERT)
- trained on limited corpora (OSCAR corpora)

A compact model trained on single script for related languages can be competitive/better than larger capacity models trained with multi-script data of a much larger magnitude. These findings are in line with concurrent work by Dhamecha et al (2021). 

Tejas Indulal Dhamecha, Rudra Murthy V, Samarth Bharadwaj, Karthik Sankaranarayanan, Pushpak Bhattacharyya. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages. EMNLP. 2021.

**Weaknesses**

- Studies on the impact of vocabulary size, pre-training corpus size, and fine-tuning corpus size will be beneficial in understanding if transliteration is beneficial in constrained scenarios only. Some previous work indicates that large multilingual models can implicitly map-related languages into a common representation space even if the scripts are different. 

   - Kudugunta, S. R., Bapna, A., Caswell, I., Arivazhagan, N., & Firat, O. (2019). Investigating multilingual NMT representations at scale. EMNLP.
   - Karthikeyan K, Zihan Wang, Stephen Mayhew, Dan Roth. Cross-Lingual Ability of Multilingual BERT: An Empirical Study. ICLR 2020. 

**Questions**

- How are the hyperparameters selected? The same hyperparameters have been used for finetuning all models, is there a bias in the selection of these hyperparameters like early stopping, learning rate, etc for certain models? Kindly clarify.

**Comments**

- Please include citations and URLs for the IndicNLP and Aksharmukha library.
   - Kunchukuttan, Anoop. "The indicNLP library." Indian language NLP Library (2020). 
   - https://github.com/virtualvinodh/aksharamukha-python

   
summary_of_the_review: The main contribution of the paper - that transliteration can help multilingual language models is an important observation for related languages. Though many in the community have believed this hypothesis, there were no solid empirical data to prove this for NLU.  Some work exists to show the benefits of single script mapping for NMT. Hence, this work fills that gap. Concurrent work by Dhamecha et al (2021) also confirms the major finding of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper talks about whether uniform script representation can alleviate the dominance of language-specific corpus size and therefore benefit cross-lingual language modeling. The authors train two ALBERTs on non-transliterated and transliterated (ISO-15919) corpus (filtered and normalized version of OSCAR). Those two models are further fine-tuned for four classification/sequence labeling tasks in IndicGLUE. The result shows that XLM-Indic achieves better or comparable performance in most settings. They also test the model’s zero shot capability in cloze style QA, and XLM-Indic also achieves better or comparable performance. By measuring lexical fertility, they claim that the success of shared scripts comes from that low-resource languages borrow better representation of shared lexical from other languages where those lexical are frequently used.
main_review: The main strength of this paper is that it provides a simple solution to the script suffering in language modeling, and compares its performance with several multilingual pretrained models as well as the non-transliteration version of the model. The transliteration ALBERT shows better performance over the non-transliterated version. The main weakness is that it only compares the performance within several Indian languages. And its claims may not stand in a larger scope. For example, this work tends to claim that tokens with the same spellings in different languages carry the same meanings. This claim is hard to stand. For instance, ‘burro’ has unrelated meanings in Spanish and Italian respectively, which are close languages under the same language family. And the lexical mapping may not be one-to-one even for different spelling systems of the same language like Simplified and Traditional Chinese (https://aclanthology.org/2020.acl-main.648.pdf). Additionally, it is not clear to see that shared tokens collapse to the same subword from the subword fertility analysis in Section 4. Finally, the experiments are not sufficient because script conversion and distinguishing closeness of languages are important problems in conducting this method. However, they are not fully discussed because the authors only select a group of very closed languages, and therefore cannot resolve the concerns when this technique is applied. In fact, since many languages can be Romanized, languages like Greek and Russian can be considered to test the hypothesis in this work.

summary_of_the_review: In summary, this work studies an important problem and tries to provide an effective solution. The experiment is done on a small scale and limited setting, and many underlying hypotheses of this work are not widely applied based on the insufficient results. I would not recommend its acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper hypothesizes that transliterating all the languages to the same script can improve the performance of multilingual language models (MLLM). Experiments are designed to validate this hypothesis by comparing the performance of ALBERT and transliteration-based ALBERT (“XLM-Indic”). The authors find that XLM-Indic outperforms ALBERT across many downstream tasks such as text classification and QA. The main contribution is that the authors empirically validate the effectiveness of introducing transliteration technique on improving the overall performance of MLLMs. The improvement is obvious especially for underrepresented languages.
main_review: Pros:
1. The idea of introducing transliteration into MLLM is interesting, where the ALBERT is pretrained secondarily on the OSCAR corpus.
2. The authors empirically validate their XLM-Indic under different settings. Besides the performance of downstream tasks after finetuning, results of zero-shot capability testing are also provided. 
3. This paper also simply discusses why transliteration can improve the performance of MLLM by analyzing the subword fertility and unbroken ratio.

Cons:
1. One main concern about this paper is that the contribution is not enough. The purpose of this paper is to use the transliteration technique to improve the performance of MLLMs. However, only ALBERT is selected for validation (Other representative MLLMs such as mBERT, XLM are not discussed.). And there is no significant novelty of improving ALBERT to XLM-Indic such as better model architecture or pretraining objective (at least the improvement is not well presented in the paper). 
2. The connection between transliteration technique and performance gain (on downstream tasks) is vague. Experimental results show that XLM-Indic (ALBERT size) significantly outperform existing MLLMs, while it is difficult for readers to get the intuition from the paper about how the performance gain is achieved, although Section 4 tries to provide some reasons (which are insufficient).
3. Many necessary details are missing. For example, the objective of pretraining ALBERT to XLM-Indic is not provided (only the hyper-parameters are provided in the appendix). There is no introduction about the details of XLM-Indic model. In addition, there is limited discussion about existing work. Using translation-based models to improve the performance of MLLMs is a widely-used strategy, and it would be better if authors can discuss and compare their XLM-Indic with existing methods. 

 
Minor comments: 
* In Tables 2&3, it would be nice to see the performance of downstream tasks on some common languages such as English. That would help readers to figure out whether knowledge of common languages is forgotten after the secondary pretraining.
* In Tables 5&6, the results about ablation study are a bit confusing. It would be better if a clearer description about the assumptions can be provided.

summary_of_the_review: The paper does not provide a significant contribution and many necessary details are missing. It lacks the substance and depth to merit a strong recommendation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to replace the native characters and words of some South Asia low-resource languages into latin-based transliteration using a rule-based converter and use these transliterated data to pre-train cross-lingual language model. The linguistic sharing of the transliterated format of these languages enable the model to alleviate out-of-vocabulary and limited data size to improve performance on the IndicGLUE benchmark, which is only focused on South Asia languages.
main_review: Strength:
* The paper suggests a simple existing transliteration method (ISO-15919 standard) to convert Indic native characters into latin-based format. The transliterated format of a word tends to be shared across different Indic characters of different languages. This may have helped the model learn these languages better.
* The paper's proposal achieves good performance of multiple tasks in the South Asia languages.

Weaknesses:
* The paper considerably lacks novelty and does not meet the standard for ICLR. This method of transliteration already exists and have been adopted in other areas of NLP. The actual procedure of transliteration is also not invented by the authors, but taken from existing work. The paper simply suggests to convert text from one format to another by a known method, which I think is not novel.
* The method may not work for other languages, such as Chinese, where a transliteration from native Chinese character to Pinyin (probably) may not work well.
* Only Indic languages are explored. And I doubt this method would work in other languages with the same situation as South Asia languages, such as Chinese, Japanese, Korean; or other Asian languages like Thai, Malay, and Vietnamese.
* The presentation of the paper is quite poor. The method section is confusing, too much text, and difficult to comprehend and imagine. A simple diagram to aid understanding would make the presentation much better.


summary_of_the_review: Overall, I think the paper significantly lacks novelty as only suggest an adoption of an existing method in a straightforward way that anyone would have thought of. The performance contribution is limited to a subset of languages and there is limited prospect that it would work in other languages and application.

**AFTER READING AUTHOR RESPONSE**

I have read the author response and very appreciate the authors' effort in responding. Here are my afterthoughts:
1. The novelty concern remained the same. The similar argument that "method X is applied in task A but never in B, so it's novel" has appeared frequently in past ICLR conferences. The general consensus is that this does not have sufficient technical novelty, as we value technical novelty in terms of machine learning techniques. It would have made a difference if the authors propose a novel technique Z based on X that makes things better for task B.
2. The authors acknowledge that this may only work in Indic languages, so it has limited application.
3. The paper, and the responses, show that the paper has lots of linguistic contribution. So I guess it will have better chance in core NLP venues, such as EMNLP.
4. I change the score to 5 to appreciate the authors' effort in the response and revision. However, this still does not warrant an acceptance for me.
5. The authors responded very late, on the last date of the deadline, which is not appreciative.
review_title: null

Meta-review: The authors show that it is possible to overcome the script barrier in MLLMs by using transliteration. In effect, they show that transliterating all text to a single script improves the performance for low-resource languages. They also provide additional analysis in the form of statistical tests and crosslingual representation analysis to substantiate their claims.

The main concerns raised by the reviewers are: 
(i) lack of novelty: the idea of using transliteration has been extensively studied in the context of NMT, Speech. It has also been studied in the context of MLLMs by some recent work (which can be considered to be contemporary). IMO, this is a concern.
(ii) focus on Indic languages: there are some concerns raised about the broader applicability of the techniques presented in the paper (personally, I disagree with this concern as Indic languages are important - for example, there are numerous papers which only report results on En-De, En-Ru translation)
(iii) limited evaluation: the technique is evaluated only using the ALBERT model and other configurations (such as ROBERTA, XLM, etc) are not considered. IMO, it would have helped if the authors presented results on these models also (at least we would know if transliteration only helps in the case of small/compact models or even in the case of large models)
(iv) missing references: there is a large body of related work on NMT, speech, etch which the authors had missed in their initial draft. This has been rectified in the updated version.

The reviewers did participate in the discussion with the meta-reviewer (not with the authors though) and even after looking at the revised draft mentioned that the novelty is limited. 

To summarise my views, I think the initial draft of the paper did need improvements and the final draft is a significantly improved version of the initial draft. However, I still feel the novelty is missing. Even the empirical novelty claimed by the authors is ;lacking due to the use of a single model (ALBERT).
Decision: Reject