ROMANSETU: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization (2024)

Abstract: This study addresses the challenge of extending Large Language Models (LLMs) to nonEnglish languages that use non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance crosslingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 (Touvron et al., 2023) on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP. Our code is available on https://github. com/AI4Bharat/romansetu.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15593–15615 August 11-16, 2024 ©2024 Association for Computational Linguistics
This study addresses the challenge of extending Large Language Models (LLMs) to nonEnglish languages that use non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance crosslingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 (Touvron et al., 2023) on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP. Our code is available on https://github. com/AI4Bharat/romansetu.

heading: 1 Introduction
text: Large Language Models (LLMs) demonstrate remarkable proficiency across a broad spectrum of natural language processing (NLP) tasks, as evidenced by various studies (Liu et al., 2023; Chung et al., 2022; Chowdhery et al., 2023; Wei et al., 2024; Goyal et al., 2022b). They excel not only
† Work done during internship at AI4Bharat ‡ Work done during employment at AI4Bharat § Corresponding Authors: Jaavid Aktar Husain (jaavidaktar@gmail.com), Raj Dabre (raj.dabre@nict.go.jp) and Anoop Kunchukuttan (ankunchu@microsoft.com)
in tasks for which they were explicitly trained but also in those for which they were not trained. This achievement is mainly due to the availability of corpora (Wenzek et al., 2020; Abadji et al., 2021; Suárez et al., 2019) as well as the advancements in LLMs that leverage these datasets for pretraining (Touvron et al., 2023; Workshop et al., 2022; Chowdhery et al., 2023). Despite their proficiency in English, these models typically demonstrate reduced effectiveness when applied to non-English languages, highlighting a significant challenge in extending their benefits to non-English languages.
The English-heavy LLMs (Touvron et al., 2023; Jiang et al., 2023; Zhang et al., 2022) still have some representation coverage from other languages due to data leakage while creating the pre-training dataset, particularly for languages that use the same script as English i.e., the Roman script. This script sharing enables cross-lingual transfer and bestows some of these LLM capabilities to these languages. For languages using non-Latin scripts, the data representation is very limited to non-existent. Tokenization of text in the languages exhibits high fertility (Ács, 2020) and byte-level representation (Artetxe et al., 2020) due to inadequate representation of these languages in the tokenizer vocabulary. Hence, these LLMs perform poorly on most of these languages, and inefficient tokenization also leads to high inference latency as well as the inability to process long sequences. This disparity in performance raises a critical question: how can we extend the capabilities of LLMs to the languages written in non-Latin scripts?
A widely explored solution is the extension of the tokenizer vocabulary to incorporate new languages and continual pre-training on native language data (Cui et al., 2023; Nguyen et al., 2023; Minixhofer et al., 2022). This approach is computationally demanding since the models need to
15593
be continually pre-trained long enough to effectively learn the new embeddings and align the representations of English and the new language. Furthermore, this approach requires the availability of large volumes of text corpora.
In this work, we explore an alternative approach for more efficient knowledge transfer. Instead of utilizing the native script, we use the romanized form of text as the interface to the LLMs in our approach RomanSetu.1 The adoption of romanized representation is justified for several reasons. In many languages, romanized text has been frequently used in informal settings and on social media in recent history. This usage creates the potential for the inclusion of some romanized data representation in the pre-training corpus. Additionally, code-mixing with English is a common occurrence and the romanized form shares tokens with English. This leads us to hypothesize that the romanized form is better aligned with English than the native script, thereby facilitating a more effective transfer from English.
In our approach, we first continually pre-train an English LLM like Llama 2 (Touvron et al., 2023) on romanized text for a language generated via transliteration. Subsequently, we perform instruction tuning on this continually pretrained model. We experiment with 5 languages from two language families and evaluate our approach on several NLU, NLG and MT benchmarks. Our experi-
1The word setu in Sanskrit means bridge, referring to the Roman script serving as a bridge between English and other languages.
ments and analysis indicate that:
• The fertility of romanized text is 2x-4x smaller than native text, making the romanized form far more efficient than the native script.
• The embeddings of the romanized text are closer to the embeddings of their English translations, compared to the embeddings of their native script equivalents, suggesting the former’s suitability for cross-lingual transfer.
• Results across several NLU, NLG, and MT tasks show that multilingual instruction finetuning on romanized data produces competitive or superior results than instruction finetuning on native script data, highlighting efficiency and better cross-lingual transfer. Figure 1 summarizes these results of various models on romanized and native inputs.
• Specifically, for generation tasks we see significant improvement due to utilization of romanization. To the best of our knowledge, ours is the first work that shows that romanization can help natural language generation tasks.
• Romanized representation can enable crosslingual transfer in decoder-only English-heavy LLMs. Previous work has primarily focused on the use of romanization in multilingual, encoder-only models.

heading: 2 Related Work
text: Transliteration refers to the conversion of text written in one script to another. Romanization is a specific instance of transliteration, where the target script is the Roman/Latin script. Romanization has special significance since Roman script is by far the most widely adopted script in the world and many language models are primarily trained for English, which is written in the Roman script. Transliteration is typically used to represent different languages in the same script to enable cross-lingual transfer.
In NMT, Amrhein and Sennrich (2020) show that transliteration shows improvements for lowresource languages with different scripts by transferring from related high-resource languages that use different scripts. Goyal et al. (2020) show that transliteration helps even when only a contact relationship exists between the languages involved. Song et al. (2020) show that transliteration during the pretraining stage for NMT also helps crosslingual transfer.
Transliteration has also been used for crosslingual transfer in the context of pretrained language models. Some works (Khemchandani et al., 2021; Dhamecha et al., 2021; Moosa et al., 2023; Purkayastha et al., 2023) employ transliteration to a common script during the pretraining phase to enable cross-lingual transfer. Other studies (Dabre et al., 2022; Muller et al., 2021; Chau and Smith, 2021) adopt transliteration during the fine-tuning. Transliteration could be done to a common nonLatin script (Khemchandani et al., 2021; Dhamecha et al., 2021; Dabre et al., 2022; Doddapaneni et al., 2023) or to the Latin script (Muller et al., 2021; Moosa et al., 2023; Purkayastha et al., 2023).
While transliteration has been explored for language modeling as described above, our work differs from previous work in some important aspects:
• Previous work explored cross-lingual transfer using transliteration with multilingual language models. We focus our attention on English LLMs and try to achieve cross-lingual transfer via romanization using English LLMs. This is a challenging scenario since the language can be unrelated to English, and very little (if any) native or romanized data in the language under study might be seen during pretraining of the English LLM. At the same time, this is a very practical need since most best
performing LLMs are English-heavy (Touvron et al., 2023; Jiang et al., 2023) and hence cross-lingual transfer via romanization is an important direction to explore.
• We investigate the utility of transliteration in decoder-only language models, which is currently the standard architecture for LLMs. In contrast, all previous work explored crosslingual transfer with transliteration in the context of encoder-only models (the exception is Dabre et al. (2022) which use encoder-decoder models).
• While previous work has explored the use of romanization for cross-lingual transfer in language understanding tasks, we show that romanization can be beneficial for generation tasks as well.
To the best of our knowledge, no prior research has investigated leveraging of romanization for mostly English LLMs in the context of crosslingual transfer to non-English languages.

heading: 3 Utilizing Romanized Data to make LLMs Multilingual
text: Our proposed approach aims to enhance the capabilities of mostly English LLMs for non-English languages by leveraging romanized data. We first continually pretrain the LLM with both romanized and English data to create a base LM that is romanization-aware. Subsequently, we extend the process by instruction tuning the continually pretrained model. The framework of our approach is illustrated in Figure 2, encompassing the stages of data romanization, pretraining, and instruction tuning.

heading: 3.1 Romanization Scheme
text: A variety of romanization schemes are available, each driven by multiple considerations. One key factor is the resemblance of the romanized representation to the way people typically write romanized text. This is particularly advantageous if the pretraining data includes romanized text, as it might aid in aligning with English. Another important aspect is the fertility achieved by the romanization scheme when considering the original LLM’s tokenizer. A third consideration is whether the transliteration scheme is lossy or lossless. A lossless scheme is preferable when the objective is to convert the output back to the native script. Typically, deterministic transliteration schemes are lossless, whereas natural transliteration schemes tend to be lossy.
In this work, we have focused on Indic languages. Hence, we evaluated two romanization schemes for Indic languages: (a) the extended ITRANS scheme from the IndicNLP library (Kunchukuttan, 2020), which defines a fixed, reversible mapping between Indic scripts and Roman characters, and (b) the IndicXlit scheme (Madhani et al., 2023), which generates romanizations as commonly used by Hindi speakers in informal contexts and is learned from parallel transliteration corpora. These mappings are inherently lossy. The IndicXlit romanization demonstrates lower fertility compared to ITRANS romanization (See Table 1). Preliminary prompting experiments with the Llama 2 model also indicated that IndicXlit outperforms ITRANS in romanized Hindi to English translation. Consequently, we selected IndicXlit as our romanization scheme for this project. However, it is important to note that the transliterations are not reversible. With continued pretraining, ITRANS transliterations might eventually achieve similar task performance to IndicXlit while ensuring script reversibility. This direction might also be relevant for other languages which do not have a transliteration model available, while romanization mappings are available for almost all scripts. We leave this exploration for future work.

heading: 3.2 Continual Pretraining
text: While the base model is trained on roman script, most of the data it has been exposed to is in English. Hence, the base model cannot be used as-is for processing romanized inputs, particularly for generation into non-English languages. Hence, the base model needs to be updated. To make the base
model romanization-aware, we continue to pretrain the base LLM with romanized document-level data. To prevent any catastrophic forgetting of English capabilities, we also incorporate an equal amount of English data into the pretraining mix. We perform full-finetuning of the models. We do not expand or change the model vocabulary, hence the romanized inputs can take advantage of the token embeddings from the base model while adapting to the new languages.

heading: 3.3 Instruction Finetuning (IFT)
text: Non-English languages possess very limited finetuning data. A common method to overcome this limitation is to translate English IFT datasets into the languages. We could choose to translate both input and outputs (Wei et al., 2023), or translate only the inputs and ask a powerful, proprietary LLM like ChatGPT to generate translations in the native language (Li et al., 2023). We chose the former approach since the quality of LLM responses (even for the best commercial models) might not be of good quality (Ahuja et al., 2023), while high-quality open-source translation models (Costa-jussà et al., 2024; Gala et al., 2023) are available making translation of both inputs and outputs feasible at scale for many languages. The translated IFT datasets are then romanized using IndicXlit.

heading: 4 Experimental Settings
text: We conducted comprehensive experiments spanning multiple languages, benchmarks and models. This section elaborates on the experimental settings. In all our experiments, we experiment with Llama 2 7B (Touvron et al., 2023) as the base model under various settings.

heading: 4.1 Languages
text: We experimented with the following Indic languages: Hindi, Marathi, Gujarati, Tamil and Malayalam. These languages span 2 language families and 4 different scripts and are mid to low-resource languages. The languages considered in this study belong to the Indo-Aryan branch of Indo-European (Hindi, Marathi, Gujarati) and Dravidian (Tamil, Malayalam) families. In these languages, the typical word order is SOV. These languages have an inflectional morphology, with the Indo-Aryan leaning somewhat towards fusional, while Dravidian languages tend to have an agglutinative morphology. It is important to note that the script systems used for these languages are abugida, effectively
faithfully capturing vowels and consonants. We see the choice of Indian languages as a strength since it spans multiple language families, multiple scripts, and diverse linguistic characteristics and resource levels. Past work has also used Indic languages as a case study for romanized representation (Moosa et al., 2023; Dhamecha et al., 2021).

heading: 4.2 Datasets
text: Continual Pretraining: For continual pretraining, we sourced approximately 500 million words of document-level data from web-crawled corpora (Doddapaneni et al., 2023) for each language under consideration along with English. To generate the romanized dataset, we transliterated the native script dataset using the IndicXlit model (Madhani et al., 2023), a state-of-the-art open-source transliteration model for Indian languages. Both the native script dataset and its romanized counterpart were then used for continual pretraining in various configurations explained later. Instruction Fine-tuning: These languages have very little native instruction tuning data for diverse tasks. Following Wei et al. (2023), we rely on translating high-quality English-supervised instructiontuning datasets into the languages under consideration. We use IndicTrans2 (Gala et al., 2023), the state-of-the-art open-source MT model for Indian languages compared to commercial offerings for translation. We sampled examples from various English instruction tuning datasets to ensure a diverse mix of tasks from the Flan collection (Longpre et al., 2023) (65k) and Dolly (Conover et al., 2023) (15k). These are translated into all the languages. The instruction tuning dataset includes a translation subset as well to learn the translation task as well as to drive alignment between English and other languages. We sample 20k high-quality manually translated examples in each direction from the BPCC-Human subset (Gala et al., 2023). Thus, the final IFT dataset has 120k examples per language. The instruction tuning datasets are further romanized using IndicXlit. Evaluation Data: We evaluate our models on a variety of NLU and NLG tasks. These include native language test sets from the IndicXTREME (Doddapaneni et al., 2023) and Indic NLG Suite (Kumar et al., 2022) benchmarks. For the translation task, we use the FLORES-200 devtest (Goyal et al., 2022a; Costa-jussà et al., 2024) for evaluation. Following recent work (Lai et al., 2023; Üstün et al., 2024; OpenAI et al., 2023), we eval-
uate the knowledge and reasoning capabilities of the model on the machine-translated versions of the benchmarks such as MMLU (Hendrycks et al., 2021a), ARC (Clark et al., 2018a), BoolQ (Clark et al., 2019a) and CommonSenseQA (Talmor et al., 2019). Note that as the CommonSenseQA benchmark comprises of a blind test set, we report the evaluations on the dev set. The translations were obtained using IndicTrans2 and the romanized inputs were created using IndicXlit.

heading: 4.3 Training and Finetuning Details
text: The models are fully finetuned during continual pretraining as well as supervised fine-tuning. Both the continued pretraining and instruction fine-tuning are multilingual i.e. a single model is trained jointly with data for all languages (including English to preserve English performance). Note that no vocabulary expansion is done. For pre-training, the loss is computed for all the tokens in the sequence. During instruction fine-tuning, the loss was computed only for the output tokens. Detailed information about the hyperparameters used for both continual pretraining and instruction fine-tuning can be found in Appendix A. We adapt open-instruct2 (Wang et al., 2023) for our continual pretraining and fine-tuning experiments.

heading: 4.4 Trained Models
text: We begin with Llama 2 as the base model. Then, we create two variants that have undergone continual pre-training: one with native script data (CPT-N) and the other with romanized data (CPT-R). For each of these continually pre-trained models, we further perform instruction fine-tuning (IFT). We fine-tune CPT-N using native script IFT data to obtain IFT-N, and CPT-R is fine-tuned with romanized script IFT data to produce IFT-R.

heading: 4.5 Evaluation Metrics
text: The evaluation metric for machine translation in our experiments is chrF (Papineni et al., 2002) computed using the SacreBLEU toolkit (Post, 2018).3 For other NLG tasks, we report Rouge-L (Lin, 2004) scores. We primarily report evaluations on the native script outputs. In the case of romanized outputs, we transliterate them back to native script
2https://github.com/allenai/open-instruct/ tree/main
3nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no| version:2.3.1
using IndicXlit combined with a unigram wordlevel language model (Madhani et al., 2023). Combining word-level results with a unigram language model is efficient and preserves most gains of incorporating a complete n-gram language model (Kirov et al., 2024). In this native script evaluation, the roman script model outputs can also have transliteration errors, which put them at a disadvantage compared to the native script model. To purely compare task performance, we also report results on romanized outputs. For this, we romanize the native script outputs using IndicXlit. Since the final output is an artifact of the same romanization model, this evaluation compares only the task performance.

heading: 4.6 Models
text: We use the 7B-parameter Llama 2 base model as our starting point for all experiments. We evaluate the following models: First, the base Llama 2 model (base) is assessed using both native script and romanized inputs. We then evaluate the continually pre-trained models with their respective script data; thus, CPT-N is evaluated with native script data, and CPT-R is evaluated with romanized data. Additionally, the instruction-tuned models are evaluated with their respective script data; therefore, IFT-N is assessed with native script data, and IFT-R is assessed with romanized data.

heading: 4.7 Prompting and Decoding
text: For the base and CPT models, we do 1-shot and 3-shot prompting. For IFT models, we experiment with 0-shot prompting. The prompt templates for various tasks are shown in Appendix B. For generation tasks, we use greedy decoding. For classification tasks, we completed the prompts with various possible options as per class labels and computed the output probabilities. We choose the class that maximizes the probability (Liu et al., 2021).

heading: 5 Results
text: In this section, we present the results of various experiments comparing romanized and native representations and discuss the results. We aim to address the following research questions:
• Is there an efficiency gain through romanization, and if so, which approach to romanization is the most efficient?
• Do romanized inputs lead to embeddings that
are better aligned to English representations compared to native inputs?
• Does romanization improve downstream task performance across multiple NLP tasks?
Efficiency Gains with Romanization: To address this question, we compute fertility scores on the FLORES-200 dev set (Goyal et al., 2022a; Costa-jussà et al., 2024). Fertility refers to the number of tokens per word generated by the Llama 2 tokenizer (Touvron et al., 2023). We compare the native script against two forms of romanized text: transliteration using IndicXlit (Madhani et al., 2023) and rule-based extended ITRANS transliteration scheme implemented in the IndicNLP library (Kunchukuttan, 2020). Table 1 presents the fertility scores on the Llama 2 model for the languages we experimented with. We observed that the fertility (and hence sequence length) in the native script is at least twice that of the IndicXlit romanized text. In languages like Gujarati, the native script fertility is very large (greater than 4 times the romanized fertility). Hence, we can expect significant efficiency gains when processing romanized text over native text in terms of reduced memory consumption, reduced generation time and increased maximum sequence length limit.
Furthermore, romanization using IndicNLP (Kunchukuttan, 2020) has slightly higher fertility compared to IndicXlit. While we have used IndicXlit-based transliterations in this work for reasons explained earlier, it would be interesting to explore deterministic and reversible romanizations like IndicNLP at the cost of slightly increased inefficiency. The advantage of reversible mapping would be avoiding transliteration errors when mapping the outputs back to the native script.
Cross-lingual Representation Similarity: We compare the sentence embeddings generated by the
native and romanized inputs as well as English on the base model. Specifically, we compare cosine similarities between sentence embeddings generated by these inputs. Sentence embeddings are computed using the last token representation of the final hidden layer. In Table 2, we see that the Roman script representations are closer to English representations compared to the native script representations, making them a better candidate for cross-lingual transfer from English.
Machine Translation: We discuss the results of machine translation, a canonical multilingual language generation task. Table 3 (Rows XX-En translation and En-XX translation) shows results on machine translation both in and out of English.
For translation into English, the base model translation quality is roughly equivalent for both native and roman scripts. We see that continual pretraining helps improve translation quality significantly for both representations. Romanized scripts show significantly more improvement on the CPT models compared to native scripts. We hypothesize that romanization is better able to align crosslingual representations between English and other languages to improve translation performance. Romanized IFT models also perform better than their native counterparts, albeit the gap is now narrower compared to CPT models.
For translation out of English, the base model performs very poorly which is understandable since the model might have seen only negligible nonEnglish data. CPT significantly improves translation quality on both representations, with romanized representation showing better performance. The IFT-R models are able to take good advantage of the romanized representations, and they show an average improvement of 8 chrF points over the
Language Native-Script Eval Roman-Script Eval
corresponding native script IFT-N models. Thus, we show that even generation into romanized script can take advantage of the English capabilities of the LLM.
Table 4 shows the effect of the the choice of script when evaluating translation out of English to each language. We also report evaluation on Roman script outputs for both models and see that the romanized IFT models report even larger gains in this evaluation setting. This suggests that the true gains from romanization might be larger and it is not reflected in native script evaluation on account of errors in Roman to native transliteration that is performed for the evaluation.
Language Generation Tasks: Table 3 shows the results on other generation tasks viz. summarization and headline generation. Similar to MT, we observe that models utilizing romanized script consistently outperform those using native script by ~4 RougeL points across both continually pretraining and instruction fine-tuning setups with different shots. On summarization using the XLSum dataset (Hasan et al., 2021), we find that CPT and IFT models demonstrate competitive performance in
both native and romanized scripts. A bit surprisingly, we find that IFT models underperform by 3-5 RougeL points compared to CPT models on IndicHeadline. We hypothesize that the format of the example was not seen in the IFT dataset, resulting in poorer zero-shot performance compared to the one-shot performance of the CPT model. Overall, we find romanization to benefit not only in terms of efficiency but also in terms of downstream performance.
Language Understanding Tasks: Table 5 shows the results on different NLU tasks part of IndicXTREME benchmark (Doddapaneni et al., 2023), including IndicSentiment, IndicCOPA, and IndicXNLI. We find that romanized script models are competitive to those that utilize native script.
Specifically, in tasks such as IndicSentiment and IndicXNLI, romanized models consistently outperform their native counterparts across various shots. However, native models generally perform superior compared to romanized variants for the IndicCOPA task, except for the IFT model. Furthermore, we observe notable performance improvements for IndicQA without context (a reading comprehension task) across all models with romanized script when we provide the relevant passage in the context. This indicates that models with the romanized script tend to benefit from knowledge transfer from English and are also efficient in terms of sequence length.
Knowledge and Reasoning tasks: Table 6 shows the results on translated versions of various knowledge and reasoning benchmarks such as MMLU (Hendrycks et al., 2021b), BoolQ (Clark et al., 2019b), ARC (Clark et al., 2018b) and CommonsenseQA (Talmor et al., 2019). For most of the tasks, the results of the romanized script are competitive with the native script. At the same time, they are efficient to process, as discussed earlier. For IndicQA without context (open domain question answering task), we observe that the romanized model shows ~10 F1 point gain over the native script model. This demonstrates the overall effectiveness of utilizing romanization for non-Latin script languages.
Discussion: From the analysis and results presented earlier, we can summarize the following. Romanized representations are significantly more efficient compared to native script representations. They are better aligned to English representations compared to native script representations, hence they are better for cross-lingual transfer. Across a range of tasks and languages, romanized representations provide competitive or better performance over their native script counterparts (detailed results per language and task are presented in Appendix D. Specifically, for a generation task like machine translation, we see significant gains while translating out of English with romanized representations. In Appendix C, we show some examples of romanized and native script translations, illustrating the gains from romanization. These results indicate that romanization is a promising alternative to extending the capabilities of English LLMs to other languages using non-Roman scripts.
We primarily base our study on English-only models due to their state-of-the-art performance
across a diverse array of tasks and consistent improvements in capabilities. In contrast, multilingual models like BLOOM significantly lag behind English-only models, even on English tasks, and their progress has been slower. This disparity is due to the limited amount of English data and the lack of fine-tuning data for non-English languages used in training multilingual models. Therefore, improving English-only models is a practical way to enhance multilingual LLMs. We fine-tuned the BLOOM 7B model with the instruction dataset in both native and roman script. We observe that the roman script model outperforms the native script model, consistent with the findings on the Llama 2 base model. We provide the evaluation results for BLOOM 7B model across different tasks in Appendix E.

heading: 6 Conclusion
text: In this study, we proposed the use of romanization to extend the performance of LLMs primarily trained in English to other languages. Our approach successfully unlocks LLM capabilities for non-English languages by using romanization to bridge English and non-English language representations. We have empirically demonstrated the effectiveness of this strategy through experiments involving few-shot prompting, continual pretraining, and instruction fine-tuning on a variety of tasks. Additionally, using romanized data increases inference speed and maximum processable sequence length, while reducing memory requirements by 2x to 4x depending on the language.

heading: 7 Future Work
text: Looking forward, we also plan to explore reversible, deterministic transliteration so that transliteration errors in output post-processing are eliminated. In addition, it would be interesting to explore if romanized representation can act as a bridge to efficiently improve native script performance also when English LLMs are extended to incorporate non-Roman tokens.

heading: 8 Limitations
text: In this work, we chose to use natural romanizations since they tended to have lower fertility and might have been seen in the base LM pretraining. Transliterating the outputs back to the native script can be lossy. We use a high-quality transliteration
model to minimize the transliteration errors. However, such high-quality transliterations might not be available for all languages. Using a deterministic and reversible transliteration scheme can alleviate this problem.
The experiments in this work have been conducted on Indian languages. Indian languages are representative of the scenarios we study and we cover multiple language families and scripts with varying resource status. Previous studies have also used Indic languages as a case study for multilinguality and multi-script scenarios (Moosa et al., 2023; Khemchandani et al., 2021). Hence, we believe the findings should generalize to other language families - but further experimentation can confirm this hypothesis.
We have done only a limited amount of pretraining data in our experiments due to compute constraints. Resource constraints led us to limit our experiments to a 7B model, but experiments with larger models can yield more insights. Larger models and pre-training on larger datasets might benefit the multilingual models (Kaplan et al., 2020; Fernandes et al., 2023). We expect the broad conclusions to be the same - that romanized models are more efficient and better/comparable in task performance to native-script models.
Some of the evaluations have been done on translated benchmarks. While this is not ideal, most low-resource languages lack testsets across diverse tasks. Hence, multiple works have relied on translated benchmarks to measure multilingual performance - these can give indicative trends on performance across languages. More efforts are needed to create multilingual benchmarks for diverse tasks.

heading: 9 Ethics Statement
text: We use romanization as a way to align English with other languages using non-Latin scripts. Given the current state of LLMs, this seems like a practical direction to extend the capabilities of the best LLMs to other languages. The intention is not to supplant the use of a native script (which is widely adopted and has a rich literary tradition) with romanized script but to use romanization as a way to efficiently bring the benefits of LLM technology to low-resource languages written in non-Latin scripts. Further advancements are needed to extend this line of research to improve native script performance efficiently.
This work does not involve any new data collec-
tion and does not employ any annotators for data collection. We utilize publicly available datasets for the experiments reported in this work. Some of these datasets originate from web crawls, and we do not explicitly attempt to identify any biases within these datasets, using them in their original form.

heading: Acknowledgments
text: We would like to thank EkStep Foundation and Nilekani Philanthropies for their generous grant towards research and building datasets, models, tools and other resources for Indian languages.

heading: A Model Training Details
text: Tables 7 and 8 show the hyperparameters used for training the CPT and IFT models.

heading: B Prompt Templates
text: We list the various prompt templates for evaluations across different test sets (see Figures 3 to 12).

heading: C Translation Examples
text: This section shows some examples of translations using the romanized and native-script IFT models (See Figures 13 to 16). We can see in these selected examples that romanized representation performs better.

heading: D Language Specific Results
text: We report the language-wise evaluation results for 5 Indic languages considered for this study across a diverse array of tasks ranging from text generation, text understanding, and text reasoning in Tables 9 to 22.

heading: E Results on BLOOM 7B
text: We report evaluation results for 5 Indic languages for the tasks such as machine translation, headline generation and summarization on instruction fine-tuned BLOOM 7B in Tables 23 to 25.
4Although this is the maximum permissible length, most examples fall far below this length during fine-tuning.

heading: I am hesitating between two options. Help me choose the more likely cause or effect.

heading: Future Work
text: We leave this exploration for future work