Venue: NAACL 2025

# Efficient Continual Pre-training of LLMs for Low-resource Languages

Arijit Nag

IIT Kharagpur

arijitanag@iitkgp.ac.in

Animesh Mukherjee

IIT Kharagpur

animeshm@cse.iitkgp.ac.in

Niloy Ganguly

IIT Kharagpur

niloy@cse.iitkgp.ac.in

Soumen Chakrabarti

IIT Bombay

soumen@cse.iitb.ac.in

###### Abstract

Open-source large language models (OsLLMs) propel the democratization of natural language research by giving the flexibility to augment or update model parameters for performance improvement. Nevertheless, like proprietary LLMs, Os-LLMs offer poorer performance on low-resource languages (LRLs) than high-resource languages (HRLs), owing to smaller amounts of training data and underrepresented vocabulary. On the other hand, continual pre-training (CPT) with large amounts of language-specific data is a costly proposition in terms of data acquisition and computational resources. Our goal is to drastically reduce CPT cost. To that end, we first develop a new algorithm to select a subset of texts from a larger corpus. We show the effectiveness of our technique using very little CPT data. In search of further improvement, we design a new algorithm to select tokens to include in the LLM vocabulary. We experiment with the recent Llama-3 model and nine Indian languages with diverse scripts and extent of resource availability. For evaluation, we use IndicGenBench, a generation task benchmark dataset for Indic languages. We experiment with various CPT corpora and augmented vocabulary size and offer insights across language families.

## 1 Introduction

Large language models (LLMs) like GPT-4 [20], ChatGPT, Llama-2 [75], Llama-3 [86], PaLM [9], _inter alia_, are opening up new possibilities for low-resource languages (LRLs). Until recently, collecting sufficient labeled LRL data to finetune LLMs for classification and generation tasks used to be challenging. Today, LLMs give decent performance with zero/few-shot inference. Having said that, there is still a substantial performance gap between high-resource languages (HRLs) and URLs for LLMs [11, 12, 13]. This is since LRLs like Indic languages are still under-represented by recent LLMs, as shown in Figure 1: Compared to English, the average number of tokens required to generate a LRL word by these LLMs is substantially higher. The inability to represent a word with a single token may lead to suboptimal learning of context thus potentially affecting LLM's performance for LRL tasks. A feasible way to overcome such shortcoming is to initiate continual pre-training (CPT), specifically with LRL text.

CPT can help LLMs learn domains/languages that are un/under-explored in the pre-training stage. While this is a viable option to improve LLM's performance, training such gigantic models consumes expensive GPU resources and time, which makes it less feasible in resource-constrained setups. To address these issues and harness CPT's potential, we propose a two-pronged approach. First, we introduce a score-based method to select a small set of high-quality, language-specific training data. Concurrently, we implement a strategy to expand the token vocabulary in LLMs. This vocabulary augmentation improves the understanding of important words in low-resource languages, leading to further performance gains. The strategies and the rigorous experiments undertaken are detailed next.

Figure 1: Average tokens generated per word for various Indic languages using different recent LLMs. The last column shows the performance in English.





**(1)**: We propose a global \(+\) local score for each sentence for selecting a small subset of text from an LRL training corpus to perform CPT and improve LLM performance. Experiments show significant performance boost.
**(2)**: We propose a method to augment the token vocabulary of the LLM to further improve URL task performance in certain situations.
**Experiments** We present a comprehensive study on the recently released and very popular white-box LLM LLama-3-8B, applying our CPT methods to nine Indian languages in six scripts, covering three resource levels [45] (Singh et al., 2024) over five LRL generation tasks provided by IndicGenBench [69], including summarization, machine translation, and question-answering.
**Observations** Single/limited word prediction tasks like QA are more sensitive to vocabulary augmentation compared to multi-word generation tasks like summarization or machine translation; the effect of vocabulary augmentation on tokenization varies across scripts; and larger CPT corpus and vocabulary do not always convert to performance improvements.

## 2 Related work

Language models use diverse subword tokenizer algorithms like Byte-Pair Encoding (BPE) [73], Sentence-Piece [101], Word-Piece [101], and Unigram [102]. Due to the limited size of an LLM's token vocabulary, over-fragmentation [103, 104, 105] is a common problem, especially for multilingual models where not all languages get equal representation. Apart from task performance degradation [103, 105, 106, 107], over-fragmentation leads to slow inference [104, 105] and increased training and inference/generation cost [105, 106, 107]. Various mitigation methods have been proposed, including vocabulary expansion [104, 105, 106, 107, 108, 109] and replacing existing tokens in the vocabulary with new ones [105, 106, 107, 108, 109]. In recent work such as ChineseLlama [105] and TamilLlama [106], the authors add new language-specific tokens and then pretrain the model with large amounts of training data. More recently, Yamaguchi2024b and Tejaswi et al. (2024) explore CPT of LLMs while varying the corpus, additional vocabulary and embedding initialization techniques. However, they do not focus on strategies to select corpus and vocabulary.

In contrast, in this work, we propose a global \(+\) local joint rank-based system to first select the small-scale training corpus and then augment the LLM's vocabulary with additional language-specific tokens for CPT. With a small amount of informative training data and added vocabulary, we show substantial LLM performance improvement for Indic languages.

## 3 Proposed method

In this work, we design a two-stage approach to improve LLM's performance with reduced resource requirements. In the first stage, we select a subset of the available LRL corpus, and in the next stage, we select prospective new tokens for vocabulary augmentation. These two together are used for the purpose of CPT as shown in Figure 2. As the figure shows, the second stage (token selection) is optional. Section 3.1 and 3.2 describe the corpus and vocabulary selection algorithms, respectively.

Figure 2: Sketch of proposed LLM CPT method. Dotted lines indicate optional steps.





### Stage I: Sentence selection

In this stage, the goal is to identify a subset of sentences from LRL corpus \(C_{l}\) that will effectively enhance the LLM as a representative of the whole of \(C_{l}\). We regard a sentence as a strong representative if it contains numerous 'important' words formed from popular subword tokens. These important words reflect the unique features of the corpus, while the popular tokens represent commonly used contexts.

**Popular subwords** In Algorithm 1, we first use the LLM's tokenizer to get all distinct subword tokens present in the corpus and compute their occurrence frequencies. Next, for a given word \(w\) we compute the sum of the frequencies of its subwords. We now subtract the frequency of \(w\) from this sum which indicates how much these tokens solely contribute to words other than \(w\). If this difference is high then it implies that the subwords of \(w\) contribute to many other words in the corpus and are thus more popular.

```
0: Large training corpus \(C_{l}\) Number of sentences to select \(K\)
0: LLM tokenizer \(\mathcal{T}\) Parameters for weighted average \(\alpha,\beta\)
1:\(\mathcal{W}\leftarrow\text{vocabulary from corpus $C_{l}$}\)
2:fill \(WC\) (word count dictionary) using \(\mathcal{W},C_{l}\)
3:\(SWC\leftarrow\left\{\right\}\) /* subword count dictionary */
4:for\(w\in\mathcal{W}\)do
5:for subword tokens \(t\in\mathcal{T}(w)\)do
6:\(SWC[t]\mathrel{+=}WC[w]\)
7:for each word \(w\) do
8: initialize \(N[w]\gets 0\)
9:/* \(N[w]\) will store aggregated popularity of subwords of we relative to itself */
10:for\(w\in\mathcal{W}\)do
11:for\(t\in\mathcal{T}(w)\)do
12:\(N[w]\leftarrow\)\(SWC[t]-WC[w]\)
13:fill \(X_{co}\) with word-word co-occurrence matrix from \(C_{l}\)
14:/* co-occurrence within a context window */
15:\(W_{g}\leftarrow\) PageRank\((X_{co})\)
16:\(w^{\prime}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!

ting both for off-the-shelf vanilla LLM and after doing the CPT over it. Details of LLM parameters and prompts are in the Appendix B (see Table 9 and Figure 5, respectively). For evaluation, we use Character-F1 (Chr\(++\)[47]) for Summarization and MT tasks and Token-F1 for QA tasks. For all languages, we sample the CPT corpus from the IndicCorpV2 dataset Doddapaneni et al. (2022) and to restrict the cost of experiments, we limit the CPT corpus size to 10K, 20K and 30K and, similarly, the augmented vocabulary size to 100, 200 and 300.

### CPT corpus helps despite small size

In Table 1, we show the effect of CPT of the vanilla LLM with the small-sized ranked corpus that we obtain using Algorithm 1. We experiment with 10K, 20K and 30K top-ranked sentences as CPT corpus and report the best among them (denoted as **TR(Best)**). As the average results (please refer to the Appendix Table 8) are similar to the best result, here we only report the best performance result. We use off-the-shelf vanilla Llama-3-8B model's performance as our baseline. We also report the change in performance (%) from vanilla to TR(Best) for individual languages as well as resource type availability. In general, we observe significant performance improvements for most of the tasks and languages. The improvements are progressively higher from the high-resource language group to the low-resource language group. This observation is expected as the vanilla LLMs are already well-trained in high-resource languages and may not get much benefit from CPT as compared to the resource-poor languages. Further for the QA tasks, both when the

\begin{table}
\begin{tabular}{l|c|c|c c c|c c} \hline  & & \multicolumn{3}{c|}{**Target(x)**} & \multicolumn{1}{c}{**Tagger(x)**} \\ \cline{6-8}  & & & **Metric\(\rightarrow\)** & **Curf+at** & **Cluff+at** & **Token-F1** & **Charreira–t** & **Toko-F1** \\ \multirow{2}{*}{**Lma**} & **Script** & **Type** & **CT data** & **Crossion Fluers++** & **XorQA** & **Hybrider(xx 

target language is Indic and English, we observe limited improvement for most of the cases and especially for English target (XorQA(en)) performance drops after CPT. This can be due to catastrophic forgetting of the English part as we do the CPT with Indic language-specific data and also as QA tasks performed here are limited word (1-2 words) prediction tasks, making it more vulnerable to such problems. In Section 4.5, we discuss a solution for them.

### Sentence scoring and ranking help

To study the effect of corpus ranking we compare TR(Best) with BR(Best). We form 10K, 20K, 30K subsets with the least scoring sentences from the corpus, perform CPT and report the best performance among them as **BR(Best)**. In Table 1, we report the change in performance (%) from TR(Best) to BR(Best) for individual languages as well as based on resource type availability. We observe that TR(Best) outperforms BR(Best) across all tasks and languages except the QA tasks, showing the effectiveness of the ranking algorithm. It might be possible that top-ranked sentences lack diversity and may constrain the output token distribution. As QA tasks are sensitive to single-word prediction, it can affect performance adversely.

### Vocabulary augmentation helps in specific cases

In previous sections, we observed CPT with a small corpus improves LLM performance for most tasks and languages. To check if the performance can be improved further, we attempt vocabulary augmentation. Our hypothesis is that vocabulary augmentation would typically work for those languages where fragment ratio (average number of tokens generated per word) is high. We find the fragment ratio of the nine languages (Table 7) and group them into large, medium and small. We compare LLM performance with and without vocabulary augmentation while running CPT with **TR(Best)** and report the average improvement in Table 2. We experiment with addition of 100, 200 and 300 tokens and report the best result. We see vocabulary augmentation helps multi-word generation tasks like CrossSum and Flores(en\(\rightarrow\)xx), when the fragmentation ratio is medium to large. At lower levels of fragment ratios, we do not see benefits from vocabulary augmentation. In case of XorQA(xx), we see performance _drop_ after vocabulary augmentation, for languages with a high fragment ratio. Poor initialization of the newly augmented words, followed by limited training, may hamper their single-word prediction abilities. We also discuss few error cases of XorQA(xx) in Appendix A.1.

For Flores(xx\(\rightarrow\)en) and XorQA(en), where

\begin{table}
\begin{tabular}{l|c c|c|c c c c}  & & & & & & \\  & & & & & & & \\ \hline \multirow{6}{*}{**Sankati**} & \multirow{2}{*}{**Occhial**} & \multirow{2}{*}{**Target**} & **CTR(Bost)** & **Caffe:** & **Carity:** & **Tolorsa:** & **Charity:** & **Clark:** & **Catto=P1** \\ \cline{3-8}  & & & **Videos:** & **Decision:** & **Decision:** & **Net:** & **Oda:** & **Chrover:** & **Year:** \\ \hline \multirow{3}{*}{**Tetogen**} & \multirow{3}{*}{**Tokgen**} & \multirow{3}{*}{**Topage**} & **TF(Bost)** & 98.90 & 12.23 & 0.67 & 20.71 & 0.41 \\  & & & Yes & 13.97 & 13.99 & 0.26 & 1.14 & 0.32 \\  & & & & (dErr)/12.194 & 14.305+ & 0.419+ & 29.995+ & 22.195+ \\  & & & & No.163 & 25.357 & 0.37 & 39.31 & 0.59 \\  & & & yes & 18.13 & 26.03 & 0.36 & 41.59 & 0.61 \\  & & & dErr)/9.875 & 13.895 & -2.709 & 10.899 & 3.399+ \\ \hline \multirow{3}{*}{**Asamese**} & \multirow{3}{*}{**Bengli**} & \multirow{3}{*}{**Medium**} & \multicolumn{3}{c}{Age Log. **85.15.63.

 

the target language is English, we do not see any improvement from vocabulary augmentation. This may be because we are adding Indic language-specific vocabulary and training with that language-specific corpus, giving no or negative improvement for English target tasks (we discuss it in Section 4.5). Another interesting observation is that with vocabulary augmentation, the LLM can generate more tokens than vanilla or without vocabulary-augmented LLM, given a similar output generation limit (more details on Appendix A.3).

### Additional corpus and tokens not always helpful

To check if CPT with a larger corpus size and an order of magnitude large vocabulary size results in even better performance, we conduct CPT with 100K ranked corpus and 2000 additional vocabulary and compare it with 30K ranked corpus and 300 additional vocabulary. In Table 3, we report the result of these two configurations and find that a large CPT corpus with more additional vocabulary does not improve the performance as compared to a small-size corpus and vocabulary augmentation. This can be due to two reasons, first, as we are ranking the corpus, it might be possible most informative sentences are already present in the smaller 30K corpus. Second, as we are doing cost-efficient CPT by using LoRA and limited training steps (2 epochs), a large corpus with more additional vocabulary finds it difficult to converge, resulting in sub-optimal performance.

### Adding English corpus to CPT improves English generation

In Table 2, we see LLM's performance drops for English target generation tasks like Flores(xx\(\rightarrow\)en) and XorQA(en) after CPT using additional vocabulary. We hypothesize that this can be due to catastrophic forgetting as English corpus is not used while doing CPT. To verify this we add 20K randomly selected English sentence corpus with existing 30K Indic language-specific ranked corpus for CPT. In Table 4, we compare the LLM's performance after doing CPT with and without 20K English sentence corpus. We see that in almost all the cases, performance improves or remains the same

\begin{table}
\begin{tabular}{l|c c|c c} \hline  & \multicolumn{3}{c}{Metric+} & **CErr+** & **Token-PT** \\ \cline{2-6}
**Lang** & **CPT data** & **Voxoid** & **Fluers(x=-en)** & **XorQA(en)** \\ \hline
**Urbu** & **30K** & Yes & 40.52 & 0.56 \\  & & 230K(Lk) & Yes & **40.72** & **0.63** \\ \hline
**Bengali** & **30K** & Yes & **49.08** & **0.61** \\  & & 240K(Lk) & Yes & 40.55 & 0.56 \\ \hline
**Telbing** & **30K** & Yes & 41.38 & 0.61 \\  & & 240K(Lk) & Yes & 43.91 & 0.61 \\ \hline
**Sankrit** & **30K** & Yes & 26.39 & **0.47** \\  & & 250K(Lk) & Yes & 28.69 & 0.57 \\ \hline
**Assumence** & **30K** & Yes & 38.49 & **0.61** \\  & & 240K(Lk) & Yes & 38.52 & 0.58 \\ \hline
**Panajial** & **30K** & Yes & 41.73 & 0.59 \\  & & 240K(Lk) & Yes & 43.58 & 0.59 \\ \hline
**Sankrit** & **30K** & Yes & 14.54 & 0.32 \\  & & 240K(Lk) & Yes & **17.93** & **0.39** \\ \hline
**Nonlinear** & **30K** & Yes & **30.36** & 0.31 \\  & & 240K(Lk) & Yes & 25.49 & **0.45** \\ \hline
**Book** & **30K** & Yes & 30.35 & 0.49 \\  & & 240K(Lk) & Yes & **32.08** & **0.61** \\ \hline \end{tabular}
\end{table}
Table 4: Comparing LLM’s performance on English target generation tasks w/o and w/ additional 20K English corpus along with 30K ranked CPT corpus for various Indic languages. covering different scripts and resource types. Best performances are marked **bold** and underlined.

\begin{table}
\begin{tabular}{l l|c|c|c c c c} \hline  & \multirow{2}{*}{Length} & \multicolumn{3}{c}{Metric+} & **Carr+** & **Token-PT** \\ \cline{3-6}  & & **CPT data** & **Voxoid** & **Fluers(x=-en)** & **XorQA(en)** \\ \hline
**Urbu** & **30K** & Yes & 40.52 & 0.56 \\  & & 240K(Lk) & Yes & **40.72** & **0.63** \\ \hline
**Bengali** & **30K** & Yes & **40.45** & **0.61** \\  & & 240K(Lk) & Yes & 40.55 & 0.56 \\ \hline
**Telbing** & **30K** & Yes & 41.38 & 0.61 \\  & & 240K(Lk) & Yes & 43.91 & 0.61 \\ \hline
**Sanakrit** & **30K** & Yes & 26.39 & **0.47** \\  & & 250K(Lk) & Yes & 28.69 & 0.57 \\ \hline
**Assumence** & **30K** & Yes & 35.49 & **0.61** \\  & & 240K(Lk) & Yes & **35.52** & 0.58 \\ \hline
**Panjial** & **30K** & Yes & 41.73 & 0.59 \\  & & 240K(Lk) & Yes & 43.58 & 0.59 \\ \hline
**Sankrit** & **30K** & Yes & 14.54 & 0.32 \\  & & 240K(Lk) & Yes & **17.93** & **0.39** \\ \hline
**Konstant** & **30K** & Yes & **30.36** & 0.31 \\  & & 240K(Lk) & Yes & 25.49 & **0.45** \\ \hline
**Bode** & **30K** & Yes & 30.35 & 0.49 \\  & & 240K(Lk) & Yes & **32.08** & **0.61** \\ \hline \end{tabular}
\end{table}
Table 3: Comparing LLM’s performance after CPT with 30K corpus, 300 additional vocabulary with 100K corpus with 2000 additional vocabulary for various Indic languages covering different scripts and resource types. Positive average improvements are marked **bold** and underlined.

 

as compared to CPT with only language-specific corpus. This justifies adding English language-specific corpus before CPT.

### Effect of tokenization after vocabulary addition

Finally, we study the LLM's tokenizer capability before and after adding additional vocabulary. In general, extra vocabulary can improve tokenization and generate a lesser number of tokens, which helps to reduce generation costs. In Table 5, we show the average number of tokens generated by the LLM's tokenizer before and after adding additional vocabulary for all languages and tasks. In our case, we stick to adding single-character tokens whenever possible, as this can help to transfer the CPT benefit to downstream tasks. During CPT, if we add multi-character tokens, it might be possible that the downstream tasks may not have that token, resulting in not passing the training benefit to target tasks. We observe that additional vocabulary augmentation reduces the average tokens per word except for the languages using Devanagari and Arabic scripts. Flores(xx\(\rightarrow\)en) and XorQA(en) are unaffected by addition of only Indic language-specific tokens. In Figure 3, we show two examples of tokenization with and without additional vocabulary augmentation. As for Devanagari scripts, the word "gget"1849 splits into 6 tokens, whereas, after vocabulary addition, it splits into 9 tokens. This is due to the fact that vanilla LLM tokenizer already splits the word better than single character split, but when we add single character tokens as additional vocabulary, it worsens the tokenization. However, for Telugu scripts where the word "gget"360s splits into 14 tokens (single character splits into multiple bytes), single character token addition improves the tokenization by splitting it into 7 characters. _Summary of observations:_ Combining the observations from Tables 1, 2 and 5, we see all the languages benefit from CPT without vocabulary augmentation, though the degree of improvement is more for low resource languages. However, a similar pattern of improvement is absent when we augment additional vocabulary during CPT; here, we see improvement only if the language is overfragmented by the LLM's tokenizer, irrespective of their resource availability type. As an example, although the language Bodo is resource-poor, it has a lesser fragment ratio (Table 7) as it shares the resource-rich Devanagari script, failing to reap the benefit of vocabulary augmentation. On the other hand, Santali, both a resource-poor and overfragmented language (Table 7), get additional gain after vocabulary augmentation. So, our conclusion from this whole exercise is our method works best for a language which is poor in both terms, resources and script representation.

## 5 Conclusion

This work proposes a technique to select a compact CPT corpus and a method to augment the LLM vocabulary with a small set of new tokens. Experiments on IndicGenBench, covering nine Indian languages with diverse scripts and resources, show that a small CPT corpus improves performance, with additional gains possible through limited vocabulary augmentation. However, improvements vary by script, and excessive token addition or a larger CPT corpus may not always help. We also observed that language-specific CPT can negatively impact English generation. Our findings offer valuable insights for leveraging LLMs in LRLs.

Figure 3: Number of tokens (mentioned in numbers) generated before and after the vocabulary augmentation for Devanagari and Telugu scripts. Red and Green shades indicate an increase and decrease of tokens, respectively, after vocabulary addition. (The strange-looking characters are not typesetting aberrations.)





## 6 Limitations

Although recently, many white box LLMs like Llama families, Mistral, Phi, Gemma, etc., are available; we have only experimented with the LLama-3-8B model to work within our computation budget and carry out experiments with various languages and tasks. Though we stick to only one LLM for our research, including more LLMs in our study would be more insightful. To initialize the newly added word embedding, we use only the mean pooling method, which takes the average embedding of constituent tokens produced by the existing tokenizer. Although there are methods of embedding initialization like FOCUS, Merge, Align, Random, etc., we choose to mean as existing studies Yamaguchi et al. (2024); Tejaswi et al. (2024) show that it produces comparable results despite being simple. Having said that, considering other embedding techniques can make the study more comprehensive. Lastly, we restrict our experiment to only Indic languages and a few generation tasks; adding resource-poor languages from other language families and some more generation and classification tasks can strengthen our study further. We leave addressing these issues to future work.

## References

* Aha et al. (2023) Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Julia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. External Links: Link Cited by: SS1.
* B. Balachandran (2023)Tamil-llama: a new tamil language model based on llama 2. External Links: Link Cited by: SS1.
* Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, and P. Fung (2023)A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. External Links: Link Cited by: SS1.
* E. C. Chau, L. H. Lin, and N. A. Smith (2020)Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, pp. 1324-1334. External Links: Link Cited by: SS1.
* A. Chowdhery et al. (2022)Palm: scaling language modeling with pathways. External Links: Link Cited by: SS1.
* Y. Cui, Z. Yang, and X. Yao (2024)Efficient and effective text encoding for chinese llama and alpaca.. External Links: Link Cited by: SS1.
* K. Dobler and G. de Melo (2023)FOCS: effective embedding initialization for monolingual specialization of multilingual models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 13440-13454. External Links: Link Cited by: SS1.
* S. Doddapaneni, R. Aralikatte, G. Ramesh, S. Goyal, M. M. Khapra, A. Kunchukuttan, and P. Kumar (2022)Towards leaving no indie language behind: building monolingual corpora, benchmark and models for indic languages. ArXivabs/2212.05409. External Links: Link Cited by: SS1.
* C.D. Downey, T. Blevins, N. Goldfline, and S. Steinert-Threlkeld (2023)Embedding structure matters: comparing methods to adapt multilingual vocabularies to new languages. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), Singapore, pp. 268-281. External Links: Link Cited by: SS1.
* A. Dubey et al. (2024)The llama 3 herd of models. External Links: Link Cited by: SS1.
* K. Fujii, T. Nakamura, M. Loem, H. Iida, M. Ohi, K. Hattori, H. Shota, S. Mizuki, R. Yokota, and N. Okazaki (2024)Continal pre-training for cross-lingual llvm adaptation: enhancing japanese language capabilities. External Links: Link Cited by: SS1.
* T. Fujii, K. Shibata, A. Yamaguchi, T. Morishita, and Y. Sogawa (2023)How do different tokenizers perform on downstream tasks in scriptio continua languages?: a case study in Japanese. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Toronto, Canada, pp. 39-49. External Links: Link Cited by: SS1.
* L. Gee, A. Zugarini, L. Rigitini, and P. Torroni (2022)Fast vocabulary transfer for language model compression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, Abu Dhabi, UAE, pp. 409-416. External Links: Link Cited by: SS1.
* A. Hendy, M. Abdelrehim, A. Sharaf, V. Raunak, M. Gabr, H. Matsushita, Y. Jin Kim, M. Afify, and H. Assadalla (2023)How good are gpt models at machine translation? a comprehensive evaluation. External Links: Link Cited by: SS1.
* V. Hofmann, H. Schutze, and J. Pierrehumbert (2022)An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics

Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatqpt a good translator? yes with gpt-4 as the engine.
* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates.
* Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.
* Minixhofer et al. (2022) Benjamin Minixhofer, Fabian Paischer, and Navid Rekabasz. 2022. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3992-4006, Seattle, United States. Association for Computational Linguistics.
* Muller et al. (2021) Benjamin Muller, Antonios Anastasopoulos, Benoit Sagot, and Djame Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 448-462, Online. Association for Computational Linguistics.
* Nag et al. (2024) Arijit Nag, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. 2024. Cost-performance optimization for processing low-resource language tasks using commercial lms.
* OpenAI et al. (2023) OpenAI et al. 2023. Gpt-4 technical report.
* Ostendorff and Rehm (2023) Malte Ostendorff and Georg Rehm. 2023. Efficient language model training through cross-lingual and progressive transfer learning.
* Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking : Bringing order to the web. In _The Web Conference_.
* Petrov et al. (2023) Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bilbi. 2023. Language model tokenizers introduce unfairness between languages.
* Popovic (2017) Maja Popovic. 2017. chrF++: words helping character n-grams. In _Proceedings of the Second Conference on Machine Translation_, pages 612-618, Copenhagen, Denmark. Association for Computational Linguistics.
* Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3118-3135, Online. Association for Computational Linguistics.
* Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 5149-5152.
* Sennrich et al. (2016) Rico Sennrich, Barry Hadlow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units.
* Singh et al. (2024) Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dineshent Tewari, and Partha Talukdar. 2024. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages.
* Tejaswi et al. (2024) Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring design choices for building language-specific llms.
* Toraman et al. (2023) Cagri Toraman, Eyup Halit Yilmaz, Furkan Sahinuc, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkhish. _ACM Trans. Asian Low-Resour. Lang. Inf. Process._, 22(4).
* Touvron et al. (2023) Hugo Touvron et al. 2023. LLaMA 2: Open foundation and fine-tuned chat models.
* Yamaguchi et al. (2024a) Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024a. An empirical study on cross-lingual vocabulary adaptation for efficient language model inference.
* Yamaguchi et al. (2024b) Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024b. How can we effectively expand the vocabulary of llvms with 0.01gb of target language text? 



## Appendix A Supplementary results

### Error cases for QA tasks

We show a few error cases in Table 6, where the LLM fails after vocabulary augmentation for XorQA(xx). In one such case, the prediction is correct after vocabulary augmentation, but the evaluation metric flags it as incorrect owing to different wording. E.g., \({v}^{q}\), \({o}{o}{o}\) and \({v}^{q}\) \({k}^{q}\) \({k}^{q}\) have same meaning as in Assamese \({\overline{x[|S|]4}}\) means 1000. There are cases where we find the vocabulary-augmented LM generates the correct response, but in English. Also, there are cases where the LM stopped generation after producing the first character, which is correct. This can be due to the adverse effect of change in vocabulary distribution after augmentation. Another case is possibly related to the undesirable change in vocabulary distribution where the model starts with newly added tokens and ultimately produces the wrong outcome.

Figure 4: Example instance from each dataset.





### Vocabulary selection

The initial parts of Algorithm 2 are identical to sentence selection, but then we create score maps for _words_ to be used to get prospective tokens for augmentation in the LLM vocabulary, not _sentences_. Here, we get the important words by sorting by decreasing values and choosing the words with top percentile scores (line 21). In our experiments, we use the 50th percentile (median) as the threshold to avoid long tail words. Next, we create a dummy corpus by concatenating each word in \(V_{target}\), \(WC[w]\) number of times, separated by space (line 24). Finally, we pass the dummy corpus \(C_{dummy}\), and the desired token size \(K\) to a dictionary building and tokenization algorithm \(\psi\) (line 26). For our case, we use the SentencePieceBPE tokenization algorithm.

```
Inputs:  * CPT corpus \(C_{\text{CPT}}\)   * (existing) LLM tokenizer \(\mathcal{T}\)   * Tokenizer training algorithm \(\psi\)    * Parameter for weighted average \(\alpha\), \(\beta\)   * \(Q\), top percentile of words to send to tokenizer     * \(K\), the number of new tokens to include      * \(|\mathcal{W}\leftarrow\text{vocabulary from corpus $C_{\text{CPT}}$}\)   * fill \(WC\) (word count dictionary) using \(\mathcal{W},C_{\text{CPT}}\)   3:\(SWC\leftarrow\left\{\right\}\)   /* subword count dictionary */
4:for word \(w\in\mathcal{W}\)do
5: forward token \(t\in\mathcal{T}(w)\) do
6: \(SWC[t]\mathrel{+}=WC[w]\)
7: initialize \(R_{l}[w]\gets 0\) for all word \(w\in\mathcal{W}\)    8:/* Local word score table   9: Initialize \(R_{g}[w]\gets 0\) for all word \(w\in\mathcal{W}\)    10:/* Global word score table   11: Initialize \(R_{g}[w]\gets 0\) for all word \(w\in\mathcal{W}\)    12:/* Joint word score table   13:for word \(w\in\mathcal{W}\)do
14:for t \(\in\mathcal{T}(w)\)do
15: \(R_{l}[w]\mathrel{+}=SWC[t]-WC[w]\)
16:\(X_{co}\leftarrow\text{Word co-occurrence matrix of $C_{l}$}\)
17:\(R_{g}\mathbin{\leftarrow}\text{PageRank}(X_{co})\)   18:for w \(c\in\mathcal{W}\)do
19: \(R_{g}[v]\mathrel{=}\alpha R_{l}[w]+\beta\,R_{g}[w]\)
20:\(V_{target}\leftarrow\left\{\right\}\)
21: sort \(w\) by decreasing \(R_{j}[w]\) and add top-\(Q\) percentile words to \(V_{target}\)
22:\(C_{dummy}\leftarrow\text{empty string}\)   33:/* Dummy corpus for training LLM tokenizer   4:for word \(w\in V_{target}\)do
23: append \(w\) to \(C_{dummy}\) a total of \(WC[w]\) times
24:\(t_{aug}\leftarrow\psi(C_{dummy},K)\)
25:return\(t_{aug}\), the tokens selected for augmentation
```

**Algorithm 2** Vocabulary extension before CPT.

### Vocabulary augmentation helps generate more tokens

An interesting observation is with vocabulary augmentation, the LLM can generate more tokens than vanilla or without vocabulary-augmented LM, given a similar output generation limit. Consequently, sometimes it can improve summarization performance by extracting more information about the context. As shown in Figure 6, the LLM generates a longer and more informative summary of the given paragraph after vocabulary augmentation. However, a thorough investigation is needed to check if more generations are always linked with more relevant information.

## Appendix B

\begin{table}
\begin{tabular}{p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}} \hline
**Cases** & **Gold label** & **w/o Vacab add** & **w/ Vocab add** \\ \hline
**Correct but different wording** & yt,ooo (87,000) & yt,ooo (87,000) & yt \% ENSTM (45004 = 1000) \\
**Correct but in English** & xt (35) & xt (35) & xt 35 \\
**Stopped after few character** & xt (NASA) & xt (25) & xt \\
**Started with added vocab and failed** & xt (35) & xt (West Bengal) & xt (35) \\ \hline \end{tabular}
\end{table}
Table 6: Error cases for XorQA(xx) tasks. The second, third and fourth columns show the gold label, predication without and with additional vocabulary augmentation, respectively, for a particular question given a context. Important information related to the answers are underlined.





### Fragment ratio

## Appendix B Experimental settings

We use meta-llama/MetaLlama38BInstruct model for our CPT and zero-shot inferencing. We run all the experiments in a single 80GB A100 GPU system. To preserve cost, we do all the experiments one time, and to make them reproducible, we fix the seed value to 42. To run CPT with 30K data, it took around 3 hours on a single 80GB A100 GPU. For Zero-shot testing, for each task, we select 100 random instances for each language.

\begin{table}
\begin{tabular}{l|c} \hline
**Language** & **Fragment ratio** \\ \hline Santali & 13.67 \\ Telugu & 12.44 \\ Assamese & 8.82 \\ Bengali & 8.04 \\ Punjabi & 7.54 \\ Sanskrit & 5.32 \\ Bodo & 3.89 \\ Konkani & 3.67 \\ Urdu & 2.85 \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline
**Langage** & **Fragment ratio** \\ \hline Santali & 13.67 \\ Telugu & 12.44 \\ Assamese & 8.82 \\ Bengai & 8.04 \\ Punjab & 7.54 \\ Sanskrit & 5.32 \\ Bodo & 3.89 \\ Konkani & 3.67 \\ Urdu & 2.85 \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline
**Length** & **Metric–+** \\ **CPT data** \\ \hline \multirow{2}{*}{**Ctrl data**} & **Clark–++** \\  & **Crypt data** \\ \hline \multirow{2}{*}{**Unfold**} & **Tbe(Best) \\  & **Tr(Mean) \\ \hline \multirow{2}{*}{**Bengali**} & **Mid** \\  & **Tr(Best) \\ \hline \multirow{2}{*}{**Bengali**} & **Mid** \\  & **Tr(Best) \\ \hline \multirow{2}{*}{**Sansrkit**} & **Octhiki** \\  & **Low** \\ \hline \multirow{2}{*}{**Ronkani**} & **Devanagari** \\  & **Low** \\ \hline \multirow{2}{*}{**Bodo**} & **Devanagari** \\  & **Low** \\ \hline \end{tabular}
\end{table}
Table 7: Degree of fragmentation on 30K rank training corpus for 9 Indic languages using LLama-3-8b model tokenizer.

\begin{table}
\begin{tabular}{l|l|l} \hline
**Hyperparameter** & **Value** \\ \hline LLM & LLama-3 \\ \hline LLM parameter size & 8 Billion \\ \hline LLM model type & 8B-Instruct \\ \hline LLM temperature & 0.5 (for summarization), 0.3(for translation), 0.001(for QA tasks) \\ \hline LLM top p & 0.95 \\ \hline Seed & 42 \\ \hline LoRA r & 8 \\ \hline LoRA alpha & 32 \\ \hline LoRA dropout & 0.05 \\ \hline LoRA task type & CAUSAAL\_LM \\ \hline Learning rate & 1e-4 \\ \hline Batch size & 32 \\ \hline Epoch & 2 \\ \hline \(\alpha,\beta\) in Algorithm 1 \& 2 & 0.5,0.5 \\ \hline \end{tabular}
\end{table}
Table 8: Vanilla LLM’s performance comparison between CPT with **TR=Top Rank** Best and Mean results using small size (\(\leq\)30K) corpus for various Indic languages covering different scripts and resource types.



Figure 5: Details of the prompts used for each task.

 



## Appendix A

Figure 6: Added vocabulary can help LLM generate more text compared to vanilla LLM, given the same output generation limit. Here the summaries generated w/ and w/o additional vocabulary augmentation are shown in Green and Red, respectively. We see that the summary generated w/ additional vocabulary contains more words and information compared to w/o extra vocabulary augmented model.