Accelerating Multilingual Language Model for Excessively Tokenized Languages (2024)

Abstract: Recent advancements in large language models (LLMs) have remarkably enhanced performances on a variety of tasks in multiple languages. However, tokenizers in LLMs trained primarily on English-centric corpora often overly fragment a text into character or Unicode-level tokens in non-Roman alphabetic languages, leading to inefficient text generation. We introduce a simple yet effective framework to accelerate text generation in such languages. Our approach involves employing a new language model head with a vocabulary set tailored to a specific target language for a pretrained LLM. This is followed by fine-tuning the new head while incorporating a verification step to ensure the model’s performance is preserved. We show that this targeted finetuning, while freezing other model parameters, effectively reduces token fragmentation for the target language. Our extensive experiments demonstrate that the proposed framework increases the generation speed by a factor of 1.7 while maintaining the performance of pre-trained multilingual models on target monolingual tasks.

heading: 1 Introduction
text: Modern large language models (LLMs) (OpenAI, 2023; Touvron et al., 2023a; Antropic, 2023) have exhibited remarkable capabilities for a variety of tasks in multiple languages (Eloundou et al., 2023; Solaiman et al., 2023). Although these models are predominantly trained on English-centric data, they have shown a significant degree of multilingual proficiency (Bandarkar et al., 2023).
However, when applied to non-alphabetic languages, these models often suffer from slower text generation due to English-centric tokenization (Rust et al., 2021; Ahia et al., 2023; Petrov et al., 2023). Current tokenization techniques used in Large Language Models (LLMs) are data-driven and optimize segmentation based on the frequency
of characters or bytes within a specific corpus (Sennrich et al., 2016; Kudo, 2018). As a result, the tokenizers of multilingual models, which are heavily influenced by English-dominant training data, are predominantly composed of English subwords. This leads to excessive fragmentation, where nonEnglish words are overly segmented into a large number of subword units (Rust et al., 2021; Ahia et al., 2023; Petrov et al., 2023). The autoregressive nature of LLMs further amplifies this inefficiency, as it sequentially requires the generation of text.
To address these challenges, previous studies (Wang et al., 2019; Rust et al., 2021; Cui et al., 2023) have proposed replacing or augmenting the existing vocabulary of pre-trained multilingual models with language-specific vocabularies to more effectively encode monolingual text corpora. Specifically, Rust et al. (2021) improved mBERT (Devlin et al., 2019) by replacing its tokenizer with a monolingual one and incorporating an additional 100,000 pre-training steps. On the other hand, Cui et al. (2023) enhanced Llama (Touvron
11095
et al., 2023a) by expanding the Chinese vocabulary and further pre-training it on a 120GB text corpus that includes Chinese texts. However, this approach requires an extensive pre-training phase with a substantial amount of data.
Another approach to address the challenges is the use of small draft models (Leviathan et al., 2023; Chen et al., 2023a). These models generate draft output tokens, which are then verified by the original language model. However, a significant challenge arises when trying to identify or train a suitable small model that can handle multiple languages with reliable performance (Conneau et al., 2020; Bandarkar et al., 2023).
In response to these challenges, our research introduces MuMo, accelerating Multilingual language models for a targeted Monolingual text generation, particularly in non-alphabetic languages. MuMo incorporates a new vocabulary of a target language into the output layer, also known as the Language Model (LM) head, and predicts the next token from this expanded vocabulary. This approach requires training only the extended portion of the output layer and specific layers of the feed-forward network. Importantly, MuMo eliminates the need for extensive text corpora or a draft model, requiring only a modest corpus of the target language, approximately 44M tokens in size. Empirical results across summarization, and translation tasks in Korean and Japanese demonstrate that MuMo significantly accelerates text generation, achieving over a 1.7-fold increase in speed without significantly compromising output quality.

heading: 2 Related Work
text: Tokenization Disparity Subword tokenization, a common approach in LMs, is typically datadriven. Most of pre-trained tokenizers, which are often trained on predominantly English corpora, frequently result in excessive fragmentation of nonEnglish scripts (Rust et al., 2021; Zhang et al., 2022). Ahia et al. (2023); Petrov et al. (2023) have found significant tokenization disparities across languages in popular LLMs (Xue et al., 2021, 2022; Scao et al., 2022; OpenAI, 2023). Our work endeavors to address the slowdown in inference that arises due to tokenization disparity in non-alphabetic languages.
Modifying Pre-trained Vocabulary Previous works have explored the adaptation of pre-trained vocabularies or the addition of new tokens (Artetxe et al., 2020; Rust et al., 2021; Hong et al., 2021; Liu et al., 2023), these methods often necessitate extensive pre-training to integrate the new tokens effectively (Wang et al., 2019; Chau et al., 2020; Cui et al., 2023; Liu et al., 2023). In contrast, our MuMo framework sidesteps the need for fine-
tuning the parameters of pre-trained models to preserve the original capabilities of the pre-trained language model. Efforts to select items of pre-trained embedding matrix have been made (Abdaoui et al., 2020; Domhan et al., 2022; Ushio et al., 2023), but these have not yielded significant speed up where the size of the embedding layer is relatively small (Bogoychev et al., 2023).
Accelerating LLM Inference The quest to accelerate inference in auto-regressive large language models (LLMs) has led to a variety of approaches. There has been a proliferation of systems specifically engineered for LLM inference (Yu et al., 2022; Sheng et al., 2023; Xiao et al., 2023). Our proposed methodology can be harmonically integrated with the aforementioned techniques. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023a) have also been explored to increase inference velocity. However, the approach often relies on the assumption that a small model can maintain high fidelity when generating a series of multiple tokens. Moreover, acquiring a small yet competitive model may be tricky, especially in a multilingual setup (Conneau et al., 2020; Bandarkar et al., 2023). Our work distinguishes itself by specifically solving the inference inefficiency that arises from excessive fragmentation in the non-alphabetic context.
Parameter Efficient Cross-lingual Transfer Learning The curse of multilinguality, which refers a trade-off between the language coverage and model capacity (Conneau et al., 2020), is a significant issue even in massively multilingual models, such as mBERT, XLM-R, and mT5 (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021; Ansell et al., 2021). The problem has been mitigated through modular parameterefficient adaptations of the multilingual models through lightweight adapters (Houlsby et al., 2019): additional trainable parameters inserted into the transformer layers of model (Pfeiffer et al., 2020; Üstün et al., 2020; Vidoni et al., 2020; Parović et al., 2022) for a target language. These techniques bear a resemblance to ours, in that they involve training partial parameters of a language model with a small amount of target language corpus. However, our goal is fundamentally different: we aim to accelerate the inference, whereas previous studies focus on improving the representational capability in target languages for multilingual models.

heading: 3 Proposed Framework
text: We propose a framework named MuMo to accelerate the inference speed of a pre-trained multilingual LM for a non-alphabetic monolingual language via a given small monolingual dataset. In the section, we introduce 1) the model architecture, 2) the finetuning process on a small targeted language dataset, and 3) the inference process of the proposed framework.

heading: 3.1 Model Architecture
text: We illustrate the model architecture of MuMo in Fig. 3.
Pre-trained Multilingual Model We consider a setting in which a pre-trained multilingual model fmulti is given. The model consists of 1) Transformer layers that consist of attention and feedforward network, and 2) an output embedding layer called language model (LM) head. We denote Vmulti as the multilingual vocabulary set of the model objective, as LMLE(pmulti,x) =∑|x|
t=1 log pmulti(xt|x<t),
Target Monolingual LM Head The primary concept involves modifying pre-trained representations to predict a singular token unit within a target monolingual vocabulary Vmono. The Target Monolingual LM head fmono projects the hidden representation h, which is composed of two main components: a feed-forward network (FFN) and an output linear layer, represented as gmono : Rdmono → R|Vmono|:
FFN(h) = q(W⊤1 h)W2 ∈ Rdmono , (1)
where W1 ∈ Rdmulti×dffn and W2 ∈ Rdffn×dmono are the weight matrices, q is non-linearity function, and dmono represents the dimension of the target language representaiton. We set dffn as dmulti/4, and the non-linearity function q as SwiGLU (Shazeer, 2020). The output linear layer gmono then generates a subword token:
fmono(h) = gmono(FFN(h)) ∈ R|Vmono|. (2)
MuMo LM Head Note that the output space of fmono is restricted to tokens in the Vmono. Inspired by Lan et al. (2023), we simply extend the fmono by concatenating the output linear layer of pre-trained multilingual model. This is particularly useful when there is no suitable token in Vmono to
predict, such as special symbols or alphabet-based tokens for non-alphabet languages.
Formally, given context representation ht−1, the output of the MuMo LM head is computed as:
fmumo(ht−1) =
[fmulti(ht−1); fmono(ht−1)] ∈ R|Vmulti|+|Vmono| (3)
where the symbol ; indicates the concatenation of two vectors, and the fmumo indicates the output of the MuMo LM head. Thus, the MuMo LM head is composed of a combination of the pre-trained language model head and Target Monolingual LM head.

heading: 3.2 Fine-tuning
text: In the proposed framework, we only fine-tune the target monolingual LM head fmono leveraging a small given target monolingual dataset. Note that the parameters of the pre-trained multilingual model remain frozen during the process. The model is fine-tuned by maximizing the log-likelihood of a sequence:
max fmono
LMLE(pmumo,x) = ∑T
t=1 log pmumo(xt|x<t) , (4)
where pmumo(xt|x<t) = Softmax(fmumo(ht−1)).

heading: 3.3 Inference
text: Despite the availability of direct generation based on the pmumo, the newly initialized Target Monolingual LM head, which is trained on limited data, may be constrained by generalization capabilities beyond the training dataset. The key concept is to leverage the probabilistic knowledge acquired by the pre-trained model pmulti, which has been extensively trained on large text corpora.
3.3.1 Step 1: Top-k Selection Initially, we select top-k candidates based on the probability pmumo(xt|x<t). We set k as 10 for all experiments. Given the fact that we do not modify the input embedding of the pre-trained model, we are unable to feed the predicted word if a word does not belong in Vmulti during the subsequent iteration. Instead, we input the predicted word as the tokenization units of the pre-trained vocabulary. For example, let’s consider the Korean word “수 소”, which corresponds to a sequence of two tokens (“수”, “소”) in Vmulti. If the Korean word “수소” is selected among the Top-k candidates, we employ these two multilingual tokens.

heading: 3.3.2 Step 2: Verification
text: Then, the feasibility of these potential completions is measured using the log-joint probability distribution over pmulti. To account for shorter sequences
naturally having higher scores (Jean et al., 2015; Murray and Chiang, 2018), we normalize each candidate’s score by its token length.
We measure the feasiblity for a candidate sequence as follows:
σ(ci) = 1
li
li∑
k=1
log pmulti(c i t+k|ci<t+k,x<t), (5)
where ci symbolizes a predicted token within the top-k candidates, pmulti represents the probability as determined by the pre-trained multilingual model, and li corresponds to the sequence length of the candidate ci.
From the k candidates, the ultimate prediction can be derived from both deterministic and stochastic manners, depending on decoding strategies.

heading: 4 Experiments

heading: 4.1 Setup
text: Languages As a case study, we focus on two non-roman alphabetical languages: Korean and Japanese. Since we aimed to utilize a pre-trained model with a reasonable level of effectiveness in the target language, it is essential that the language is explicitly mentioned as being trained within the pre-training corpus. In this context, we considered languages included in the Llama-2 (Touvron et al., 2023b) pre-training corpus. Moreover, the chosen language needed to exhibit the excessive fragmentation problem (Ahia et al., 2023; Petrov et al., 2023) by the English-centric pre-trained tokenizer. (See the Figure 1) This criterion led to the exclusion of most European languages such as French, German, and Portuguese. Finally, we conduct a study on multiple tasks, necessitating the existence of an instruction dataset for the target language. Due to these considerations, we only implement the experiment in Korean and Japanese.
Model We utilize the Llama-2 13B model (Touvron et al., 2023b) for all experiments. We observed some language alignment discrepancies between instructions and responses when using the Llama-2 13B chat model.1 To address the issue, we conduct multilingual instruction tuning (Muennighoff et al., 2022) for English, Korean, and Japanese languages using the ShareGPT and Alpaca (Chen et al., 2023c). This process improve the model’s fluency in each language (Muennighoff et al., 2022;
1meta-llama/Llama-2-13b-chat
Chen et al., 2023b). We also report our results test on Llama-1 13B (Touvron et al., 2023a) in Appendix.
Implementation of MuMo To construct targeted monolingual vocabularies in MuMo Framework, we levergage the tokenizers from the off-the-shelf model, as shown in Table 2. We selected monolingual tokens by filtering vocabulary items based on the Unicode range of each monolingual script. Additionally, we excluded items from the selection if they were already present in the pre-trained vocabulary. In terms of the preprocessing algorithm, we employ a forward maximum matching strategy to identify words in a target language vocabulary. This strategy identifies the longest sequence of tokens that aligns with a word in the target language vocabulary.
Regarding the initialization of gmono, we utilize the LM head of the pre-trained multilingual model. For example, when the Korean word "태양" is tokenized into subword units (“\0xed”, ..., “\0x91”) using the pre-trained vocabulary, we initialize the Korean LM head of "태양" by taking the mean of the corresponding subword embeddings of the multilingual LM head. This process ensures that the initialized embeddings of Target Monolingual head represent the original word in the multilingual context.
Fine-tuning We only train the Target Monolingual LM head gmono with the translated ShareGPT and Alpaca datasets (Chen et al., 2023c) in Korean, and Japanese. The training is done with 1500 steps with one batch consisting of 128 examples. We use the AdamW (Loshchilov and Hutter, 2019) optimizer with a learning rate of 0.001, weight decay of 0.01, and 150 steps of warm-up.
Evaluation We choose two representative generation tasks: summarization and translation. For summarization, we use 500 examples from XLSum (Hasan et al., 2021), and for translation, we use 500 examples from the FLoRes-200 (Goyal et al., 2022) dataset. We translate English sentences to each target language sentence.
For each task, we report 0-shot results for summarization, and 3-shot results for translation. We set the maximum sequence length as 512. We utilize flash-attention 2 (Dao, 2023) and bfloat16 types for text generation.
Metrics In the summarization task, we gauge the reliability of the generated content by calculating the ROUGE-2 and ROUGE-L (Lin, 2004) scores, averaging the results across 5 different generated summaries. Likewise, for the translation task, we measure the quality of the translations by computing the BLEU (Papineni et al., 2002) score, again averaging over 5 translation results.2 We report Tokens/sec to measure the inference speed of the models.

heading: 4.2 Baselines
text: We consider the following several baselines for the comparison with the proposed method. Note that all the baselines are implemented instruction-tuned model with multilingual instruction dataset (Chen et al., 2023c).
Vanilla Decoding The autoregressive generation is to sequentially sample the subsequent word based on the probability distribution over the pretrained vocabulary. This approach serves as the standard against which improvements are measured. Accounting for the nature of task, all the baselines and our framework utilizes samplingbased decoding strategy with temperature as 0.1, k
2We utilize SacreBLEU scores with the signature BLEU |nrefs:1 |case:mixed |eff:no |tok:ko,ja-mecab|smooth:exp |version:2.2.0.
as 10 for top-k sampling (Fan et al., 2018) and p as 0.7 for nucleus sampling (Holtzman et al., 2020).
Speculative Decoding Speculative decoding approach (Chen et al., 2023a; Leviathan et al., 2023) employs a preliminary "draft" model to rapidly generate a set of token candidates at each decoding step. Subsequently, these candidates undergo a validation process by the original language model to ascertain their likelihood as plausible continuations of the text. We implement two variants of this method: one with the capability to reject unsuitable candidates (Spec.) and another without its rejection module (Spec. w/o Rejection). For the draft model, we utilize Llama-2 7B (Touvron et al., 2023b). Following the implementation of Chen et al. (2023a), we generate 5 draft tokens at each iteration.
Lexical Shortlisting Lexical Shortlisting (Shortlisting) (Abdaoui et al., 2020; Ushio et al., 2023), or vocabulary selection, is the approach that optimizes the decoding process by allowing it to generate a word within a set of tokens during the inference stage (Ushio et al., 2023). We implement to filter out tokens that are not present within the corresponding target language subset of the mC4 corpus (Xue et al., 2021), as Ushio et al. (2023).

heading: 4.3 Results
text: Table 3 shows the generation results in both summarization and translation tasks. For the summarization task in Korean, MuMo outperforms all baselines in terms of speed, achieving a 1.92x speedup over the Vanilla Decoding while maintaining competitive ROUGE scores. In translation, MuMo again demonstrates superior efficiency with a 1.70x
speed-up and even shows an improvement in BLEU score compared to Vanilla Decoding.
In the case of Japanese, the results are similar, with MuMo achieving a 2.02x speed-up in summarization and a 1.75x speed-up in translation. The ROUGE and BLEU scores for MuMo are on par with or slightly below Vanilla Decoding, indicating that the increase in speed does not significantly compromise the quality of the output.
Shortlisting shows only marginal gains in speed across both languages and every tasks, while preserving the generation capability. This is likely because the relative computational cost of processing the embedding matrix is reduced in larger models, making vocabulary reduction less impactful (Berard et al., 2021; Ushio et al., 2023). On the other hand, the Spec. heavily relies on the capacity of the draft model, as shown as the comparison with (Spec. w/o Rejection). If the draft model lacks of sufficient multilingual capacity, it may not generate high-quality candidates, leading to a lower acceptance rate by the original model and thus reduced efficiency.
The superior performance of MuMo in terms of inference speed can be primarily attributed to its capability to predict larger linguistic units compared to those in the pre-trained vocabulary. We found that the target language tokens in Vmono are typically tokenized into 3-4 separate tokens in Vmulti, suggesting that the decoding step could potentially be reduced by 3-4 times. It is hypothesized that the inference speed is significantly influenced by the disparity between the pre-trained multilingual vocabulary and the target language.

heading: 5 Further Analysis

heading: 5.1 Comparative Analysis of Fine-Tuning Strategies
text: In the section, we provide a comparative analysis of three distinct fine-tuning strategies for multilingual models. This analysis aims to highlight the
advantages and disadvantages of each strategy, particularly in terms of dataset requirements. and the number of parameters to train.

heading: 5.1.1 Setup
text: The two strategies compared in the analysis are:
1. Vanilla Fine-tuning: This strategy, which serves as a baseline, involves fine-tuning a standard multilingual model on a target monolingual instruction dataset (44M tokens) without any modifications to the pre-trained vocabulary.
2. Vocabulary Expansion: Inspired by prior work (Chau et al., 2020; Cui et al., 2023), this strategy involves expanding the vocabulary of the pre-trained multilingual model and fine-tuning on the instruction dataset. This method, unlike MuMo, expands not only the LM head but also the token embedding in the input layer. Two implementations of this strategy are considered. The first involves pre-training on large-scale text corpora (60B tokens)3 before fine-tuning on the instruction dataset. This strategy is marked with a dagger in Table 4. The second only undergoes the fine-tuning phase on the instruction dataset.
To account for the variability of token unit between the different strategies, we report the inference speed with the morphemes per second (Morphemes/sec), providing a standardized measurement.4 We only compare the baselines in Korean, because of the availability of model.

heading: 5.1.2 Discussion
text: Table 4 reveals a consistent trend across both summarization and translation tasks. The vocabulary expansion strategies, which expand the dimension of both the token embeddings and LM head, exhibit significant increases in inference speed, but this is accompanied by a substantial decrease in the quality of the generated output when not trained on
3We use the off-the-shelf checkpoint from beomi/llama-2koen-13b
4python-mecab-ko
large-scale text corpora. This indicates that merely fine-tuning with an expanded vocabulary on a limited downstream dataset may not suffice to maintain high-quality text generation, as suggested by (Conneau et al., 2020). Furthermore, while vocabulary expansion with pre-training achieves notable speed improvements, it does not exhibit significant enhancements in generation quality.
In contrast, our proposed method exhibits a modest increase in speed while also slightly improving BLEU scores relative to vanilla fine-tuning. The principal advantage of our method lies in its capacity to attain these results without necessitating vast monolingual text corpora. This approach not only reduces the number of parameters that need to be fine-tuned, making it more parameter-efficient but also lessens the dependency on large-scale data for pre-training, making it a more data-efficient solution.

heading: 5.2 Initialization of Target Monolingual LM Head
text: We investigate the impact of three different initialization strategies on the target monolingual LM head gmono in the Target Monolingual LM head. The first strategy involves leveraging embeddings that correspond to the pre-trained representation of a targeted monolingual LM head, termed as MONO-INIT. The second strategy is initializing the parameters with random value using Gaussian distribution (RANDOM-INIT). Lastly, we utilize the embeddings from the pre-trained multilingual
LM head (MULTI-INIT), as the main experiment. This is achieved by averaging the output embeddings of the multilingual model.
Table 17 shows that MULTI-INIT achieves a ROUGE-L score of 36.3 and a BLEU score of 21.7, which are close to the 36.2 ROUGE-L and 20.9 BLEU scores of MONO-INIT. On the other hand, RANDOM-INIT shows a decrease in performance, with a ROUGE-L score of 35.5 and a BLEU score of 17.2.
The result demonstrates that the MULTI-INIT approach is almost equally effective with MONOINIT. This suggests that our framework can be utilized some languages that have an off-the-shelf vocabulary set but lack suitable pre-trained representations.

heading: 5.3 Effectiveness of Verification Step
text: We design an ablation study to investigate the role of the verification step in the inference process ( Sec. 3.3.2). To assess the impact of the verification step, we generated sequences without employing the verification step.
From the results in Table 6, conducted in both Korean and Japanese, we notice that the overall generation speed is approximately 1.2 times faster when the verification is excluded. However, it is crucial to highlight that the exclusion of the verification step in the inference phase leads to a significant reduction in the generation quality. This is evident in the decrease in ROUGE-2, ROUGE-L, and BLEU scores for both languages when the ver-
ification module is not used, as shown in the table. This suggests that while the verification step may slightly slow down the generation process, it plays a vital role in preserving the model’s generation capability.

heading: 5.4 Comparative Study in Single-Task Training
text: In the experiment, our primary objective is to investigate whether the inherent capabilities of the instruction-tuned multilingual model, which handles a variety of tasks, could be compromised when trained exclusively on single tasks using either Vocabulary Expansion or MuMo. Both methods introduce newly initialized parameters, raising concerns about potential impacts on the model’s versatility. To address these concerns, we separately trained the model on each task - Question Answering (QA) (Lim et al., 2019; Kurihara et al., 2022) and Summarization (Hasan et al., 2021) - and subsequently conducted a comparative analysis between Vocabulary Expansion and MuMo.
For evaluation, we utilize multiple-task datasets, specifically Korean5 and Japanese6, which consist solely of questions. For the measurement, We adopt the single-answer grading setup from LLM-as-ajudge (Zheng et al., 2023). This involves presenting a question along with model-generated answers to GPT-4 (acting as the judge) for assessment. The answers are graded on a scale from 1 to 10.
As depicted in Figure 4, the instruction-tuned model initially achieves an average grading of 7.2 in the Korean experiment. However, when finetuned using only the QA task, Vocabulary Expansion receives a grading of 1.8, while MuMo receives a grading of 5.9. When trained solely on the summarization task, Vocabulary Expansion receives a grading of 1.6, while MuMo receives a grading of 4.7. Similar trends are observed in the Japanese experiment. The original model receives an average grading of 6.8. When fine-tuned with only the QA task, Vocabulary Expansion receives a grading of 2.1, while MuMo receives a grading of 5.2. When trained exclusively on the summarization task, Vocabulary Expansion receives a grading of 1.2, while MuMo receives a grading of 4.4.
These results suggest that while the grading of the model decreases when trained on single tasks using either method, the decrease is less pro-
5Korean-MT-bench 6Japanese-MT-Bench
nounced with MuMo. This indicates that MuMo is more effective at preserving the model’s multitask proficiency compared to Vocabulary Expansion. However, it is also clear that neither method can fully maintain the model’s original instructionfollowing abilities on multiple tasks when trained solely on single tasks. These findings suggest that the instruction dataset, which the model was originally trained on, is crucial for preserving the pretrained model’s capabilities.

heading: 6 Conclusion
text: Our study has successfully tackled the challenges in generating text for non-alphabet languages, particularly those associated with excessive fragmentation issues. The approach not only speeds up text generation but also paves the way for more efficient multilingual language applications. Our future work will broaden our experimental scope to languages that were not sufficiently represented in the pre-trained multilingual language model.
Limitations
Our proposed framework has not been evaluated with languages that exhibit excessive fragmentation issues, such as Tamil, Hebrew, and Arabic (Ahia et al., 2023; Petrov et al., 2023). These languages were not explicitly mentioned in the pre-training corpus of Llama-2 (Touvron et al., 2023b). Additionally, our framework requires off-the-shelf tokenizers for target languages to make Target monolingual LM Head. Our method does not alter the input sequence length, as we focus solely on improving the unit of prediction. This approach This approach differs from the the previous studies (Rust et al., 2021; Cui et al., 2023) which efficiently encode text at the input-level sequence length for excessively tokenized languages. Furthermore, the language models evaluated in the study are restricted to a maximum size of 13B. Larger models, such as Llama-2 30B or 70B, were not implemented due to constraints on available computational resources.

heading: Acknowledgements
text: We thank Chaehun Park and Daeyoung Kim for their valuable discussions and feedback on the paper.

heading: A Dataset Details
text: Training Data Our study employed a multilingual instruction dataset from Chen et al. (2023c), encompassing Korean and Japanese, for multilingual instruction tuning. Specifically, we utilized ShareGPT and Alpaca-GPT4 for each respective language. The dataset comprises 56k, 55k, and 168k examples for Korean, Japanese and English respectively. To train MuMo LM head, we use ShareGPT and Alpaca-GPT4 (Chen et al., 2023c) in Korean and Japanese for each language.
Evaluation Data In summarization task, we use validation and test split of XLSum (Hasan et al., 2021), which consist of 1100 examples. We found that more than half of the samples within the validation and test split surpassed the maximum sequence length of Llama-2. Consequently, we filtered out examples exceeding 1536 tokens. From the remaining examples, we randomly selected 300 for our experiments.
Regarding translation task, the dev-test set of FLoRes-200 (Goyal et al., 2022) is employed, consisting of 1012 parallel examples across both languages. We randomly use 3 examples as 3-shot prompts from training set for individual run.
When evaluating multiple-task benchmark dataset 6, we exclude examples in coding and math categories.

heading: B Additional Results
text: Experiment on other Language Model Table 13, and Table 13 present the comparative study in Llama-1 13B (Touvron et al., 2023a) and Mistral 7B (Jiang et al., 2023) respectively.
Generation Results Table 15 and Table 16 present generated texts in summarization and translation tasks.

heading: C Environment Details
text: All experiments are implemented using an A10040GB GPU. The library versions utilized across all experiments include Python 3.9.10, Pytorch 2.1.0, and Transformers 4.34.0.
D Hyperparameter Details

heading: Future Work
text: Our future work will broaden our experimental scope to languages that were not sufficiently represented in the pre-trained multilingual language model. Limitations
Our proposed framework has not been evaluated with languages that exhibit excessive fragmentation issues, such as Tamil, Hebrew, and Arabic (Ahia et al. , 2023; Petrov et al. , 2023). These languages were not explicitly mentioned in the pre-training corpus of Llama-2 (Touvron et al. , 2023b). Additionally, our framework requires off-the-shelf tokenizers for target languages to make Target monolingual LM Head. Our method does not alter the input sequence length, as we focus solely on improving the unit of prediction. This approach This approach differs from the the previous studies (Rust et al. , 2021; Cui et al. , 2023) which efficiently encode text at the input-level sequence length for excessively tokenized languages. Furthermore, the language models evaluated in the study are restricted to a maximum size of 13B. Larger models, such as Llama-2 30B or 70B, were not implemented due to constraints on available computational resources