How Vocabulary Sharing Facilitates Multilingualism in LLaMA? (2024)

Abstract: Large Language Models (LLMs), often show strong performance on English tasks, while exhibiting limitations on other languages. What is an LLM’s multilingual capability when it is trained only on certain languages? The underlying mechanism remains unclear. This study endeavors to examine the multilingual capability of LLMs from the vocabulary sharing perspective by conducting an exhaustive analysis across 101 languages. Through the investigation of the performance gap before and after embedding fine-tuning, we discovered four distinct quadrants. By delving into each quadrant we provide actionable and efficient guidelines for tuning these languages. Extensive experiments reveal that existing LLMs possess multilingual capabilities that surpass our expectations, and we can significantly improve the multilingual performance of LLMs based on these attributes of each quadrant 1.

heading: 1 Introduction
text: Large Language Models (LLM), such as GPT (Brown et al., 2020; OpenAI, 2023), PaLM (Chowdhery et al., 2022), and LLaMA (Touvron et al., 2023a,b), are trained on massive amounts of text data. While these models show strong capabilities on English tasks, their performance in other languages is often limited (Zhu et al., 2023a; Bang et al., 2023).
Significant research effort has been dedicated to enhancing multilingual capabilities by using methods such as continued training with abundant monolingual data (Cui et al., 2023; Yang et al., 2023), or employing instruction-tuning 2 techniques (Zhu et al., 2023b; Li et al., 2023). Despite the encouraging results, the underlying mechanism of LLM’s multilingual capability remains mysterious.
1https://github.com/CONE-MT/ Vocabulary-Sharing-Facilitates-Multilingualism.
2Instruction tuning is a method used to train large language models to follow specific instructions to solve a task. We provide an example of instruction tuning format in Appendix A.
Multilingual capability (Lin et al., 2019) refers to how effectively models that have been fine-tuned in one source language can be applied to tasks in other languages and achieve decent performance. This ability has been extensively studied in machine translation (Johnson et al., 2017; Gu et al., 2018; Neubig and Hu, 2018; Aharoni et al., 2019; Zhang et al., 2020) and multilingual pre-trained models (Pires et al., 2019; Libovický et al., 2019; Wu and Dredze, 2020). However, it has not been investigated for English-centric LLMs, given that the pre-training data is predominantly in English. We aim to address this issue by focusing on the multilingual foundation of pre-trained LLMs and providing some guidance to help other people train LLMs more efficiently for non-English languages. Generally, multilingual capabilities are built on two key foundations: the volume of multilingual data used during the pre-training stage (Touvron et al., 2023a,b; Li et al., 2023; Scao et al., 2022), and the
12111
vocabulary (Pires et al., 2019; Chung et al., 2020; Liang et al., 2023). In this work, we focus on the latter: vocabulary.
To investigate the multilingual foundation provided by the vocabulary of an existing LLM, we only fine-tune the embedding layer and keep the rest of the parameters frozen, denoted as Embed FT. This approach requires fewer adjustments to the model parameters than full fine-tuning, and unlike LoRA (Hu et al., 2021), it doesn’t require any additional model structure. In our experiments, we focus on the LLaMA as a case study, but the analysis method can be applied to other LLMs.
To examine the multilingual capabilities of LLMs without loss of generality, we applied Embed FT to a 10k en→x bilingual instruction translation dataset generated by 10k sentences pairs across four distinct datasets: Lego-MT (Yuan et al., 2023), Wikimatrix (Schwenk et al., 2021) and Newscommentary (Tiedemann, 2012), and Ted (Ye et al., 2018). We evaluated the bilingual performance (refers to the performance of the fine-tuning languages) and multilingual performance (refers to the performance of other languages) of each model to determine if there was a significant positive or negative change compared to the original model. From the results, all languages can be categorized into four distinct quadrants.
The multilingual capability quadrant of the TED dataset, illustrated in Figure 1, includes four quadrants: the reciprocal quadrant, the altruistic quadrant, the stagnant quadrant, and the selfish quadrant. The full definition of each quadrant is in Section 3. The selfish quadrant refers to scenarios where the fine-tuned model only improves on the fine-tuning language directions but not other languages. It is considered a default quadrant, as languages that fall into the selfish quadrant exhibit behavior that aligns intuitively with the effects of bilingual fine-tuning.
Certain languages such as Bulgarian fall into the reciprocal quadrant, where training with bilingual data (e.g. English→Bulgarian) not only enhances bilingual performance but also boosts the multilingual capabilities of other languages. The majority of these languages in this quadrant are from the Indo-European family, benefiting from the pretraining data and vocabulary sharing. For these languages, we find that there is no need to fine-tune all parameters, which could lead to overfitting to a specific language. We recommend fine-tuning only the embedding layer, which yields bilingual perfor-
mance on par with full fine-tuning while preserving the model’s multilingual capabilities.
Remarkably, certain languages exhibit altruistic characteristics. When we use these languages as training data, their primary effect is to enhance multilingual performance. Upon further analysis, we discovered that the decline in bilingual performance is primarily due to a change in error types: from those that are easy to score to those that are more challenging. The improvement in multilingual performance, on the other hand, stems from vocabulary sharing. For such languages, employing a small dataset for full fine-tuning can be more effective for multilingual capabilities.
Indeed, there are certain languages located in the stagnant quadrant that are quite stubborn. This means that using data from these languages doesn’t improve bilingual performance or bring about multilingual benefits. Regardless of parametereffective tuning strategies (LoRA) or extensive fine-tuning on large datasets, the results are still disappointing. Interestingly, even expanding the vocabulary for full fine-tuning doesn’t lead to better results. Then, we find that existing LLMs often over-tokenized these languages, which reduces the density of information they carry. By simply removing the common prefix of tokenized representation, we have seen an average improvement of 2.5 spBLEU points. Our main contributions are:
• We conduct a systematic analysis of the impacts of LLM’s vocabulary on their multilingual capabilities, and discover four quadrants based on their embedding fine-tuning performance gap.
• We provide practical and efficient technical guides to improve multilingual capabilities for each quadrant.
• We perform extensive experiments to verify the effectiveness of quadrant-specific fine-tuning techniques (e.g. 2.5 spBLEU improvement in stagnant quadrant).

heading: 2 Background
text: Multilingual Large Language Model Large language models (LLMs; OpenAI, 2023; Zhang et al., 2022; Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a,b have shown demonstrated performance in English, but the performance in other languages is limited. To address this limitation, researchers have proposed multilingual language models (MLLMs) that can handle multiple
languages simultaneously. The first line of research proposes to learn a shared representation space for multiple languages by first pre-training on multilingual data and then fine-tuning for specific tasks or languages. Representative works include mBERT (Devlin et al., 2019), XLM (Lample and Conneau, 2019), XLMR (Conneau et al., 2020), BLOOM (Scao et al., 2022), XGLM (Lin et al., 2022b), and PolyLM (Wei et al., 2023). Another line of research adopted existing monolingual LLMs to multilingual using techniques such as prompt engineering (Muennighoff et al., 2023; Yong et al., 2023), instruction tuning (Zhu et al., 2023b; Li et al., 2023; Jiao et al., 2023), or continue training (Cui et al., 2023; Yang et al., 2023).
The Multilingual Foundation of LLM The robust multilingual capabilities of LLM are founded on: the presence of diverse multilingual data (Touvron et al., 2023a,b; Li et al., 2023; Scao et al., 2022) and vocabulary (Pires et al., 2019; Chung et al., 2020; Liang et al., 2023).
The size of multilingual data is a critical factor in the multilingual capabilities of LLM. LLaMA (Touvron et al., 2023a) is pre-trained on a vast scale, with over 1.6 trillion tokens, of which less than 9% is multilingual data, 3 spanning 20 different languages. LLaMA2 (Touvron et al., 2023b) further enhances the proportion of multilingual data to approximately 11% and increases the number of languages to around 26. PolyLM (Wei et al., 2023) is trained on 640 billion tokens and supports 18 of the most commonly spoken languages. BLOOM (Scao et al., 2022) is trained with data from 46 natural languages. The existing language data in the pre-training phase provides LLM with a robust foundation for multilingual capabilities.
Another key factor is vocabulary construction. A common approach to constructing vocabulary involves tokenizing text into subwords: including Byte-level Byte-Pair-Encoding (BBPE), BytePair-Encoding (BPE), SentencePiece (SP) (Sennrich et al., 2016; Kudo and Richardson, 2018; Wang et al., 2019), which are units smaller than words that can encapsulate morphological variations. Nevertheless, in a multilingual context encompassing a diverse range of scripts, the base
3The original wording (4.5%) in the LLaMA paper, which only mentioned the inclusion of 20 languages of Wikipedia data. After meticulously checking the datasets involved in the LLaMA pre-training to provide a rigorous account of the quantity of non-English data, we discovered that the Gutenberg dataset includes some multilingual data.
vocabulary comprising subwords can become exceedingly large, leading to inefficiency and sparsity. Further Information on BBPE is in Appendix B.

heading: 3 Inherent Multilingual Capabilities
text: In this section, we begin by exploring the inherent multilingual capabilities of LLMs and give some fascinating observations detailed in Section 3.1. Drawing on these insights, we then proceed to conduct an in-depth examination of the multilingual capability of LLM in Section 3.2.

heading: 3.1 Observation
text: Setting We fine-tune a single LLaMA model using en→x data on the Lego-MT dataset, yielding 101 bilingual-tuned models. We train the LLaMA-7B with en→ro, en→no, en→ms, and en→luo data separately, and then thoroughly evaluate each bilingual-tuned model on all 101 language pairs (en→x) to probe its multilingual translation performance on Flores-101’s devtest set. Additionally, we follow the same settings for all models throughout the paper. Over 50 bilingual models were tuned using the Wikimatrix and Newscommentary datasets, and more than 55 bilingual models were tuned using the Ted dataset. All these models were trained with identical parameter settings, specifically a learning rate of 2e-5 and a total of 3 epochs, and evaluated bilingual and multilingual performance with beam size = 4.
Phenomena We observe that LLM demonstrates superior multilingual capabilities far beyond expectation. Some interesting phenomena are:
Phenomenon 1: LLaMA can support additional languages beyond those explicitly mentioned in their pretraining corpus. In the leftmost part of Figure 2, it is evident that the bilingual-finetuned en→ro, en→ms, and en→no models exhibit a significant improvement over the original model in en→af translation. This outcome is quite surprising considering that neither LLaMA’s pretraining corpus 4 nor our fine-tuning data contain any text related to af. Similar observations can be made for numerous other languages, as depicted in Figure 2. This indicates the LLaMA may possess a more robust capability for handling multiple languages than previously expected.
4LLaMA utilizes Wikipedia for its pre-training data which includes 20 languages: bg, ca, cs, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, uk.
Phenomenon 2: The performance distribution of bilingual-tuned models across multiple languages exhibits remarkable consistency. Intuitively, different bilingual models shall have very different multilingual performance distributions. However, from line plots in Figure 2, we observe that three bilingual models (en→ro, en→no, and en→ms) showcase an exceptional level of consistency. We speculate such a phenomenon might be caused by a similar instruction-tuning process. However, further experiments on the en→luo model reject the above hypothesis. Therefore, we hypothesize that such a phenomenon only occurs in certain languages and might be related to certain underexplored mechanisms.

heading: 3.2 Quantify Multilingual Capability at Scale
text: Given the phenomena above, we scale our evaluation to more languages to validate our findings.
Setting We conduct experiments on LLaMA with 3 epochs on all 101 language pairs(en→x) in Flores-101 using parallel multilingual corpora. For each language pair, we sample at most 10k sentence pairs at random unless it has fewer than 10k sentence pairs. We then train models using the Embed FT method. For evaluation on Flores-101’s devtest including 12 respective languages (details provided in Appendix E), we use a beam size of 4 and spBLEU (SentencePiece BLEU) as the metric.
Observation Inspecting the large-scale evaluation results, we make the following observation: some bilingual models exhibit highly similar yet surprising behaviors. As a counterintuitive example, we showcase a group of “selfless” bilingual models (see Table 1). In common belief, fine-tuning LLM on one language pair shall def-
initely improve its performance. However, to our surprise, fine-tuning these “selfless” bilingual models(column LG) might even hurt their performance(comparing en→LG column with LLaMA column). What’s even more interesting is that the multilingual performance of these models is significantly improved.

heading: Type LG LLaMA en→af en→ro en→LG Multilingual
text: To quantitatively investigate the language clustering behavior, as well as dig the root of the phenomena mentioned above, we propose to categorize languages into four quadrants using a two-dimensional Cartesian system. As shown in Figure 1, the x-axis represents bilingual performance, and the y-axis represents multilingual performance. Before clustering, we first establish a categorization criteria.
Criteria We use the bilingual/multilingual performance changes before and after fine-tuning to measure whether the tuning results in gain or loss:
∆lg =
{ Ppost Ppre
− 2, if Ppre ≥ T Ppost−2T
Ppre , otherwise
(1)
where the Ppost represents the translation performance after fine-tuning, Ppre indicates the performance before the fine-tuning process, T serves as a threshold for smoothing, and 2 is a hyperparameter quantifies the extend for significant changes. We select based on a preliminary study, for further information see Appendix D. The calculation of ∆lg for bilingual performance is straightforward, for the multilingual performance, we consider the average performance of en→af and en→ro translations. This is primarily due to our observation that changes in multilingual performance are significantly mirrored in that of en→af and en→ro, details are in Appendix E.
Quadrant Details We calculate the above criteria on four multilingual corpora: Lego-MT (Yuan et al., 2023), Wikimatrix (Schwenk et al., 2021) and Newscommentary (Tiedemann, 2012), and Ted (Ye et al., 2018), and obtain a consistent language classification results as in Table 2. The details of datasets and categorization are in Appendix E. We summarize the behavior of four quadrants below (also shown in Figure 1):
• Reciprocal Quadrant: Models trained on languages from reciprocal quadrant, demonstrate strong bilingual and multilingual performance at the same time.
• Altruistic Quadrant: Models trained on these languages prioritize enhancing others, with minimal impact on their bilingual performance.
• Stagnant Quadrant: Existing tuning strategies appear to have minimal impact on these languages.
• Selfish Quadrant: The selfish quadrant is the most intuitive one: training in a specific language typically improves the performance of that language and merely affects other languages.
Please note that the categorization proposed is merely one possibility derived from certain criteria, and there might exist alternatives that lead to slightly different classification results. Nonetheless, We only focus on the consistent classification, produced by Eq. 1, across four distinct datasets for our later analysis. We leave the exploration of a better classification metric as future work.

heading: 4 Enhancing Multilingual Capability
text: This section conducts a comprehensive analysis of the properties and training strategies of each
quadrant to effectively enhance the multilingual capability of LLMs.

heading: 4.1 Reciprocal Quadrant
text: Language within the reciprocal quadrant indicates that using any of these languages as training data invariably improves performance in other languages within the same group. We will delve into this relationship to uncover some intriguing insights.
Interpretation: Reciprocal quadrant consists of linguistically similar languages. The reciprocal quadrant is predominantly occupied by IndoEuropean languages. These languages are grouped mainly due to their shared vocabulary and grammatical affixes. Furthermore, the original 20 languages supported by LLaMA are predominantly Indo-European, providing a solid foundation. Consequently, tuning one language within the IndoEuropean family can effectively enhance the performance of other languages within the same family.
Practice Guidance 1: The recommended training strategy for reciprocal languages is Embed FT, which achieves the best performancegeneralization trade-off. Figure 3 illustrates the performance disparity between the models obtained through Embed FT and Full FT strategies under varying amounts of training data. We randomly selected 11 languages from the reciprocal quadrant for testing, including es, pt, ca, de, da, cs, bg, pl,
fr, ru, nl, and averaged the bilingual/multilingual performance across all 11 languages.
For bilingual performance, the Embed FT strategy works as well as the Full FT strategy. As depicted by the bar in Figure 3, the results indicate that when working with a limited dataset, the model trained by Embed FT demonstrates a slightly inferior performance compared to Full FT. However, as the size of the dataset increases, the model developed using Embed FT not only matches but may even exceed the performance of Full FT.
For multilingual performance, the Embed FT strategy excels in adapting to various languages, while the Full FT strategy tends to make the model overly specialized to a particular language. As illustrated by the line in Figure 3, the findings suggest that full fine-tuning of a bilingual dataset may lead to overfitting, but this can be effectively mitigated by using the Embed FT strategy.

heading: Practice Guidance 2: While the Full FT model’s
text: multilingual capabilities are influenced by language quantity, the Embed FT model remains unaffected. Considering Phenomenon 3, which observes a consistent multilingual distribution, we are curious to explore whether a richer language number could bring additional performance gains. To investigate this, we randomly select some languages from the reciprocal quadrant to establish a multilingual setting, and the results of this experiment are displayed in Table 3. In the Full FT, the performance of the multilingual model improves with an increase in the number of languages. However, in the Embed FT, the number of languages does not have a significant impact.

heading: 4.2 Altruistic Quadrant
text: Languages that fall into this quadrant demonstrate a "selfless" characteristic. Training based on the data from these languages does not necessarily improve, and may even decrease their performance. Inter-
estingly, it can lead to performance enhancements in other languages. We will conduct a thorough examination of the underlying causes of this phenomenon and propose potential solutions.

heading: Interpretation for bilingual performance de-
text: cline: The model transitions from an error type that is easy to score to a less score-friendly error type. The primary error for LLaMA is “source copy”, which simply duplicates the source sentence as the translation. This error often leads to moderate scores when there are names, numbers, and punctuation in the translation tasks. However, after tuning, the main error shifts to “oscillatory hallucination” (Li et al., 2023), a state where the model becomes stuck in a specific translation state and generates repeated n-grams until it reaches the maximum length. This error makes it challenging to earn the score of spBLEU. Therefore, the performance of the fine-tuned model is lower than that of the original model.

heading: Interpretation for multilingual performance im-
text: provement: Those languages’ vocabulary encompasses the majority of English tokens. We estimate the linguistics of these languages on the Flores-101 benchmark, a multilingual parallel corpus translated by professional translators through a controlled process. For an altruistic language, LG we first employ LLaMA’s tokenizer to segment the words in both the LG and English data from Flores. This allows us to compile the sets of tokens that belong to the LG language, denoted as SLG, and the English language, denoted as SEn. Finally, we calculate the ratios of the size of SLG ⋂ SEn to the size of SLG and the size of SEn respectively. Intriguingly, as shown in Figure 4, we discovered that most tokenized results used in these languages exhibit a high degree of consistency with English.
Practice Guidance: Full FT with a minimal dataset can effectively enhance bilingual per-
formance and maintain a robust multilingual effect. As shown in Table 4, the altruistic trait is exemplified across different training strategies. However, with Full FT and LoRA, as the dataset size increases, the model tends to overfit the specific language, thereby diminishing its multilingual capabilities. For Embed FT, an increase in data volume does not significantly alter bilingual performance, but it does markedly enhance the multilingual effect. Interestingly, the multilingual effect is not significantly different from that of Full FT with a small dataset. In summary, by employing a small dataset for full fine-tuning, we can strike a balance between bilingual and multilingual performance.

heading: 4.3 Stagnant Quadrant
text: Languages in this quadrant exhibit remarkable inertia, as training with their data neither enhances their own performance nor influences the performance
of other languages. In this section, we will delve deeper into the inertia phenomenon, examining its potential causes and proposing possible solutions.
Interpretation: Most languages in the stagnant quadrant are characterized by overtokenization. The LLaMA tokenizer, based on the BBPE algorithm, is fundamental for multilingual language processing tasks. Its universal applicability to all languages and the lack of a need for an ‘unknown’ token make it optimal for vocabulary sharing and increase its robustness. Despite being suitable for multilingual learning, BBPE results in byte sequence representation of text that is often much longer (up to 4x) than a character sequence representation. Upon investigation, we find that the over-tokenization phenomenon is prevalent in LLaMA. In an extreme case, a sentence in lo that contains 6 words expands to 352 tokens after tokenization. Additional details in Appendix F.
A comparison between active and stagnant languages, as shown in Table 5, reveals that:
• activating a stagnant language with full finetuning requires more data;
• the performance improvement with increasing data is modest;
• certain parameter efficiency fine-tuning strategies, like LoRA, do not affect them.

heading: Practice Guidance 1: Expanding the vocabu-
text: lary is not an effective strategy for stagnant languages. Over-tokenization leads to an increased demand for data. When a language is not adequately represented by its vocabulary, the common approach is to expand the lexicon (Tai et al., 2020; Cui et al., 2023; Ji et al., 2023). Regrettably, in most instances, this strategy of vocabulary enlargement proves ineffective for stagnant languages. As shown in Table 6, we present three distinct methods to expand the vocabulary:
• BBPE (Wang et al., 2019): This follows the approach used in LLaMA for vocabulary construction and involves learning a vocabulary for stagnant language;
• BPE (Sennrich et al., 2016)): This utilizes the BPE algorithm and is based on subword units to learn a vocabulary;
• SP (Kudo and Richardson, 2018): This learns a vocabulary using the SentencePiece algorithm.
Meanwhile, to mitigate potential issues from data quality, we have utilized both MC4 and Flores101 dev to construct vocabulary.
After training LLaMA on Lego-MT 80k bilingual data, the experimental results indicate that:

heading: Source Type 3k 6k 12k Source Type 3k 6k 12k
text: • When there is a substantial amount of data, the impact of data quality on vocabulary expansion can be disregarded;
• Among all the vocabulary expansion methods, SP tends to yield better results compared to other solutions;
• Almost all vocabulary expansion techniques fail to enhance the performance of stagnant languages significantly.
Practice Guidance 2: Shortening the subword sequences can significantly boost the performance of stagnant languages. Given the existence of the over-tokenization problem, we find that among these over-tokenized languages, there are a
large amount characters. For example, a Chinese character “饕” is encoded into three code units “[227, 234, 260]”. We refer to such characters as ‘over-tokenized characters’ for the sake of simplicity. We then gather all these over-tokenized characters along with their three-byte representations. Interestingly, these over-tokenized characters constitute a significant proportion, about 63.8%, of the corpus, as indicated in Table 7. Moreover, in the case of over-tokenized languages, all over-tokenized characters begin with the same token (e.g., 227). Therefore, the obtained three-byte representations are very sparse and result in low information density in representation.
Furthermore, we propose a post-tokenization technique to address the over-tokenization problem. We simply remove the shared prefix of overtokenization characters and obtain the shortened yet lossless new representations. As a concrete example, we remove饕’s prefix [227] from its threebyte representation [227, 234, 260] to get a more compact two-byte representation [234, 260]. Subsequently, we utilized this adjusted representation to train LLaMA on the 80k Lego-MT bilingual dataset. Remarkably, our method outperforms both direct fine-tuning of LLaMA and vocabulary extension, achieving a substantial performance boost with an average of 2.5 points.

heading: 4.4 Guidance Summary
text: Guidance for reciprocal languages For languages, primarily Indo-European languages, situated in the reciprocal quadrant, the optimal strategy is to solely fine-tune the embedding layer and keep the remaining parameters frozen. This is primarily due to these languages having shared vocabulary and grammar rules.
Guidance for altruistic languages For languages residing in the altruistic quadrant, applying full fine-tuning with a minimal dataset can effec-
tively enhance bilingual performance while maintaining a robust multilingual effect. This is mainly because the tokens in the vocabulary of these languages highly overlap with English.
Guidance for stagnant languages Shortening subword sequences can markedly enhance the performance of stagnant languages. Most languages in the stagnant quadrant are over-tokenized, which refers to a situation where a text in this language is typically segmented by a tokenizer into an excessively large number of tokens on average. Expanding the vocabulary does not necessarily enhance the performance of these languages. However, in this paper, we demonstrate that simply removing the identical prefix from over-tokenized characters can significantly improve performance.
We provide more analysis about stagnant languages (in Appendix F), tuning analysis (in Appendix G), and different NLP tasks (in Appendix H) of LLaMA experiments in the Appendix.

heading: 5 Conclusion
text: In this study, we performed a comprehensive analysis of 101 languages, categorizing them based on shared characteristics into four distinct quadrants: reciprocal, altruistic, selfish, and stagnant quadrants. Upon examining each quadrant in-depth, we identified the primary reasons for the placement of languages within their respective quadrants and provided some practical guidance for training. However, the primary focus of this study is the analysis of persistent language characteristics within each quadrant. A thorough investigation into the conditions that trigger language migration across various phenomena is a subject for our future research.

heading: Limitation
text: In this paper, we find some interesting phenomena in LLaMA. After expanding our evaluation to include more languages, we found that many of them demonstrated remarkably similar behaviors on translation tasks. Then we grouped them with categorization criteria. Although language classification is not our primary focus, our main interest lies in understanding the reasons behind these classifications and enhancing the multilingual capabilities of LLM. Meanwhile, to delve deeper into the role of Embed FT, we provide a more detailed analysis in Appendix H.

heading: B BBPE
text: In a multilingual context encompassing a diverse range of scripts, the base vocabulary comprising subwords can become exceedingly large, leading to inefficiency and sparsity. To mitigate this problem, BBPE has emerged as the standard practice in most modern language modeling efforts (Muennighoff et al., 2022; Scao et al., 2022; Zhang et al., 2022; Touvron et al., 2023a,b), which leverages UTF-8 encoding that encodes each Unicode character into 1 to 4 one-byte (8-bit) code units. BBPE is a tokenization algorithm capable of tokenizing any word in any language, thereby eliminating the necessity for an unknown token. It optimizes vocabulary sharing across numerous languages and delivers superior performance, facilitating knowledge transfer between languages with non-overlapping character sets.

heading: C Language Information
text: In this section, we classify languages according to their respective language families, as depicted in Table 9. We standardize all language codes using the ISO 639-1 standard. For clarity, we list all languages by their full names and shade the corresponding languages in gray for easy identification.

heading: D Hyper-parameter Setting
text: We use the criteria to measure the bilingual/multilingual performance changes before and
after fine-tuning:
∆lg =
{ Ppost Ppre
− 2, if Ppre ≥ T Ppost−2T
Ppre , otherwise
The threshold term, T , is used to smooth the dramatic numerical change that might be caused by low-performing languages(e.g., performance change from 0.01 to 0.02, although negligible, will be considered significant without re-balancing using T ). We set T to the vanilla model’s average translation performance on the Flores-101 dataset.
The hyper-parameter, set to a value of 2, defines the thresholds for determining significant changes before and after tuning. Here, we consider a language to have significant bilingual/multilingual performance changes if the performance after tuning is twice that of the performance before tuning. In Figure 5, we have thoroughly tested different significance thresholds and found that if we consider a 20-fold difference (a very large value) in performance before and after tuning, then all languages would be regarded as stagnant languages.

heading: E Quadrant Division
text: We use some different publicly multilingual datasets: Lego-MT, Wikimatrix & Newcommentary, and Ted, which come from a different domain, as shown in Table 10.
Conducting a comprehensive evaluation of the translation performance for all en→x pairs in Flores-101 across all models is a task that demands
significant labor and resources. Therefore, we randomly select one representative language from each language family for subsequent testing, as shown in Table 11.
The bilingual and multilingual performance of the model trained on the TED dataset on Flores-101 devtest is shown in Table 12.

heading: Lang Language Family Lang Language Family

heading: F Stagnant Quadrant
text: The LLaMA tokenizer, built on the BBPE algorithm, serves as the foundation for multilingual language processing tasks. Its universal applicability across all languages, coupled with the elimination of the need for an “unknown” token, enhances
vocabulary sharing and boosts its robustness. However, a phenomenon known as over-tokenization, marked by excessive segmentation of text into tokens, may occur in certain languages, which could potentially affect the efficiency of language processing tasks.
To thoroughly examine the “over-tokenization”, we conduct our research using the MC4 (Xue et al., 2021) and Flores-101 (Goyal et al., 2022) dataset. Despite having only 1012 samples, Flores-101 provides a high-quality multilingual parallel corpus that allows for an in-depth exploration of the variations in expressing the same sentence across different languages.
The over-tokenization phenomenon is observable across various datasets and LLMs. For certain languages, such as te and lo, the length of the tokenized sequence that LLaMA processes can extend
to 300 or even more. Interestingly, analysis results from the Flores-101 dataset reveal that languages prone to over-tokenization require more tokens to express the same meaning. The magnitude of this phenomenon is notably larger than what was observed in the MC4 dataset, as shown in Figure 6.
We also present tuning results based on our analysis of the Flores-101 dataset, where we examined the effects of full bilingual fine-tuning and Lora tuning on varying amounts of data, as shown in Table 13. Interestingly, we found that the characteristics of stagnant language are preserved.

heading: G Single-layer Tuning
text: To determine whether fine-tuning parameters of layers other than the embedding layer in the model is equally effective, we conducted a bilingual translation task on eight language pairs in the Flores101
dataset. These models were fine-tuned on the Alpaca-En dataset, which was primarily used as
the training set to minimize any potential impact from language variations. The results of these tests
are displayed in Table 14. In these tests, English served as the source language, while the target languages comprised eight different languages.
As observed from the table, the average scores of fine-tuning the embedding layer and Layer 0 are the highest, and they are very close to each other. The model’s performance gradually decreases as the layer number increases, with a noticeable drop around the middle layers (Layers 15-17). This trend is remarkably consistent across all language pair tests.
The aforementioned results suggest that solely fine-tuning the parameters of the lower layers can also activate the model’s multilingual capabilities, and its effectiveness is comparable to that of embedding fine-tuning. Furthermore, the activation of different language capabilities in the model through single-layer fine-tuning occurs synchronously.
Additionally, we fine-tuned all the lower layers, from Layer 0 to Layer 14, together. As shown in Table 15, this strategy did not yield any additional gains compared to the other tuning strategies.

heading: H More Analysis
text: The performance of Embed FT remains stable across reciprocal languages, regardless of the dataset being utilized. As depicted in Table 16, the Embed FT strategy delivers performance that is competitive with the FT and LoRA strategies across all training sets: Alpaca-En, Alpaca-X, and Bilingual. Alpaca-En is a comprehensive English dataset with 52k instructions and demonstrations. Alpaca-X is derived from Alpaca-En through translation, with X denoting the target languages. The Bilingual dataset comprises 52k instruction data for translation tasks, based on the open-source LegoMT dataset. Unlike the FT strategy, which updates all model parameters. Furthermore, it avoids the need for an additional model structure, like the LoRA strategy. This implies that Embed FT is a more effective strategy for activating multilingual capabilities.
In the Flores-101 dataset, the same evaluation metric, spBLEU, is used. Before calculating BLEU, all data is de-tokenized and sentence piece tokenization is applied to each language. This allows for a more accurate assessment of model quality on the long tail of low-resource languages. NLU: We evaluate various tasks to test different aspects of the model. These include XCOPA (Ponti et al., 2020), a multilingual common reasoning
task supporting 11 languages; XStoryCloze (Lin et al., 2022a), a story completion task in 11 languages; XNLI (Conneau et al., 2018), a crosslingual natural language inference task for 15 languages; PAWS-X (Yang et al., 2019), a paraphrase identification task in 7 languages; and MGSM (Shi et al., 2022), a mathematical reasoning task in 11 languages. Besides fine-tuning the embedding layer, adjusting the lower layers can also be effective. To further investigate the functionality of the Embed FT strategy, we separately fine-tuned each layer of LLaMA using the Alpaca-En dataset and then tested on the Flores-101 en→ro devtest. The layers of the LLaMA model, excluding the embedding layer, are numbered from 0 to 31, with 0 being the closest to the embedding layer and 31 being the furthest. The bilingual performance of en→ro is illustrated in Table 16. Our experiments showed that fine-tuning the lower layers is just as effective as fine-tuning the embedding layer. However, we found that fine-tuning the higher layers did not produce satisfactory results.

heading: I Used Scientific Artifacts
text: Below lists scientific artifacts that are used in our work. For the sake of ethic, our use of these artifacts is consistent with their intended use.
• Stanford Alpaca (Apache-2.0 license), a project that aims to build and share an instruction-following LLaMA model.
• Lego-MT (MIT license), a dataset for machine translation.
• Transformers (Apache-2.0 license), a framework that provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.

heading: Acknowledge
text: This work is partially supported by the National Key R&D Program of China (NO.2022ZD0160100).

heading: Future Work
text: A thorough investigation into the conditions that trigger language migration across various phenomena is a subject for our future research