Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages (2024)

Abstract: While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages (English and French) across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages.1

heading: 1 Introduction
text: Contemporary large language models (LLMs; OpenAI, 2022, 2023; Touvron et al., 2023a,b; Jiang et al., 2023; Google et al., 2023) are predominantly trained on multilingual corpora. However, the language distribution in the data is highly imbalanced. For instance, LLMs like LLaMA-2 (Touvron et al., 2023b), with English as the primary language, have also been trained on Japanese text, yet the quantity
∗Corresponding authors. 1The source code is available at https://github.
com/THUNLP-MT/SDRRL.
of English tokens used during pre-training exceeds that of Japanese by a factor of 897.
The imbalanced data distribution above has led to significant limitations in the capabilities of LLMs across most languages. To enhance the multilingual capabilities, a common approach follows the translating and then supervised fine-tuning (SFT; Ouyang et al., 2022) paradigm, as shown in Figure 1(b). Specifically, training data is translated into the target language using either the model itself or an external machine translation (MT) system before continuing the training process, thereby offering more data in the target language and improving multilingual capabilities.
However, the translate-then-SFT method encoun-
11189
ters several challenges: First, the multilingual enhancement gained from translated “questionanswer” pairs is limited and may sometimes even degrade the capabilities in the original primary language (Zhu et al., 2024). Second, constrained by the accuracy of machine translation (especially for the low-resource languages), the translated texts used for training can be highly noisy, containing numerous awkward sentences and incorrect content, adversely affecting the quality of the generated text and the multilingual abilities of the LLMs. Therefore, we explore a new question along this trajectory: Besides translating the training pairs, can we enhance the abilities in other languages by leveraging the original relatively strong capabilities of LLMs in resource-rich language?
In this paper, we introduce SDRRL, a method that uses Self-Distillation from Resource-Rich Languages) to achieve the goal mentioned above. Specifically, as illustrated in Figure 1(c), SDRRL comprises two parts: (1) Self-Distillation: Instead of the ground-truth answer, responses from LLMs in resource-rich languages are collected to construct a transfer set. These are then translated into other languages using machine translation systems and code-switching tools, forming “questionanswer” pairs that are semantically identical but linguistically varied, and conducting sentence-level knowledge self-distillation within the same batch. (2) Incorporating External Parallel Corpus: We further involve a small amount of machine translation data in the distillation, aiming to align the linguistic representation spaces better and mitigate the negative impact of the noise in machine translation systems on the generative capabilities of LLMs.
Our experiments, based on LLaMA-2-7B (Touvron et al., 2023b) and SeaLLM-7B (Nguyen et al., 2023) with English as the resource-rich language, demonstrate that even with a smaller set of English instruction data as the transfer set, SDRRL can effectively distill English capabilities into 14 other languages, showing effectiveness in both multilingual comprehension and generation tasks. Further analysis indicates that SDRRL helps preserve the original capabilities in high-resource languages and improves the quality of generated responses.

heading: 2 Related Work
text: Multilingual Language Models. Using multilingual data during the pre-training is a common approach to enhance the multilingual capabilities of
LLMs (Li et al., 2022; Lample and Conneau, 2019; Workshop et al., 2022; Lin et al., 2022; Xue et al., 2021). Despite being pre-trained and fine-tuned targeting a few resource-rich languages, recent instruction-following LLMs (Touvron et al., 2023b; Jiang et al., 2023; Wang et al., 2023a) have been found to still possess significant multilingual understanding and generation capabilities (Bandarkar et al., 2023; Niklaus et al., 2023). However, limited by the imbalanced training data distribution (Yang et al., 2023), the multilingual capabilities of these popular LLMs lag behind those of languages with abundant resources (Pahune and Chandrasekharan, 2023).
Cross-Lingual Transfer. To enhance the capabilities in languages with scarce resources, one line of work is cross-lingual transfer, where skills learned from one source language can be readily transferred to other languages (Etxaniz et al., 2023; Huang et al., 2023; Ranaldi and Zanzotto, 2023). This has been approached by designing prompts that leverage LLMs to self-translate questions into resourcerich languages (Qin et al., 2023), or by utilizing external machine translation systems for assistance (Zhao et al., 2024). Efforts have also been made to distill synthetic data from high-resource languages to low-resource ones (Chai et al., 2024). Shaham et al. (2024) and Kew et al. (2023) leverage similarities between languages to stimulate capabilities in others. Compared to their work, we focus on proficiency in the resource-rich language and leverage it to improve performance in other languages.
Cross-Lingual Alignment. Another line of work is cross-lingual alignment (Schuster et al., 2019a). Given the scarcity of multilingual data, the construction of alignment data or loss functions of varying granularity can align mid- and low-resource languages with those that are resource-rich. This includes the construction of pre-training tasks using multilingual aligned lexicons (Chi et al., 2021), alignment of word embeddings (Wen-Yi and Mimno, 2023; Schuster et al., 2019b), using aligned data on one side of a problem to improve mathematical reasoning processes (Zhu et al., 2024), and encouraging language switching in chain-of-thought (CoT; Wei et al., 2022) reasoning (Chai et al., 2024). Mao and Yu (2024a) have leveraged the LLM’s own capabilities to generate aligned data, while others have constructed it with the aid of external systems (Ranaldi and Pucci, 2023; Chen et al., 2023a). Deriving and construct-
ing multilingual supervision signals from existing datasets overlooks the fact that the model’s own responses in high-resource languages can also serve as effective supervision signals. We show in our experiments that self-distillation not only improves the LLM’s multilingual performance but also helps maintain the performance in the original resourcerich languages. Knowledge Distillation. Knowledge distillation (Hinton et al., 2015) is a widely used method for transferring knowledge (Gou et al., 2021). In the text generation domain, sequence-level knowledge distillation (Kim and Rush, 2016) has been used as a means of data augmentation in areas such as machine translation (Gordon and Duh, 2019). In particular, self-distillation (Zhang et al., 2019, 2022b; Pham et al., 2022) is often utilized to distill knowledge from one component of a model to another (Zhang et al., 2022a), or from one stage of a model to another (Yang et al., 2019). In this work, we apply distilling knowledge between the different linguistic representation spaces within the same LLM to enhance multilingual capabilities.

heading: 3 Method
text: In this section, we first revisit the supervised finetuning (SFT) and translate-then-SFT paradigm, subsequently dividing the discussions into two parts of our proposed SDRRL. In the first part, we construct a transfer set using responses in the resource-rich language from LLMs through sentence-level self-distillation. In the second part, we employ parallel translation-based instruction data to further improve multilingual generation capabilities.

heading: 3.1 SFT and Translate-then-SFT Paradigm
text: We consider the given instruction dataset comprised of N entries D = {(xi,yi)}Ni=1, where xi symbolizes the input sentence (question) for the i-th data point, and yi signifies the corresponding ground-truth response (answer). Supervised Fine-Tuning. For a LLM Mθ parameterized by a set of parameters θ, which produces a response denoted as ŷ = Mθ(x) for the given input question x, the objective of SFT is to align the output sentence ŷ as closely as possible with the ground-truth response y. Specifically, the crossentropy (CE) loss is employed to assess the discrepancy between the model output ŷ and the groundtruth output y for a single sample (x,y), defined
as:
ℓCE(y, ŷ) = − |V|∑
j=1
yj log(ŷj) (1)
where yj is the one-hot encoding of the ground truth output y at position j, ŷj is the probability of the model output ŷ at position j, and |V| is the size of the vocabulary in the LLM.
For the entire dataset D, the total loss is calculated as the average of all sample losses:
LSFT = 1
N
N∑
i=1
ℓCE(yi,Mθ(xi)) (2)
Translate-then-SFT. For the translation-then-SFT paradigm, we define the machine translation system as a function T , which accepts text in one language as the source language (Src) and outputs equivalent text in the target language (Tgt). Using the machine translation system T , each pair (xi,yi) is translated into the target language, resulting in the translated dataset DMT = {(xMTi ,yMTi )}Ni=1 = {T (xi,yi)}Ni=1.
Similar to Eq. 1, the LLM Mθ is then trained on the translated dataset D′, where the loss for a single sample (xMT,yMT) is computed as:
ℓCE(y MT, ŷMT) = −
|V|∑
j=1
yMTj log(ŷ MT j ) (3)
where ŷMT = Mθ(xMT) is the response of models to the question xMT in target language.

heading: 3.2 Self-Distillation from Resource-Rich Languages (SDRRL)
text: LLMs exhibit superior comprehension and generation capabilities in resource-rich languages, which we suppose can be a learning reference for other languages to enhance the multilingual capabilities of LLMs. To achieve this, we propose sentencelevel knowledge distillation from resource-rich language responses. The core motivation is that the responses of LLMs in the resource-rich language serve as samples from the resource-rich language representation space. By adding these responses and their translations to the transfer set, the gap for cross-linguistic learning is reduced, facilitating the improvement of multilingual capabilities.

heading: 3.2.1 Transfer Set Construction
text: We construct a transfer set for sentence-level distillation by collecting LLM responses in the resourcerich language. For the original instruction dataset
D = {(xi,yi)}Ni=1, LLM Mθ generates responses for each question xi, yielding ŷi = Mθ(xi), then we get the generated dataset G = {(xi, ŷi)}Ni=1 = {(xi,Mθ(xi))}Ni=1. The synthesized transfer set Dsynth is obtained by equally probable random sampling from both datasets D and G:
Dsynth = Sample(D) ∪ Sample(G) (4)

heading: 3.2.2 Transfer Set Translation
text: The above constructed transfer set Dsynth contains question xi, ground-truth answer yi, and response ŷi by LLM Mθ. We consider translating them into the target language using the machine translation system T , resulting in xMTi = T (xi),yMTi = T (yi), and ŷMTi = T (ŷi). Moreover, we use WMT22-cometkiwi-da (Rei et al., 2022b) as a reference-free metric to assess the translation quality where the translation quality with scores below a threshold τ = 0.8 is rejected.
In particular, four sub-datasets are generated, each containing different language combinations of questions and responses:
• DLL: Both the questions and responses remain in the resource-rich language, i.e., {xi,yi} or {xi, ŷi}.
• DTL: The questions are translated into the target language, while responses remain in the resource-rich language, i.e., {T (xi),yi} or {T (xi), ŷi}.
• DLT: The questions remain in the resourcerich language, while responses are translated into the target language, i.e., {xi, T (yi)} or {xi, T (ŷi)}.
• DTT: Both the questions and responses are translated into the target language, i.e., {T (xi), T (yi)} or {T (xi), T (ŷi)}.
This approach, by providing semantically identical but linguistically diverse samples, aids in the implicit alignment of language representation spaces, enhancing unified multilingual performance. Furthermore, DTL and DLT enhance LLM’s crosslinguistic generative capabilities, helping mitigate off-target issues in target language generation.

heading: 3.2.3 Applying Code-Switching
text: Through the aforementioned machine translation process, we achieve alignment in sentence level (i.e., the sentence of question-answer pairs). Additionally, token-level alignment is introduced using
a code-switching tool, applied only to the question components xi of DLL, DTL, DLT, and DTT to increase language diversity without compromising generative capabilities.
Specifically, given xi composed of a sequence of tokens xi = xi,1, xi,2, . . . , xi,n, where xi,k denotes the k-th token in question xi (similarly for x̂MTi ), the code-switched version xi,k for each token is generated by applying the rule:
xi,k = { Dict(xi,k) with probability p; xi,k with probability 1− p,
(5) where each token xi,k in xi is replaced by its corresponding token in the bilingual dictionary for code-switching Dict(xi,k) with a p = 0.15 probability if xi,k is found in the bilingual dictionary. Responses, either in the source language yi (similarly for ŷi) or the target language yMTi (similarly for ŷMTi ), remain unchanged.

heading: 3.2.4 Incorporating External Parallel Corpus
text: The target language sequences ŷi synthesized by the external machine translation system T may contain low-quality translations, thereby introducing a significant amount of noise into the knowledge distillation transfer dataset Dsynth.To mitigate the impact of noise on the multilingual generative capabilities of LLMs, we leverage a tiny external parallel corpus P = {(si, ti)Li=1} between the resourcerich language Src and the target language Tgt. Based on the templates in Table 1, we can construct
two parts of instruction data: machine translation task instructions (Dmt) and sentence completion task instructions (Dcomp). By incorporating these two parts, the transfer set includes non-synthetic natural target language texts, which helps improve the generative quality of LLMs in the target language.

heading: 3.2.5 Training Objective
text: The final training dataset Dtrain includes DLL, DTL, DLT, DTT, Dmt, and Dcomp. The total loss function is defined as:
LSDRRL = ∑
d∈U
1 |Dd| ∑
{x,y}∈Dd ℓCE(Mθ(x),y),
(6) where U = {LL,TL,LT,TT,mt, comp} and Dd corresponds to the respective data subset (e.g., DLL, DTL, etc.).

heading: 4 Experiments

heading: 4.1 Setup
text: We use LLaMA-2-7B (Touvron et al., 2023b) as the base model. Drawing from the distribution of language in pre-training corpus, we use English (ENG) as a resource-rich language and conduct experiments on 14 languages: Czech (CES), Danish (DAN), Ukrainian (UKR), Bulgarian (BUL), Finnish (FIN), Hungarian (HUN), Norwegian (NOB), Indonesian (IND), Japanese (JPN), Korean (KOR), Portuguese (POR), Slovenian (SLV), Vietnamese (VIE), and Polish (POL). Stanford Alpaca instruction data (Taori et al., 2023) serve as the base of the transfer set D, providing questions and ground-truth answers in English. For machine translation, we utilize open-source NLLB200-3.3B (Costa-jussà et al., 2022) model. To improve translation quality, we follow Zeng et al. (2021) to filter low-quality translations and use CLD3 (Ooms, 2024) model to remove off-target translations. We also follow Lin et al. (2021) to construct bilingual dictionaries for code-switching. See appendix A for more details.
Implementation Details Our code is implemented using DeepSpeed (Rasley et al., 2020) on eight NVIDIA A800-SXM4-80GB GPUs. Following Wang et al. (2023a), we set the training duration to four epochs with an automatically calculated learning rate and employ early stopping. Other hyperparameters are set according to Hiyouga (2023).
Baselines. For comparison, we consider the following baseline systems that enhance LLaMA-2’s multilingual capabilities using different instruction fine-tuning methods:
• SFT (Ouyang et al., 2022): It only involves English instruction datasets in the process of fine-tuning, which is not multilingualoriented.
• Translate-then-SFT (Chen et al., 2023a, TSFT): It uses an external machine translation system to translate English instruction data into non-English languages and construct multilingual data for instruction fine-tuning.
• Cross-Lingual Instruction Tuning (CIT; Li et al., 2023a): It constructs cross-lingual instructions for fine-tuning, imposing models to respond in the target language given the source language as context.
• Cross-Lingual Chain-of-Thought Reasoning (XCOT; Chai et al., 2024): It applies codeswitching to multilingual instruction training data, using high-resource instruction data to supervise the training of low-resource languages with cross-lingual distillation.
Datasets. We evaluate the multilingual capabilities of LLMs on four representative datasets:
• BELEBELE (Bandarkar et al., 2023): A widely used language understanding dataset covering 122 languages, where each question, linked to a short passage, has four multiplechoice answers. This dataset proves challenging for state-of-the-art LLMs. Accuracy is reported in our experiments.
• FLORES (Goyal et al., 2022): A benchmark for machine translation with parallel text from Wikipedia for 204 languages, making up over 40,000 directions. We evaluate the bidirectional translation results between the target language and English, reporting scores using SacreBLEU (Post, 2018) and COMET score using WMT22-comet-da model (Rei et al., 2022a).
• XL-SUM (Hasan et al., 2021): A multilingual abstractive summarization benchmark for 44 languages, comprising multiple long news texts requiring summarization into a single sentence. ROUGE-1 and ROUGE-L F1 scores are reported.
• MKQA (Longpre et al., 2020): An opendomain question-answering dataset across 26 diverse languages, providing multiple possible short answers as ground truth for each question. We use the official evaluation script and report token overlapped F1 scores.

heading: 4.2 Main Results
text: Table 2 shows the experimental results of the multilingual understanding task. Tables 3, Table 4 and Table 5 show the results on multilingual generation tasks. From the experimental results, we can observe that:
(1) SDRRL effectively enhances performance in the target languages. Specifically, in every comprehension and generation task, our method surpasses the baselines in almost all target languages. As shown in Table 2, SDRRL improves [erformane in comprehension tasks by approximately +1.5 BLEU score. On the Flores dataset, SDRRL yields up to about +6.0 BLEU score improvement in both directions and about +4.0 COMET score improvement (Table 3). This demonstrates that using proficient responses in resource-rich languages as supervisory signals for knowledge distillation significantly enhances performance in other target languages.
(2) SDRRL exhibits stronger robustness in generation tasks. For example, on the XL-SUM dataset (Table 4), which requires the generation of longer texts, the average performance of CIT and XCOT decreased due to the quality of machinetranslated texts and pipeline noise, yet SDRRL still achieved about +0.55 ROUGE-L F1 score improvement. On the FLORES dataset (Table 3), which requires cross-lingual text generation, T-SFT and CIT lead to decrease of -1.36 and -2.08 BLEU scores, respectively, while our method improves by +5.88 BLEU scores. This suggests that adding machine-translated data constructed instructions to the self-distillation process effectively improves multilingual generation and mitigates the negative impact of low-quality translated texts.
(3) SDRRL can maintain the original strong capabilities in English. The results show that it is more challenging to retain the original English capabilities for languages with unique alphabets (e.g., Japanese and Korean). For example, in the Japanese comprehension task (Table 2), all baseline methods lead to a performance drop in English compared to SFT, while only our method success-
fully preserving the original English capabilities.

heading: 4.3 Ablation Study
text: We further investigate the effectiveness of each component of our proposed SDRRL. The results are shown in Table 6, where average scores on natural language understanding and generation tasks are reported. Our observations include:
(1) Rows 1 to 5 demonstrate that removing any single component results in performance degradation, affirming the necessity and efficacy of each component in SDRRL.
(2) Insights from row 3 suggest a significant performance decline in both target languages and English when model-generated responses (ŷi) are removed from Dsynth, highlighting the effectiveness of utilizing responses in resource-rich languages as additional supervision signals for improving multilingual capabilities. Moreover, row 2 indicates that substituting sentences with their semantic counterparts in different languages also contributes to multilingual performance improvement.
(3) Row 4 and 5 reveal that Dmt, Dcomp, and code-switching provide a limited amount of ground truth. This additional supervisory signal is beneficial for generative tasks and helps improve the quality of responses.
(4) Despite introducing a small amount of parallel data through Dmt and Dcomp, as shown in row 6, relying solely on these additional data for LLM training supervision leads to severe performance degradation. Compared to row 4, this indicates that these data do not inherently bring positive performance gains but are used to mitigate the deterioration of the LLM’s multilingual generative representation space caused by noisy machinetranslated text, serving as a regularization mechanism in knowledge distillation.

heading: 4.4 Visualization of Representation Space for Source and Target Langauges
text: We visualize the sentence representations of input instructions to investigate the effect of SDRRL on the multilingual representation space. Following common practices in sequence classification Li et al. (2023c), we input instructions into the LLaMA-2 and use the last hidden states of the last token as the vector representation of the sentence. We then apply t-SNE (Van der Maaten and Hinton, 2008) to reduce the 4096-dim representations to 2-dim for visualization.
CES DAN UKR BUL FIN HUN NOB IND JPN KOR POR SLV VIE POL AVG.
Performance on Target Language SFT 49.33 48.33 46.67 49.11 39.78 43.22 49.22 46.15 42.01 37.99 55.98 42.79 42.91 44.69 45.58 T-SFT 48.22 51.67 47.11 51.22 47.11 45.67 51.33 49.72 41.56 43.69 56.20 46.03 47.60 48.72 48.28 CIT 50.11 53.44 47.22 51.44 48.00 45.67 53.33 49.94 43.24 46.26 56.65 46.70 45.59 49.72 49.09 XCOT 51.56 54.22 47.83 52.78 47.00 45.67 51.33 49.16 43.02 46.15 56.42 46.48 46.82 48.49 49.07 SDRRL 52.11 55.00 48.33 54.00 49.56 46.44 53.89 52.40 45.81 46.82 57.88 47.26 48.38 50.17 50.58
Performance on English Language SFT 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 65.39 T-SFT 63.91 65.25 66.03 65.25 65.70 65.36 65.25 65.70 61.01 60.45 63.80 65.47 65.47 65.92 64.61 CIT 63.46 65.47 65.59 64.02 61.23 63.46 64.13 65.92 62.01 63.46 64.02 63.24 62.91 62.91 63.70 XCOT 65.70 65.47 66.15 66.48 65.81 65.70 66.55 64.92 63.24 65.43 62.46 66.50 63.91 66.37 65.34 SDRRL 66.26 65.70 67.15 66.53 65.92 66.70 66.59 67.15 65.13 65.45 66.48 66.59 65.57 66.82 66.29
BLEU scores on English-to-X Tasks SFT 13.00 21.91 11.18 12.98 8.39 9.07 18.53 34.54 17.03 18.15 43.06 28.46 25.06 27.65 20.64 T-SFT 22.68 27.78 23.11 27.59 15.31 16.96 25.60 31.79 19.52 18.11 39.75 26.17 25.09 26.04 24.68 CIT 22.03 28.57 19.92 26.85 14.54 17.46 25.97 29.46 13.81 15.33 35.24 22.60 22.33 22.84 22.64 XCOT 23.11 32.20 21.97 27.33 15.80 17.38 25.96 30.33 9.31 15.13 38.04 26.56 25.43 26.03 23.90 SDRRL 27.91 39.00 27.25 33.93 20.88 22.09 29.64 35.32 20.51 20.47 43.36 30.09 29.87 27.86 29.16
COMET scores on X-to-English Tasks SFT 85.35 87.60 84.58 84.97 85.69 84.40 86.35 73.54 63.41 45.44 78.91 80.98 63.43 68.46 76.65 T-SFT 84.71 84.26 83.33 83.82 83.78 82.02 83.31 78.94 78.39 72.95 80.38 81.82 73.81 79.06 80.76 CIT 81.71 82.84 80.06 81.72 82.14 82.29 83.37 84.62 76.16 73.88 83.71 76.38 80.60 78.97 80.60 XCOT 84.40 84.47 83.11 83.90 84.67 81.96 84.68 83.50 78.83 75.66 83.23 76.48 79.46 78.75 81.65 SDRRL 86.04 88.51 84.82 86.08 86.98 85.70 87.15 89.46 83.33 79.02 85.15 84.02 81.43 83.08 85.06
COMET scores on English-to-X Tasks SFT 57.19 70.93 55.25 54.99 60.29 53.94 71.97 83.82 82.46 82.14 84.57 55.96 80.78 82.14 69.75 T-SFT 78.94 81.34 79.92 81.43 78.53 76.01 82.69 84.90 82.76 80.58 86.42 69.06 81.62 82.86 80.50 CIT 79.87 82.47 78.63 81.70 78.39 76.18 83.19 84.18 78.15 78.45 85.12 80.12 80.77 80.17 80.53 XCOT 79.22 83.29 79.16 80.86 78.63 75.51 83.30 84.68 74.27 77.31 86.01 78.65 82.24 82.72 80.42 SDRRL 84.29 86.91 83.51 85.40 84.62 81.06 85.55 86.00 83.65 82.66 87.64 82.63 83.93 83.61 84.39
Table 3: Results of baselines and our SDRRL on FLORES benchmark.
As shown in Figure 2, after applying SDRRL, the representations of semantically equivalent instructions in the source and target languages are drawn closer together. This implies that SDRRL has improved the multilingual representation space by aligning the representation space of the target language closer to that of the resource-rich, bettermodeled source language, thereby enhancing the performance in target languages.

heading: 4.5 SeaLLM as Different Backbone Model
text: By using the responses of LLMs in high-resource languages as the supervisory signal for knowledge
IND JPN KOR POR VIE UKR AVG.
Performance on Target Language (ROUGE-1) SFT 20.82 6.17 0.66 23.38 9.30 8.10 11.41 T-SFT 25.61 32.11 7.67 26.68 20.59 14.19 21.14 CIT 24.64 16.11 5.80 26.33 20.55 11.21 17.44 XCOT 22.55 32.39 7.26 26.21 19.84 13.38 20.27 SDRRL 26.08 33.15 8.18 27.40 20.98 14.35 21.69
Performance on Target Language (ROUGE-L) SFT 16.03 4.13 0.61 15.84 7.21 6.72 8.42 T-SFT 20.15 22.83 6.93 18.41 15.18 11.73 15.87 CIT 19.02 11.22 5.14 18.06 14.91 9.00 12.89 XCOT 17.19 22.32 6.52 18.05 14.57 10.78 14.91 SDRRL 20.47 22.81 7.35 19.09 15.52 11.84 16.18
Performance on English Language (ROUGE-1) SFT 26.35 26.35 26.35 26.35 26.35 26.35 26.35 T-SFT 27.49 26.89 26.68 27.28 26.42 26.75 26.92 CIT 27.84 27.40 26.57 27.39 27.17 24.83 26.87 XCOT 26.44 25.45 25.43 26.78 26.00 25.90 26.00 SDRRL 28.18 27.73 27.44 27.57 27.52 27.23 27.61
Performance on English Language (ROUGE-L) SFT 18.68 18.68 18.68 18.68 18.68 18.68 18.68 T-SFT 19.64 19.11 18.94 19.54 18.73 19.01 19.16 CIT 19.93 19.56 18.81 19.56 19.34 17.43 19.11 XCOT 18.63 17.83 17.86 18.91 18.29 18.15 18.28 SDRRL 20.25 19.88 19.56 19.69 19.66 19.44 19.75
Performance on English Language SFT 41.62 41.62 41.62 41.62 41.62 41.62 41.62 41.62 41.62 41.62 T-SFT 44.92 42.63 44.24 44.21 41.65 42.11 42.63 42.65 42.81 43.09 CIT 44.09 43.86 43.55 44.12 42.83 43.29 42.51 42.52 43.41 43.35 XCOT 43.23 43.16 43.53 43.06 42.59 42.58 43.39 42.58 43.29 43.05 SDRRL 45.42 45.33 45.47 44.78 43.26 43.58 43.99 45.77 44.71 44.70
distillation, SDRRL is applicable to various LLMs, not limited to LLaMA-2. In this part, we conduct
experiments on SeaLLM-7B (Nguyen et al., 2023), a specialized language model optimized for Southeast Asian languages.
As shown in Table 7, SDRRL results in an improvement of +2.39 average scores on the target languages. In English, SDRRL maintains its original performance, while the baselines exhibit a performance drop of at least -2.02 average scores compared to vanilla SFT. This demonstrates the generalizability of SDRRL in different LLMs. See appendix B for detailed results on more datasets.

heading: 4.6 Further Analysis
text: Non-English Source Languages. SDRRL is also capable of transferring multilingual performance using other source languages in highresource. In appendix C, we opt for experiments with French instead of English. Experimental results reveal that, despite the LLM and the machine translation system exhibiting stronger performance in English, SDRRL still achieves positive distillation gains with French as the source language.
Case Study. In appendix D, we provide several case studies to offer deeper insights into the impact of SDRRL on the generation capabilities of LLMs. It is observed that the SDRRL process is able to alleviate off-target issues in the target language, reduce grammatical errors and hallucinations, and enhance the fluency of the output text.

heading: 5 Conclusion and Future Work
text: We introduce Self-Distillation from Resource-Rich Languages (SDRRL) to enhance the multilingual capabilities of LLMs. SDRRL uses the model itself to generate high-quality responses in resource-
rich source languages and their target language counterparts as supervision signals for knowledge distillation, aiming to align additional target languages with resource-rich languages. We conduct comprehensive experiments across 16 languages on LLaMA-2 and SeaLLM. The results demonstrate that, compared to various baselines, our method significantly enhances the performance of target languages while preserving the capabilities of source languages. This highlights the multilingual potential of LLMs and illuminates paths for further research into multilingual LLMs.
Limitations
Firstly, within our method pipeline, some components are interchangeable. For example, our approach relies on external machine translation systems to provide translations in the target language, while future research could explore self-translation with LLMs that achieve great low-resource translation capabilities, thereby simplifying the process. Additionally, our method uses a small amount of machine-translated parallel corpus to construct the transfer set, but employing monolingual texts in the target language represents a promising research direction. Secondly, our experiments are conducted with only a single source language and target language. Subsequent research could investigate using a mix of multiple languages as both source and target languages and explore the mutual influences between different languages to further enhance the multilingual capabilities of LLMs. Thirdly, our method does not involve engineering on the architecture of LLMs. For specific extremely lowresource languages, modifying the architecture and introducing additional data, such as vocabulary expansion or continuing pre-training, might be beneficial in enhancing multilingual performance.

heading: Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (No. 61925601, 62276152, 62236011).

heading: B More Detailed Results on SeaLLM
text: We conduct experiments on three common Southeast Asian languages: Indonesian (IND), Thai (THA), and Khmer (KHM). As shown in Table 8, 9, 10, and 11, SDRRL still outperforms the baselines, demonstrating the generalizability of SDRRL in different LLMs.

heading: C Experiments with Non-English Language as the Source Language
text: SDRRL aims to transfer the proficiency of LLMs from resource-rich languages to another target language, indicating that the source language is not limited to English. In this section, we use French instead of English to generate responses and replicate experiments conducted in Indonesian, Japanese, and Korean languages. Table 12 presents the average knowledge distillation gains from SDRRL over SFT when employing English and French as the source languages, respectively. Due to the inferior performance of LLaMA-2-7B in French (Ban-
darkar et al., 2023) compared to English and the decreased effectiveness of the external translation system NLLB when French is used as the source lan-
guage (Costa-jussà et al., 2022), the knowledge distillation gains generated with French as the source language are smaller.

heading: D Case Study
text: As shown in Table 13, we provide several cases of SDRRL in several languages. In the case of Hungarian, LLM encounters severe off-target issues, where the response is in English and is inconsistent with the input language Hungarian, which is a frequent problem during multilingual generation. After SDRRL, the severe off-target issue has been effectively mitigated. In the Slovenian case, LLM produces hallucinations when answering factual questions about history, leading to factual inaccuracies, noise tokens like emojis, and off-target English phrases. For example, The time frame mentioned (1685-1700) and the description does not match any king named Charles. It seems there’s a mix-up with historical figures. After SDRRL, the hallucination issue has been alleviated, and the generated content becomes more detailed, refined, and fluid. In the Bulgarian scenario, the response contains several grammatical errors, such as “ледников”, “оттикват” and “снижаваниеa”. In this case, the SDRRL process enhances the clarity and natural flow of the output text while also eliminating grammatical errors in Bulgarian. See appendix E for statistical results regarding off-target issues.

heading: E Off-Target Issue Analysis
text: We delve deeper into the effectiveness of SDRRL in alleviating off-target language issues during LLM responses. We evaluate the responses of LLaMA-2 on Dolly (Conover et al., 2023) and its multilingual extension, Bactrain-X (Li et al., 2023b). As
depicted in Figure 3, we showcase the variation in the off-target occurrence rates across each target language throughout the SDRRL process. This indicates that SDRRL plays a constructive role in mitigating off-target issues, ensuring consistency between the input and the response languages.

heading: F Results on Bigger Models
text: As shown in Table 15 and Table 14, we also provide experiments on LLaMA-2-13B as the base model. The results show that our method still outperforms the baselines.

heading: G Potential Risks of Our Method
text: Because our method involves distilling knowledge from other target languages towards high-resource languages to achieve cross-linguistic alignment, it may lead to cultural unfairness for mid- and lowresource languages. For instance, after aligning to English using SDRRL, responses of LLMs in African languages may also adhere to the cultural practices and social norms of English.
CES DAN UKR BUL FIN HUN NOB IND JPN KOR POR SLV VIE POL AVG.
BLEU scores on X-to-English Tasks SFT 37.16 45.16 35.49 37.01 30.10 30.69 40.30 36.52 21.45 20.54 46.02 31.52 29.26 29.01 33.59 CIT 0.72 0.71 0.03 0.22 0.48 0.70 1.11 0.50 0.11 0.09 0.11 1.53 0.79 0.63 0.55 XCOT 34.56 39.54 24.99 26.47 19.30 22.80 35.16 34.52 3.22 0.32 44.16 28.17 16.37 21.63 25.09 SDRRL 37.59 46.96 36.24 38.61 32.45 32.75 40.75 39.60 25.87 25.72 47.09 32.89 33.33 29.94 35.70
BLEU scores on English-to-X Tasks SFT 16.79 25.65 17.99 17.88 12.85 11.96 21.80 19.89 4.63 10.24 35.34 13.11 16.55 14.86 17.11 CIT 1.49 1.68 0.77 1.69 1.10 0.73 0.93 1.93 0.53 0.63 0.53 2.01 1.06 1.20 1.16 XCOT 27.27 36.34 27.61 33.98 20.44 22.28 30.64 31.07 11.92 7.70 40.59 22.66 30.14 21.05 25.98 SDRRL 29.11 40.36 28.80 35.50 23.36 24.34 30.30 36.69 16.76 20.10 44.85 25.43 33.40 23.00 29.43
COMET scores on X-to-English Tasks SFT 86.48 88.47 85.18 85.97 87.10 85.38 87.12 87.00 83.53 83.35 88.19 84.26 84.06 84.27 85.74 CIT 40.48 39.35 37.61 40.06 40.26 39.07 38.47 40.17 38.29 39.44 38.29 43.38 40.44 40.63 39.71 XCOT 85.45 87.54 80.36 81.77 81.47 81.87 85.91 86.21 65.16 64.16 87.76 83.13 77.52 81.44 80.70 SDRRL 86.80 89.01 85.58 86.70 87.93 86.61 87.29 87.70 85.35 87.57 88.29 85.45 85.46 84.24 86.71
COMET scores on English-to-X Tasks SFT 64.15 75.81 66.76 61.55 68.78 59.98 77.36 76.74 51.96 71.10 81.47 60.61 68.20 65.73 67.87 CIT 42.57 39.78 41.29 42.97 44.83 40.12 38.84 42.48 43.49 43.52 43.49 44.23 39.69 42.66 42.14 XCOT 82.80 86.46 83.79 85.29 83.65 80.91 86.80 85.10 78.99 77.15 86.48 80.57 82.62 80.66 82.95 SDRRL 86.05 88.45 85.68 86.63 87.11 84.10 86.79 88.60 83.83 87.57 87.67 83.56 85.78 85.02 86.20
Table 15: Results of baselines and our SDRRL on FLORES benchmark.

heading: Future Work
text: For example, our approach relies on external machine translation systems to provide translations in the target language, while future research could explore self-translation with LLMs that achieve great low-resource translation capabilities, thereby simplifying the process. Additionally, our method uses a small amount of machine-translated parallel corpus to construct the transfer set, but employing monolingual texts in the target language represents a promising research direction. Secondly, our experiments are conducted with only a single source language and target language. Subsequent research could investigate using a mix of multiple languages as both source and target languages and explore the mutual influences between different languages to further enhance the multilingual capabilities of LLMs. Thirdly, our method does not involve engineering on the architecture of LLMs. For specific extremely lowresource languages, modifying the architecture and introducing additional data, such as vocabulary expansion or continuing pre-training, might be beneficial in enhancing multilingual performance