X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024)

Abstract: Large language models respond well in highresource languages like English but struggle in low-resource languages. It may arise from the lack of high-quality instruction following data in these languages. Directly translating English samples into these languages can be a solution but unreliable, leading to responses with translation errors and lacking languagespecific or cultural knowledge. To address this issue, we propose a novel method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. Specifically, the language model first learns to generate appropriate English instructions according to the natural web texts in other languages as responses. The candidate cross-lingual instruction tuning samples are further refined and diversified. We have employed this method to build a large-scale cross-lingual instruction tuning dataset on 10 languages, namely X-Instruction. The instruction data built using our method incorporate more language-specific knowledge compared with the naive translation method. Experimental results have shown that the response quality of the model tuned on X-Instruction greatly exceeds the model distilled from a powerful teacher model, reaching or even surpassing the ones of ChatGPT. In addition, we find that models tuned on cross-lingual instruction following samples can follow the instruction in the output language without further tuning. 1

heading: 1 Introduction
text: Large language models illustrate a series of remarkable abilities, e.g., generating better response and zero-shot task generalization, after tuning on instruction following samples (Ouyang et al., 2022a; Wei et al., 2022; Mishra et al., 2022). However,
†These authors contributed equally to this work. *Corresponding author. 1Our code and data are available at https://github.com/
ZNLP/X-Instruction
the responses in low-resource languages, which are prone to contain unsafe content (Yong et al., 2023) and ignore cultural nuances (Liu et al., 2023a), are inferior to the ones in high-resource languages. In addition to the limited pre-training corpus in these languages, it may come from the lack of highquality instruction following data.
Distillation from the teacher model or manual annotation are common methods to obtain highquality instructions. The former cannot be applied to low-resource languages since the teacher model may respond poorly in these languages. Considering the high requirements, including creative thinking and professional knowledge, to write instruction-following samples, it is hard to find suitable annotators that excel in low-resource languages. Translating English instruction tuning samples into the target language is also unreliable, which will introduce translation errors, especially in the samples with codes, and overlook the cul-
546
tural nuances between languages. As shown in Figure 1(a), the Urdu sample translated is wrong due to the pronunciation nuance between English and Urdu. Therefore, aligning language models in low-resource languages remains a significant challenge.
In the preliminary experiment, we found that LLaMA (Touvron et al., 2023a,b) can understand unseen low-resource language instructions quite well through responding in English, when tuned with limited instruction-following samples, but it performs much worse when replying in its own language (Table 1).
Based on the observation, we first adopt the cross-lingual instruction following format like Figure 1(b), in which the languages of instructions and responses are not the same, to exploit the understanding ability rapidly learned in low-resource languages and better generation performance in English. We further propose an automatic pipeline to excavate and refine cross-lingual instruction tuning data from the unsupervised multilingual corpus, which contains language-specific knowledge and native expressions. Specifically, the model first learns from seed data to generate appropriate English instructions for the given texts in other languages, which compose candidate cross-lingual instruction following samples. The candidate samples are refined in an iterative manner. In each iteration, an evaluator is trained through a synthetic rating dataset and finds better cross-lingual samples, which are exploited to improve the evaluator in the next iteration. Thus, the quality of samples found is improved with the stronger evaluator. The final cross-lingual instruction tuning dataset is sampled from the different clusters of the remaining data after k-th iteration to increase the diversity.
We conducted experiments on 10 languages, which include 5 medium-resource languages and 5 low-resource languages2, and generated 320k highquality cross-lingual instruction tuning samples. On four instruction-following evaluation benchmarks, the response quality of the base model tuning on our data can surpass those generated from translation and distillation baseline models, and even exceed the ones of ChatGPT (OpenAI, 2022) in these languages. In addition, we find that crosslingual instruction tuning models can follow the instruction in the language of output without fur-
2We follow the language category used in Lin et al. (2022) sorted by the data ratios in the CC100 corpus (Conneau et al., 2020; Wenzek et al., 2020).
ther tuning and maintain 90.6% response quality. To sum up, our contributions are as follows:
• We propose an automatic pipeline to excavate cross-lingual instruction-tuning samples with language-specific knowledge.
• We have built and will release a high-quality cross-lingual instruction-tuning dataset for 10 languages, including 5 low-resource languages. Experimental results demonstrate that models tuning on it can generate better responses in low-resource languages.
• We find that cross-lingual instruction tuning models obtain a zero-shot instruction following ability in the output language.

heading: 2 Related Works
text: Our work is related to multilingual instruction generation and tuning, which will be briefly introduced below.
Instruction Generation Large language models obtain instruction following and better zero-shot task generalization ability after tuning on diverse instruction following samples (Wei et al., 2022; Mishra et al., 2022; Chung et al., 2022). The key challenge is how to build high-quality instruction following samples that have a great influence on the performance of the model after tuning. Distilling from more powerful models (Wang et al., 2023; Taori et al., 2023; Xu et al., 2023) and human labeling (Köpf et al., 2023; Zhou et al., 2023) are main methods to generate high-quality instruction following samples. Our work is more similar to the self-align methods (Bai et al., 2022; Sun et al., 2023; Li et al., 2023c), which show promising results by iteratively self-generating and filtering instructions without strong models and a lot of manual annotation. However, it cannot be applied to the condition when language models are
unable to generate high-quality instructions in unseen languages. In contrast, our method avoids the weakness in the low-resource language generation and exploits superior generation ability in English.
Multilingual Instruction Tuning After tuning on large-scale multilingual instruction following samples translated from English, large language models show better zero-shot multilingual performance and language generalization results (Muennighoff et al., 2023; Li et al., 2023a). On the other hand, Li et al. (2023b) translate 67k instructions only and distill language-specific outputs from ChatGPT, which shows better responses and multilingual abilities than the ones tuning on translated samples.
Different from distilling language-specific outputs from more powerful models, our method attempts to excavate high-quality samples from the neural multilingual corpus and is applicable to languages that are not supported by teacher models.

heading: 3 Method
text: As shown in Figure 2, we design a three-step pipeline to automatically excavate high-quality cross-lingual instruction following samples from multilingual web corpora: 1) The base language model is fine-tuned to generate candidate English instructions given multilingual corpus (Section 3.2). 2) Samples with inappropriate cross-lingual instruction and output are discarded to improve the quality of the dataset (Section 3.3). 3) We diversify the cross-lingual instruction data by sampling from the multiple clusters of the English instruction embedding (Section 3.4).

heading: 3.1 X-Instruction Definition
text: The instruction following sample consists of an instruction i, an optional input x, and an output y in the same language. Considering the vague boundary between instruction and input, the input x is ignored in this work for simplicity. We define the Cross-lingual Instruction (X-Instruction) sample (ia, yb) that the language a of instruction is different from the one of output (b). Taking the cross-lingual instruction sample in Figure 1(b) as an example, the language of instruction is English, while the language of output is Urdu.

heading: 3.2 X-Instruction Generation
text: To exploit the better generation performance in high-resource languages like English, we generate
English instructions for a corpus in other languages. Specifically, given seed cross-lingual instruction samples D0 = {(ihj , ylj)}nsj=1, where the h and l denote the high- and low-resource languages respectively, the language model is fine-tuned to generate the instruction ihj given the response y l j as input. Then, we use it to generate candidate cross-lingual instructions for the multilingual corpus, basing the assumption that some texts among the corpus are good responses in X-Instruction samples.

heading: 3.3 X-Instruction Refinement
text: After generating cross-lingual instruction samples, it is important to find and discard the inappropriate ones due to the quality of instruction following samples having a great influence on the performance of model aligned (Chen et al., 2023; Zhou et al., 2023). To achieve this, we design an iterative method named “X-Instruction Refinement”, which is shown in Figure 2(II).
In the k-th iteration, we first synthesize a pseudorating dataset Drk = {(ihj , ŷj , sj)}nrj=1 from a part of seed data Dr0, where Dr0 ⊂ D0, to train the evaluator that outputs three-level ratings. Given the instruction ihj , the best response (sj = 2) is the vanilla output ylj , while the worst one (sj = 0) is selected from the mismatched output ylm (m ̸= j). The reasonable but flawed output (sj = 1) is chosen from the modified output ŷlj with some parts deleted or duplicated in the vanilla output ylj , or the output of the cross-lingual instruction-following model tuned on Dxk−1. In the first iteration (k = 1), the seed data Dx0 = D0 \ Dr0, which is not used in the pseudo-rating dataset Dr0, is adopted to finetune the cross-lingual instruction following model. In the subsequent iteration (k ≥2), Dxk−1 consists of the best cross-lingual instruction samples found by the evaluator and Dx0 .
As the iteration proceeds, the quality of output from the instruction following model improves by tuning on higher quality X-Instruction samples, which reduces the gap between the second-level sample and the highest one in the pseudo-rating dataset. Thus, the trained evaluator can find better cross-lingual instruction tuning samples. The number of iterations is set to 3 by default, which is investigated in Section 5.4.1.

heading: 3.4 X-Instruction Diversification
text: In addition to the quality, the diversity of instruction tuning samples also has a great impact on the
performance (Wan et al., 2023). To diversify XInstruction samples, we first obtain the embedding for each instruction ih using a pre-trained sentence encoder. After applying k-means on these embeddings, the same amount of X-Instruction samples are sampled from each cluster (Figure 2(III)). It aims to diversify cross-lingual instruction examples by avoiding the dominance of samples from a few domains in the final dataset.

heading: 4 X-Instruction Dataset
text: We apply the automatic pipeline to a large-scale multilingual corpus using LLaMA-2-7B, and construct a cross-lingual instruction tuning dataset with 320k samples for 10 languages, which is named XInstruction dataset. Codes and data will be made public after review to advocate future research.

heading: 4.1 Seed Data
text: For each language, we extract the first conversation turn of message trees in the Open Assistant dataset, and adopt the highest quality response as the output for each sample (Köpf et al., 2023). To construct the X-Instruction sample, the instruction of each sample is translated into English by Google Translate. However, there are only a few or even no human-labeled samples for the most of languages involved in this work. Thus, we translate the output of 3k English samples into the target language using Google Translate to supplement the seed data for each language. The statistics of seed data are reported in Appendix B.1.

heading: 4.2 Unsupervised Multilingual Corpus
text: The multilingual corpus for each language is extracted from the CulturaX dataset (Nguyen et al., 2023), which contains web texts in 167 languages
and is filtered from mC4 (Xue et al., 2021) and OSCAR (Suárez et al., 2019). We only sample 1M web texts for each language from this dataset due to the constraint on the computation budget.

heading: 4.3 Statistics
text: To investigate the diversity of X-Instruction, we parse English instructions and count the ones with verb-noun structure using the Berkeley Neural Parser (Kitaev and Klein, 2018; Kitaev et al., 2019). Figure 3 illustrates the top 16 most common root verbs and their top direct noun objects. We can find that the great diversity of X-Instruction constructed from web corpora. The additional information of X-Instruction is reported in Table 2.
Pie plot with `ax.pie`

heading: 4.4 Quality
text: For each language, we randomly sample 200 instructions and corresponding texts to further evalu-
ate the quality. There are two questions designed to conduct this evaluation, and results are reported in Table 3. It can be found that the quality of XInstruction is good for more than 80% of samples are valid. The top-flows come from additional information in web text like navigation bar (11.3%), incorrect URL and HTTP status codes (3.8%).

heading: 5 Experiments

heading: 5.1 Experiments Settings
text: Base models LLaMA-2 models with 7B and 13B parameters are taken as base models and fine-tuned on cross-lingual samples in each language from the X-Instruction dataset. Hyperparameters are reported in Appendix A.
Baselines used in this work are:
• Alpaca-MT: We translate the vanilla Alpaca dataset (Taori et al., 2023) into 10 languages using Google Translate and fine-tune the same base model for comparison.
• Bactrian-X (Li et al., 2023b): LLaMA models are fine-tuned with 3.48M instruction tuning samples in the Bactrian-X dataset, which consists of outputs from ChatGPT in 51 languages except English.
• Bactrian-M: We fine-tune LLaMA-2 models using 67k monolingual instruction tuning samples in each language from the Bactrian-X
dataset. It distills the monolingual capabilities of ChatGPT, providing a strong baseline for our work.
• ChatGPT (OpenAI, 2022) is fine-tuned through reinforcement learning with human feedback(Ouyang et al., 2022b) on the GPT3.5 model. It demonstrates remarkable multilingual capabilities across a wide range of tasks (Asai et al., 2023). We use the “gpt-3.5turbo-0613” API in this work.
Datasets To confirm the effectiveness of XInstruction, we conduct the evaluation of openend generation on Vicuna (Zheng et al., 2023), WizardLM (Xu et al., 2023), LIMA (Zhou et al., 2023), and Koala (Geng et al., 2023) datasets, which cover a variety of task categories. Prompts in these datasets are translated into the 10 languages involved. A comprehensive overview of these datasets is presented in Appendix B.2.

heading: 5.2 Evaluation Results from GPT-4
text: We take GPT-4 (OpenAI, 2023)3 as a judge to conduct automatic evaluation, which is found a higher correlation with human judgements (Liu et al., 2023b; Li et al., 2023d). Considering the excellent multilingual understanding ability of GPT-4 (OpenAI, 2023), it is reasonable and effective to employ GPT-4 to automatically evaluate the quality of responses in low-resource languages.
Specifically, we adopt pair-wise evaluation and request GPT-4 to determine the better response between responses (r1, r2) from different models given the instruction i. In the evaluation, GPT-4 judge outputs a score, named GPT-4 score in this work, from 0 to 10 based on their helpfulness, relevance, and accuracy. The details of GPT-4 evaluation prompt can be found in Appendix F.
To alleviate the position bias in the evaluation by GPT-4 (Zheng et al., 2023), we first request GPT-4 to evaluate (r1, r2), then switch the position of r1 and r2, which is (r2, r1), in the second evaluation. The better response is the one that wins twice or wins once and draws once.
Table 4 reports the win rates of models against ChatGPT on four benchmarks in three low-resource languages. It can be found that X-Instruction exhibits a significant performance advantage over Alpaca-MT and Bactrian-X in all benchmarks. The average improvement of X-Instruction models over
3It points to the “gpt-4-0613” API during our experiments.
Bactrian-M models, which distill outputs from ChatGPT, reaches 15.8%. Compared with ChatGPT, X-Instruction13B demonstrates even better performance with a 67.66% win rate.
Languages Generalization To evaluate the effectiveness of X-Instruction in diverse languages, we extend experiments to ten languages, including five medium-resource and five low-resource languages, on Vicuna and WizardLM datasets. Table 5 reports the detailed results on ten languages across two benchmarks, which exhibit a certain level of uniformity across ten languages. The X-Instruction models with only 3k labeled seed data on each language obtain the highest win rate across ten languages on average, indicating that our method can be extended to more languages.
We further examine the generative quality by calculating the average GPT-4 score for each model from all comparison pairs, shown in Table 5. The average generative quality of X-Instruction is the most pronounced, achieving an average score of 6.9 in all languages. Notably, X-Instruction outperforms Bactrian-X by a large margin (1.3) and achieves better performance than Bactrian-M (5.6). Although the response quality of X-Instruction13B drops to 7.34 in five low-resource languages, the
average win rate against ChatGPT increased by 14.9% compared to the one in the other 5 mediumresource languages. It comes from the worse performance of ChatGPT in low-resource languages.
From Generation to Understanding Since XInstruction models only learn how to reply in lowresource language, there is a conjecture naturally comes to mind: Can the generation capability of the model be generalized to its understanding capability?
To validate our conjecture, we design a zeroshot evaluation using prompts in the output language instead of the English prompt used in training. Table 5 shows the performance of zero-shot generation in ten languages. Compared with the vanilla cross-lingual generation, the win rate of X-Instruction models under zero-shot evaluation only drops by 5.7% on average. The average quality of responses in these languages incurs a minor performance degradation (-0.7), which is 90.6% of the vanilla one, indicating our model achieves exceptional zero-shot learning ability.
Skill Distribution Figure 4 shows the detailed performance of models given prompts in different categories from the Vicuna dataset. As shown in Figure 4(a) and 4(b), X-Instruction outperforms
Bactrian-M in all categories, and surpasses ChatGPT in eight categories except fermi. The excellent performance in commonsense, writing, and knowledge categories may come from the native multilingual corpus used in the X-Instruction dataset.
The primary reason for the inferior performance on code and math in Figure 4(a) is the lack of relevant corpora, which are filtered out by the language identification tool used in multilingual corpus cleaning. It can be alleviated by adding language-agnostic instructions-following samples from code and math domains. Figure 4(c) compares the outputs from X-Instruction13B given English instructions or instructions in other languages (the zero-shot evaluation). It can be found that the performance in the knowledge category declines the least when instructed in the output language.
Difficulty Distribution To evaluate models on instructions of different difficulties, we perform elaborate analysis on the WizardLM dataset, which contains labels of difficulty. Following the evaluation of WizardLM (Xu et al., 2023), we split the test set into “Easy”, “Medium”, and “Hard” three parts with difficulty levels on [1, 4], [5, 7], and [8, 10]. As shown in Figure 5, with the increase of difficulty, the win rate of X-Instruction13B im-
proves, while the one of Bactrian-M13B decreases. It reflects that the quality of responses from ChatGPT decreases more than the one of X-Instruction models when given more difficult instructions. It is noted that X-Instruction13B under zero-shot evaluation surpasses ChatGPT in all difficulty skills.
Response Quality We prompt GPT-4 to evaluate the responses in the following three views: helpfulness (0-10), relevance (0-10), and accuracy (0-10), and report the average results of 10 languages in Table 6. It can be found that the responses from X-Instruction13B are uniformly better than the baseline model in the three dimensions, especially in the relevance dimension (+1.9).

heading: 5.3 Evaluation Results from Human
text: To enhance the comprehensiveness and reliability of the evaluation, we further conduct the human evaluation in three low-resource languages, focus-
ing on the general quality of responses on the Vicuna dataset. Specifically, human evaluators are required to compare answers A and B generated by two models for each instruction, and choose an option from “A wins”, “B wins”, and “Tie” based on their judgment. For the sake of fairness, we randomize the order of answers and eliminate the position bias.
The results in Figure 6 demonstrate the better responses from our model compared with the ones of ChatGPT and Bactrian-M13B, and indicate the consistency between GPT-4 and human evaluation. Moreover, we provide examples in Appendix D.2 for qualitative analysis of the responses from different models.

heading: 5.4 Analysis

heading: 5.4.1 Iteration of Refinement
text: To investigate the effect of X-Instruction refinement, we statistic the win rates of X-Instruction7B against ChatGPT using 32k cross-lingual samples
from different iterations of refinement on 4 benchmarks. As shown in Figure 7(a), the win rate of models in Urdu and Bengali increases with more refinement iteration, which reflects the improvement in the quality of cross-lingual instruction tuning samples. Thus, we set the number of iterations to 3 by default, where the improvement in the quality of response is almost saturated.

heading: 5.4.2 Data Quantity and Quality
text: We study the impact of data quantity and quality on the X-Instruction model using the Urdu samples from the third refinement iteration. Figure 7(b) shows that the quality of the model responses will increase with more samples used and is close to saturation at 32k, which is similar to the findings of Li et al. (2023c). In addition, tuning on the samples with higher ratings brings better responses, which demonstrates that the evaluator trained does find higher quality instruction tuning samples.

heading: 5.4.3 Multilingual Tasks
text: In addition to the generation ability in these languages, we further evaluate the performance of X-Instruction models on multilingual natural language inference (Conneau et al., 2018) and commonsense reasoning tasks (Ponti et al., 2020; Lin et al., 2022). As shown in Table 7, the average improvement on the zero-shot in-context learning
performance of the base model is 3.1%, which is higher than the 1.8% improvement from the Bactrian-X dataset. It further confirms that crosslingual instruction tuning on X-Instruction enhances the multilingual language understanding abilities of language models.

heading: 6 Conclusion
text: In this paper, we propose a pipeline to excavate cross-lingual instruction-tuning samples from multilingual corpora by exploiting the better generation ability in high-resource languages of large language models. We apply it to 10 languages and construct a large-scale cross-lingual instructions dataset named X-Instruction. Experimental results from GPT-4 and human evaluation demonstrate that models tuned on X-Instruction can generate better responses and follow the instructions in the output language without further tuning.
We hope X-Instruction can bring more attention to the better instruction construction method for low-resource languages, future work could focus on directly generating high-quality instruction following samples in low-resource languages.
Limitations
Firstly, the X-Instruction sample is excavated by the LLaMA-2-7B model which is lower efficiency in processing multilingual corpus for its Englishdominant vocabulary. We acknowledge that higher quality X-Instruction samples can be built by adopting more powerful base models, e.g., the model with more parameters.
In addition, due to limited computation resources, our method is only applied to 10 languages in this work and other languages can be investigated in the future.
It is noted that the evaluation of open-end generation is conducted on the single-turn conversation for the limited quota of our OpenAI account. The evaluation results might be different in multi-turn dialogue benchmarks.
Ethical Considerations
The instruction-tuning samples excavated by our method may contain cultural bias and toxic content from the web corpus used. It can be alleviated by adopting the corpus after rigorous cleaning like CulturaX (Nguyen et al., 2023) or incorporating the evaluation of safety into the X-Instruction re-
finement stage, which can be investigated in the future.

heading: Acknowledgements
text: We would like to thank Junhong Wu and the anonymous reviewers for their helpful discussions and valuable comments. The research work was supported by the National Key R&D Program of China (No. 2022ZD0160602), the Natural Science Foundation of China (No. 62122088), and the STI2030Major Project (No. 2021ZD0204105).

heading: A Hyperparameters
text: The instruction tuning models and evaluators are tuned by AdamW (Loshchilov and Hutter, 2019) optimizer with an initial learning rate of 1e-5 and
3 epochs under the cosine schedule. The batch size is set to 32 for most models and reduced to 8 when training samples are less than 3k. We use mixed precision training and ZeRO to speed up the training process and save memory (Micikevicius et al., 2018; Rasley et al., 2020). The nucleus sampling (Holtzman et al., 2020) is adopted to generate responses from models under p = 0.9 and T = 0.7. All experiments are conducted on a GPU server with 8*A100 80GB RAM.
X-Instruction Dataset We empirically set the number of iterations in X-Instruction Refinement to 3. The length of web texts used is limited to 64∼2048 characters, and the whole part of text is used as the candidate response. In the cross-lingual refinement stage, 2.5k samples from seed data, named Dx0 , are used to fine-tune the cross-lingual following model, the samples left are exploited to construct the synthetic dataset Drk. There are two system templates Sa and Sw designed for seed data and augmented samples respectively, which is in line with the settings in Li et al. (2023c). The sentence embedding model “all-mpnet-base-v2” (Song et al., 2020) is used to obtain the embedding of English instructions in the final diversification stage.

heading: B Data Statistics
text: B.1 Seed Data The statistics of seed data used are reported in Table 8. It is noted that all outputs of five low-resource languages are translated from English samples for none samples in these languages are found in the Open Assistant dataset (Köpf et al., 2023).
B.2 Datasets Table 10 shows the details of other multilingual instruction tuning datasets. We also report the
information of four open-end generation datasets in Table 9. Notably, both Vicuna and WizardLM datasets have diverse categories, thereby ensuring the richness and diversity of the test sets. It not only guarantees a thorough evaluation, but also minimizes the risk of evaluation bias since the test sets encompass various instruction categories.

heading: C Additional Results and Analyses
text: C.1 Detailed Results from GPT-4 Figure 8 shows the detailed results of X-Instruction13B vs. ChatGPT on four datasets. Moreover, we report the evaluation results in all languages on two benchmarks from GPT-4 in Table 11.
C.2 Diversification Figure 9 illustrates the impact of different numbers of clusters in the final diversification stage when sampling the same amount of data. Given the amount of final data to 32k, the output quality of models tuned drops when the number of clusters reaches 2000, which may come from the smaller inter-cluster distance and less diversity in the sampled data. Thus, the number of clusters is set to 1000 by default.
C.3 Improvement over Seed Model To take a deep look into the detailed improvements brought by the cross-lingual instructions augmented in X-Instruction, we statistic the detailed performance on different categories in three languages (tr, sw, ur) for the seed model and X-Instruction7B when compared with ChatGPT. As shown in Table 12, the performance on prompts of all categories is improved, especially for the
X-Instruction!"# wins ChatGPT winsTie
LIMA
WizardLM
Koala
Vicuna
245
Bengali Swahili Urdu
LIMA
WizardLM
Koala
Vicuna
LIMA
WizardLM
Koala
Vicuna
137
105
68 48
22 53
28 53
26 29 213
110
92
52 6 22
18 70
44 64
31 56 240
149
107
63 512
27 46
26 43
25 35
0 80 180 218 300 0 80 180 218 300 0 80 180 218 300 The Number of Questions The Number of QuestionsThe Number of Questions
Figure 8: Comparing X-Instruction13B and ChatGPT in 3 low-resource languages across four benchmarks.
“Counterfactual”, “Common-sense” and “Roleplay” categories.

heading: D Case Study
text: D.1 Quality Evaluation
We report two valid samples and invalid samples in Figure 10(a) and 10(b). Although there are inappropriate instructions in the invalid samples, we can find that the semantics of the English instruction generated are related to the given text.
D.2 Instruction Following
To qualitatively analyze responses from different models, we report four cases in Figure 14, Figure 15, Figure 16, and Figure 17. It can be found that X-Instruction13B provides a more detailed and co-
herent response for the same instruction provided. In some cases, e.g., Figure 17, X-Instruction13B provides suggestions in texts rather than codes needed for instructions about code generation, which may arise from the lack of code data in the multilingual corpus as responses.

heading: E User Interface in Human Evaluation
text: E.1 Quality Evaluation
We conducted the quality evaluation of the XInstruction dataset with five annotators. We paid $0.1 for the evaluation of each sample. The user interface in quality evaluation is illustrated in Figure 11.
E.2 Response Comparison
In the human evaluation experiment, three evaluators were requested to choose the better response according to their preference and received $0.2 for each annotation. We illustrate the user interface for response comparison in Figure 12.

heading: F Prompt Template in GPT-4 Evaluation
text: We provide the prompt template used in GPT-4 Evaluation in Figure 13.

heading: G Additional Information about Language Code
text: Table 13 presents more information about the language codes involved in this work.

heading: H Dataset License
text: The X-Instruction dataset is built from the CulturaX dataset (Nguyen et al., 2023), which is filtered from mC4 (Xue et al., 2021) and OSCAR (Suárez et al.,
2019). Therefore, the license of X-Instruction follows the one of mC4 (ODC-BY4) and OSCAR (the Creative Commons CC0 license5).
4https://opendatacommons.org/licenses/by/1-0/ 5http://creativecommons.org/publicdomain/zero/1.0/
English Instruction How do I create a Campus ID for my current user?
Finnish Text
Luo Sopron nykyiselle käyttäjälle Campus tunnus seuraavasti. Mene valvojan tunnuksella ja oikeuksilla Sentraaliin. Valitse siellä Rekisterit/Henkilökunta ja valitse ko. henkilö. Oikeassa reunassa on nappi jossa lukee ”Muuta käyttäjätunnuksen tietoja”, pai... (Create a Campus ID for Sopro's current user as follows. Go to Central with the administrator ID and rights. There, select Registers/Personnel and select person. On the right side there is a button that says "Change username information", pai...)
English Instruction Is this going to be a continuous trend or not? The record-breaking temperatures around the world are
becoming more common.
Hindi Text
लेकिन जसेै-जसेै दनुनया भर में ररिॉर्ड-तोड़ तापमान बढ रहा है, हो सिता है यह लगातार न बना रहे। वुर्वेल क्लाइमेट और वाइिाटो िे शोधिताडओ ंने एि तापमान सीमा िा पता लगाया है, जजसमें भववष्य में पौधों िे िाबडन अवशोवित िरने िी दर िम और िाबडन छोड़ने िी दर तेज होगी। (But as record-breaking temperatures rise around the world, this may not last. Researchers at Woodvale Climate and Waikato have identified a temperature threshold in which plants will have a slower rate of carbon absorption and a faster rate of carbon release in the future.)
(a) Valid samples
English Instruction Create a watermark for a PDF document.
Bengali Text
ওয়াট়ারম়ার্ক পিপিএফ - অনল়াইন, পনর়ািদ, ব্যপিগত এব্ং পব্ন়ামলূ্লয এর্টি পিপিএফ ফ়াইল্ল এর্টি জলছপব্ যুি র্রুন. আম়াল্দর পিপিএফ টুলগুপল আিন়ার ফ়াইলগুপলল্র্ ইন্ট়ারল্নল্টর ম়াধ্যল্ম স্থ়ান়ান্তর র্ল্র ন়া র়্ারণ আিন়ার ফ়াইলগুপলর অি়াল্রশনগুপল ব্র়াউজ়ার পনল্জই র্ল্র। আিন়ার গগ়ািনীযত়া এব্ং পনর়ািত্ত়া য়াল্ত সল্ব্ক়াত্তমভ়াল্ব্ সুরপিত থ়াল্র্। (Watermark PDF - Online, Secure, Private and Free Add a watermark to a PDF file. Our PDF tools do not transfer your files over the Internet because the operations on your files are done by the browser itself. So that your privacy and security are best protected.)
English Instruction Rewrite the following text so that it reads naturally and has the same meaning. People who are outgoing are more
likely to make friends than those who are not.
Indonesian Text
Orang yang tulus mau berteman, pasti akan menerima kita apa adanya. Jadi, kita gak perlu menjadi seperti orang lain hanya demi diterima dalam pergaulan. Memaksakan diri menjadi orang lain hanya akan membuat kita seolah memakai topeng layaknya sebuah peran. (People who sincerely want to be friends will definitely accept us as we are. So, we don't need to be like other people just to be accepted in society. Forcing ourselves to be someone else will only make us seem like we are wearing a mask like a role.)
English Instruction I want to write a modern operating system from scratch for AMD64 systems with your assistance. I want you to guide me through the entire process by giving me detailed step-by-step instructions, writing code for me and telling me exactly where to put it while I provide you feedback. Where do I start? Answer in Swahili.
Instruction Source Dataset: WizardLM Question_id: 36 Category: Code Generation
Swahili Instruction Ninataka kuandika mfumo wa uendeshaji wa kisasa kutoka mwanzo kwa mifumo ya AMD64 kwa usaidizi wako. Nataka uniongoze katika mchakato mzima kwa kunipa maelekezo ya kina ya hatua kwa hatua, kuniandikia msimbo na kuniambia ni wapi pa kuiweka huku nikitoa maoni. Nianzie wapi?
Kuandika mfumo wa uendeshaji kutoka mwanzo ni kazi ngumu na yenye changamoto sana, na inategemea uzoefu wako wa kutengeneza programu na uelewa wa kiufundi. Ikiwa huna uzoefu wowote wa kutengeneza programu au kiufundi, ninapendekeza kuanza na kozi maalum au kitabu cha mafunzo ili kupata ufahamu wa msingi wa jinsi mifumo ya uendeshaji inavyofanya kazi. Ikiwa unaamua kuendelea na mradi huu, hapa kuna hatua za kuanzia:

heading: 2. Jifunze misingi ya AMD64: Kabla ya kuanza kuandika mfumo wa uendeshaji, ni muhimu kuelewa jinsi mfumo wa AMD64 unavyofanya kazi. Hii itajumuisha kuelewa muundo wa hifadhidata, mfumo wa uendeshaji wa kutofautisha, na michakato ya kuingiza na kutolea.

heading: 1. Chagua lugha ya programu: Mifumo ya uendeshaji inaweza kuandikwa katika lugha nyingi tofauti za programu, ikiwa ni pamoja na C, C++, Rust, na zaidi. Chagua lugha ambayo inafaa zaidi kwako na ambayo ina jumuiya inayoendelea na rasilimali nyingi za mtandaoni.

heading: Future Work
text: Researchers at Woodvale Climate and Waikato have identified a temperature threshold in which plants will have a slower rate of carbon absorption and a faster rate of carbon release in the future. )
(a) Valid samples
English Instruction Create a watermark for a PDF document. Bengali Text
ওয়াট়ারম়ার্ক পিপিএফ - অনল়াইন, পনর়ািদ, ব্যপিগত এব্ং পব্ন়ামলূ্লয এর্টি পিপিএফ ফ়াইল্ল এর্টি জলছপব্ যুি র্রুন. আম়াল্দর পিপিএফ টুলগুপল আিন়ার ফ়াইলগুপলল্র্ ইন্ট়ারল্নল্টর ম়াধ্যল্ম স্থ়ান়ান্তর র্ল্র ন়া র়্ারণ আিন়ার ফ়াইলগুপলর অি়াল্রশনগুপল ব্র়াউজ়ার পনল্জই র্ল্র। আিন়ার গগ়ািনীযত়া এব্ং পনর়ািত্ত়া য়াল্ত সল্ব্ক়াত্তমভ়াল্ব্ সুরপিত থ়াল্র্। (Watermark PDF - Online, Secure, Private and Free Add a watermark to a PDF file. Our PDF tools do not transfer your files over the Internet because the operations on your files are done by the browser itself. So that your privacy and security are best protected. )
English Instruction Rewrite the following text so that it reads naturally and has the same meaning. People who are outgoing are more
likely to make friends than those who are not. Indonesian Text
Orang yang tulus mau berteman, pasti akan menerima kita apa adanya. Jadi, kita gak perlu menjadi seperti orang lain hanya demi diterima dalam pergaulan. Memaksakan diri menjadi orang lain hanya akan membuat kita seolah memakai topeng layaknya sebuah peran. (People who sincerely want to be friends will definitely accept us as we are. So, we don't need to be like other people just to be accepted in society. Forcing ourselves to be someone else will only make us seem like we are wearing a mask like a role. )
English Instruction I want to write a modern operating system from scratch for AMD64 systems with your assistance. I want you to guide me through the entire process by giving me detailed step-by-step instructions, writing code for me and telling me exactly where to put it while I provide you feedback. Where do I start? Answer in Swahili. Instruction Source Dataset: WizardLM Question_id: 36 Category: Code Generation
Swahili Instruction Ninataka kuandika mfumo wa uendeshaji wa kisasa kutoka mwanzo kwa mifumo ya AMD64 kwa usaidizi wako. Nataka uniongoze katika mchakato mzima kwa kunipa maelekezo ya kina ya hatua kwa hatua, kuniandikia msimbo na kuniambia ni wapi pa kuiweka huku nikitoa maoni. Nianzie wapi?
Kuandika mfumo wa uendeshaji kutoka mwanzo ni kazi ngumu na yenye changamoto sana, na inategemea uzoefu wako wa kutengeneza programu na uelewa wa kiufundi. Ikiwa huna uzoefu wowote wa kutengeneza programu au kiufundi, ninapendekeza kuanza na kozi maalum au kitabu cha mafunzo ili kupata ufahamu wa msingi wa jinsi mifumo ya uendeshaji inavyofanya kazi. Ikiwa unaamua kuendelea na mradi huu, hapa kuna hatua za kuanzia: