UltraLink : An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset (2024)

Abstract: Open-source large language models (LLMs) have gained significant strength across diverse fields. Nevertheless, the majority of studies primarily concentrate on English, with only limited exploration into the realm of multilingual abilities. In this work, we therefore construct an open-source multilingual supervised fine-tuning dataset. Different from previous works that simply translate English instructions, we consider both the language-specific and language-agnostic abilities of LLMs. Firstly, we introduce a knowledge-grounded data augmentation approach to elicit more languagespecific knowledge of LLMs, improving their ability to serve users from different countries. Moreover, we find modern LLMs possess strong cross-lingual transfer capabilities, thus repeatedly learning identical content in various languages is not necessary. Consequently, we can substantially prune the language-agnostic supervised fine-tuning (SFT) data without any performance degradation, making multilingual SFT more efficient. The resulting UltraLink dataset comprises approximately 1 million samples across five languages (i.e., En, Zh, Ru, Fr, Es), and the proposed data construction method can be easily extended to other languages. UltraLink-LM, which is trained on the UltraLink dataset, outperforms several representative baselines across many tasks. 1

heading: 1 Introduction
text: Thanks to the collaborative efforts of the active large language models (LLMs) community, opensource LLMs are becoming increasingly powerful (Touvron et al., 2023a,b; Jiang et al., 2023), even outperforming some representative closedsource counterparts (OpenAI, 2023; Anil et al., 2023) in some specific tasks (Wei et al., 2023b).
* Equal contribution. † Corresponding authors. 1 Both UltraLink and UltraLink-LM will be publicly avail-
able at https://github.com/OpenBMB/UltraLink.
These accomplishments are closely related to the contribution of open-source supervised fine-tuning (SFT) data (Ding et al., 2023; Anand et al., 2023; Peng et al., 2023; Wang et al., 2023; Kim et al., 2023; Xu et al., 2023), which plays a pivotal role in eliciting the instruction-following ability of LLMs and aligning the model behaviour with human preferences. Nevertheless, the focus of existing works is primarily on the construction of English SFT data, resulting in a comparatively limited availability of multilingual SFT resources.
To mitigate the challenge of data scarcity, some researchers suggest translating English SFT data into multiple languages. Lai et al. (2023) utilize ChatGPT2 to translate the two essential components, instructions and responses, from Alpacastyle (Taori et al., 2023) English data to other languages. Chen et al. (2023) propose to translate
2https://chatgpt.com/
11929
both the Alpaca and the ShareGPT3 data. While directly translating English SFT data can effectively support multilingual SFT, there are still two major drawbacks associated with this approach:
• Low cultural diversity and imprecise translations caused by cultural differences: translation of English data may not adequately encompass topics specific to non-English regions (e.g., subjects related to Russian culinary culture), leading to a deficiency in language-specific knowledge for LLMs. Moreover, for certain instructions (e.g., what are the most important holidays of the year?), the answers vary in different cultural backgrounds, so directly translating all English conversations may result in numerous distorted translations.
• Linearly increased data volume: the total volume of translated SFT data linearly increases with the number of languages. However, the translations across different languages are semantically equivalent, making the model repeatedly learn the same content.
We believe that a good multilingual LLM should not only possess language-specific knowledge but also be equipped with language-agnostic skills. Figure 2 gives an example of the two types of instructions. We thus propose a new approach to better construct multilingual SFT data, applicable to any language. Compared to conversation translation (Lai et al., 2023; Chen et al., 2023), our advantages can be illustrated as follows:
• Higher cultural diversity and less distorted translations: for language-specific data, we propose a knowledge-grounded data augmentation method. Concretely, Wikipedia is employed4 as a data source for each language to provide more language-specific contexts. For language-agnostic chat data (e.g., the second example in Figure 2), we propose a two-stage translation mechanism. Given high-quality English SFT data, we first filter out the conversations that are specific to certain regions. Then we translate the remaining languageagnostic data.
• Pruned data volume: for language-agnostic skills like math reasoning and code generation,
3https://sharegpt.com 4https://www.wikipedia.org
through our experiments, we find that it is unnecessary for the model to repeatedly learn identical problems, thanks to the strong crosslingual transfer capabilities of modern LLMs. We can thus significantly prune the amount of math and code SFT data for non-English languages without compromising the model performance.
We apply the aforementioned approach to four non-English languages, including Chinese, Russian, French, and Spanish. Note that our method can also be easily extended to other languages. Finally, we train a SFT LLM on the proposed UltraLink dataset, which outperforms several representative open-source multilingual LLMs, demonstrating the effectiveness of our dataset.

heading: 2 Data Curation
text: Automatically generating SFT data is now an important research topic for LLMs (Taori et al., 2023; Wang et al., 2023; Ding et al., 2023). For multilingual SFT, it is crucial to consider the influence of cultural diversity on language-specific data, while also integrating language-agnostic universal data that is related to the general abilities of LLMs (i.e., math reasoning). In this work, we propose a data construction framework consisting of two pipelines, as shown in Figure 3.

heading: 2.1 Language-Specific Data Curation
text: The cultures around the world are vibrant and diverse, reflecting the lifestyles and perspectives of people from various countries and regions. To better cater to diverse users, the cultural diversity of multilingual LLMs should be improved. In this aspect, we propose a knowledge-grounded data augmentation method, leveraging language-specific data sources to provide intricate and varied cultural backgrounds. Our method mainly contains two
steps: (1) preparing and sampling knowledge from data sources as cultural backgrounds, and (2) steering LLMs to generate informative conversations given the provided cultural backgrounds.

heading: 2.1.1 Knowledge Preparation
text: For each language, we utilize Wikipedia dumps5 as the data source, encompassing a diverse array of topics closely related to the respective culture. We first use an open-source extraction toolkit6 to preprocess the raw dumps and get text descriptions for each entry. Then we use the language identification model provided by fastText (Joulin et al., 2017) to remove contents that are not in the expected language. For Chinese, we also use OpenCC7 to convert traditional Chinese texts into simplified Chinese. Finally, we filter out documents that are shorter than 1K tokens or longer than 10K tokens. The number of tokens is calculated by tiktoken8.
Given that most LLMs have a limited context length, we divide the whole text into segments whose lengths are between 1K and 2K. We do not split whole sentences when performing text segmentation. The preprocessed texts are used as contexts for the following dialogue generation pro-
5https://dumps.wikimedia.org 6https://github.com/attardi/wikiextractor 7https://github.com/BYVoid/OpenCC 8https://github.com/openai/tiktoken
cedure.

heading: 2.1.2 Dialogue Generation
text: To automatically generate multi-turn dialogues, we designed a question generator and an answer generator, which are both based on GPT-3.5. When generating the dialogue, both the question and answer generators are conditioned on a provided text segment as the cultural background. The used prompts can be divided into four parts: system prompt, principles, cultural background, and dialogue history. The prompt structure is shown in Figure 4.
The system prompt is used to describe the task (i.e., generating the initial question). The principles provide some detailed suggestions for the LLM, which are found important for improving the quality of the generated data. The cultural background is the preprocessed text segment that contains language-specific knowledge. The dialogue history provides the historical questions and
answers, which are set to an empty string when generating the initial question.
Generating the Initial Dialogue The principles used to generate the first question are shown in Figure 5. We ask the involved LLM (i.e., GPT-3.5) to understand the provided cultural background and then propose a related question that can be answered according to the cultural background. For the generation of answers, we provide only a concise description of the principles in Figure 6 due to space limitations. For each language, the principles are translated by humans into the target language. We only show the English version of the prompt to understand the method better.
Generating Subsequent Dialogues After generating the initial question and answer, we iteratively produce subsequent dialogues. To improve the diversity of constructed dialogues, we propose two types of subsequent questions. At each turn, we randomly decide whether to present an in-depth ques-
tion for a more detailed exploration of the same topic or to generate an expansive question to delve into other subjects. The principles used to ask an in-depth question are shown in Figure 7, while the principles used to ask an expansive question are shown in Figure 8. Note that when generating subsequent dialogues, the cultural background is also provided to the model.
Using the aforementioned approach, we automatically construct language-specific multi-turn conversations in four languages. The details of constructed data will be illustrated in Section 3, including the average length and some other statistics.
Note that the proposed knowledge-grounded data augmentation approach can also be applied to any other language.

heading: 2.2 Language-Agnostic Data Curation
text: In addition to language-specific abilities, the general abilities that are language-agnostic are also essential for LLMs. As numerous high-quality English SFT datasets already encompass a broad spectrum of general abilities, we suggest employing a two-stage translation mechanism to maximize the utility of existing English resources. Our goal is to reduce translation errors caused by cultural differences since some questions can not be directly translated into other languages (e.g., write an English poem where each sentence starts with the letter “A"). In the first stage, we introduce a multi-criteria mechanism to filter out English-specific conversations that are difficult to translate accurately into other languages. Then we use GPT-3.5 to translate the remaining languageagnostic data. In this study, we consider three key components of general abilities for LLMs: chat, math reasoning, and code generation. For chat, we use ShareGPT as the English chat data, which consists of multi-turn dialogues between human users and ChatGPT. For math reasoning, we use MetaMath (Yu et al., 2023) as the English math data. For code generation, we use the Magicoder dataset (Wei et al., 2023b) as the English code data.

heading: 2.2.1 Multi-Criteria Filter
text: The criteria employed to filter out English-specific conversations are outlined in Figure 9. Our goal is to retain only conversations whose topics can be discussed in any cultural background. GPT-3.5 is utilized to ascertain whether a conversation contains information relevant to the specified features. For instance, the conversations that include English jokes will be removed before translation.

heading: 2.2.2 Translator
text: After the filtering process, the remaining conversations undergo the translation procedure, wherein they are translated into four languages using GPT3.5-turbo to maintain fluency and accuracy. We also provide some translation principles to help GPT-3.5 better perform the translation, which is shown in Figure 10.

heading: 1. Ensure the completeness and consistency of content during the translation process, without adding or deleting any information.

heading: 4. If there are no equivalent translation terms for certain vocabulary, please directly copy the original text.

heading: 5. For citations and references, please directly copy the original text.

heading: 2.3 Data Pruning
text: English math and code datasets are frequently extensive, exemplified by MetaMath (Yu et al., 2023) with 395K training examples and Magicoder (Wei et al., 2023b) comprising 186K training examples. Assuming the English data consists of N training examples, the overall multilingual dataset would encompass k ×N examples if we translate all the English training examples into other languages, where k is the number of languages. The linear increase in data volume will result in higher training costs during SFT. As math and code problems are not closely tied to the cultural backgrounds of different countries, LLMs may have the capability to transfer English math and code abilities into other languages with only limited training examples. In other words, it may not be necessary for LLMs to learn all translated math and code problems. To verify the assumption mentioned above, we conduct experiments on Chinese math and code tasks.
For comparison, we fine-tune Llama-2-7b (Touvron et al., 2023b) in the following two different ways:
• From En SFT Model: we first use English math or code data to fine-tune the base model, and then use different amounts of Chinese data to further tune the model.
• From Base Model: we directly use Chinese math or code data to fine-tune the base model.
Figure 11 and 12 show the performances of the two types of models. Surprisingly, the involved LLM exhibits strong cross-lingual transfer capabilities. For instance, utilizing only 2K Chinese mathematical training examples can yield a score of 45.6 when fine-tuning from the English SFT model. In contrast, directly fine-tuning the base model with an equivalent amount of Chinese data results in a significantly lower score of 22.0, highlighting the superior performance achieved through transfer from the English SFT model. In the Chinese code generation task, we observe a similar trend, wherein transfer learning from the English SFT model substantially enhances the performance of the model.
Moreover, we find that using more Chinese SFT data does not consistently lead to improved performance. For the math task, using 32K Chinese training examples achieves the best result. For the code task, the peak performance is attained with 16K Chinese code generation examples. Hence, we incorporate only 32K mathematical training examples and 16K code training examples for each non-English language in the UltraLink dataset.

heading: 2.4 Data Overlap Strategy
text: UltraLink consists of four parts, including the language-specific chat part, the language-agnostic chat part, the code part, and the math part. It is worthwhile to design different strategies for them because of their intrinsic features.
For language-specific data, we utilize respective Wikipedias to prevent overlap, recognizing that different languages often focus on distinct frequentlydiscussed topics. The content of language-agnostic chat data is shared across languages to capture general knowledge. For code and math data, as there is no prior indication that certain languages favour specific code or math topics, we randomly selected examples for each language.

heading: 2.5 Data Quality Ensurance
text: It is important to ensure the quality of the data generated by LLM. We deal with this issue from the following aspects. Firstly, We primarily filter Wikipedia articles based on length, as longer texts often contain more detailed information, providing GPT with a comprehensive context for generating accurate dialogue data. Additionally, we attach a paragraph split from Wikipedia to each QA generation to improve answer quality. For instance, answers generated by GPT with a Wiki paragraph in an autobiography context show high relevance and can accurately describe when events occurred,
ensuring correctness. We also leverage the tendency of longer inputs to elicit longer responses to obtain comprehensive answers. To prevent the negative impact of truncation on LLM, we filter out truncated responses from the API based on the window size of the used LLM. As for the quality check of UltraLink, the pass rate for manual inspection is 96%, and the pass rate for answers is 99%.

heading: 3 Dataset Statistics

heading: 3.1 Data Distribution
text: Table 2 presents the scale of each component in UltraLink, encompassing five languages. Each language contributes four types of SFT data: chat data with language-specific knowledge, chat data with language-agnostic knowledge, math data, and code data. The quantities of language-agnostic segments are approximately equal for the four non-English languages.

heading: 3.2 Comparison with Existing Datasets
text: Before us, there are some existing multilingual SFT datasets, where we select four representative datasets for comparison, including the Okapi dataset (Lai et al., 2023), the Guanaco dataset (Attardi, 2023), Multialpaca (Wei et al., 2023a), and the Phoenix SFT data (Chen et al., 2023). We conduct a comparison based on the number of dialogues, the number of conversation turns, and the average lengths across the respective datasets. As shown in Table 1, we find that UltraLink contains fewer dialogues than the Guanaco dataset, but the latter only contains single-turn conversations. Only the Phoenix SFT data and UltraLink include multiturn conversations.
We use the number of tokens estimated by tiktoken as the length for each question and answer. The question token length does not include the document. On average, UltraLink exhibits the longest average length per turn (i.e., 378.21 tokens), considering both questions and their corresponding answers. Compared to UltraLink, the Phoenix SFT data has longer questions (165.27 vs. 87.86), but its answers are shorter (200.07 vs. 290.35). For each language, we also estimate the average lengths of questions and answers, and the results are shown in Figure 13. Across all languages, the answer is significantly longer than the question.
The primary distinctions between our dataset and existing multilingual SFT datasets can be clarified from the following aspects:
• Think from two dimensions: Our dataset takes into account both language-specific and language-agnostic aspects, filling a gap left by previous works, which can facilitate the development of more versatile and effective language models.
• Not just translation: Our dataset is grounded by Wikipedia, which ensures a higher degree of correctness and significantly reduces the occurrence of misconceptions. This feature is particularly beneficial for tasks that require a solid grounding in factual information. The extracted text from Wikipedia can provide more cultural backgrounds, improving the cultural diversity of the resulting dataset.
• Focus on generation: Unlike many existing datasets that are primarily designed for question-answering tasks, our dataset is geared towards knowledge infusion and the generation of extended text. This unique focus makes it a valuable resource for advancing research in these areas.

heading: 4 Experiment

heading: 4.1 Setup
text: Baselines For thorough comparison, we select several representative multilingual baselines in our experiments, including Bloomz-7b1-mt (BigScience, 2023), Phoenix-inst-chat-7b (Chen et al., 2023), PolyLM-Multialpaca-13b (Wei et al., 2023a), PolyLM-Chat-13b (Wei et al., 2023a), Chimera-inst-chat-13b (Chen et al., 2023), Okapi7b (Lai et al., 2023), Guanaco-7b (Attardi, 2023), Guanaco-13b (Attardi, 2023), and Aya-101 (Üstün et al., 2024). Okapi-7b is fine-tuned by ourselves based on Llama-2-7b using the Okapi dataset. Besides, the Aya-5-reimplement is fine-tuned in this
work based on Llama-2-13b, using the same languages as UltraLink. The training examples are from Aya Dataset and Aya Collection (Singh et al., 2024). The other baselines are downloaded from Hugging Face Hub9.
Training details Based on Llama-2-13b (Touvron et al., 2023a), UltraLink-LM is fine-tuned with the constructed UltraLink dataset for 3 epochs. We use the cosine learning rate schedule and the peak learning rate is set to 2e-5. The warm-up ratio is set to 0.04. Each mini-batch contains 128 training examples in total. The maximum sequence length is 4096. We train the model using 32 A100 GPUs for about 140 hours. Aya-5-reimplement uses the same training setting as UltraLink-LM.
9https://huggingface.co
Evaluation We examine the model performance in two categories of tasks, Natural Language Generation (NLG) and Natural Language Understanding (NLU). The NLG evaluation consists of three tasks, including chat, math reasoning, and code generation. For chat, we use OMGEval (Liu et al., 2023) for evaluation, which is a multilingual version of the widely-used English benchmark AlpacaEval (Li et al., 2023). OMGEval is not a mere translated version of AlpacaEval. Instead, it localizes the English questions according to the cultural backgrounds of each language. We employ MGSM (Shi et al., 2023) to evaluate math reasoning abilities, which is also a multilingual benchmark. Since there are no existing multilingual test sets for code generation, we use GPT3.5 with carefully designed prompts to translate HumanEval (Chen et al., 2021) into other lan-
guages, which serves as the multilingual benchmark to evaluate the code abilities of LLMs. In terms of NLU tasks, we use 3 different benchmarks, MMLU, Hellaswag, and ARC. We use the Okapi (Lai et al., 2023) version of the multilingual evaluation dataset.
We use the UltraEval toolkit10 for model inference and evaluation, which supports a wide range of open-source models.

heading: 4.2 Results
text: Table 3 describes the results of Guanaco-13b, Aya101, Aya-5-implement, and UltraLink-LM. The detailed results of all baselines can be found in the Appendix (Table 5 and Table 6). In terms of general chat abilities, our model achieves the best average results, which implies the superiority of the proposed UltraLink dataset.
For the code generation task, previous multilingual SFT datasets did not take into account the multilingual code abilities, which we think is very important in many real-world scenarios. Our model achieves a score of 60.4 in the English HumanEval benchmark, surpassing even CodeLlama34b-Python (Rozière et al., 2024).
In the math reasoning task, our model consistently outperforms all other baselines across all five languages. The performance of UltraLink-LM in both math and code tasks underscores the effectiveness of our method in enabling multilingual LLMs to acquire general abilities.
Within the scope of NLU tasks, UltraLink, though not specifically optimized for short-answer datasets, exhibits an appreciable performance superiority. This outcome illustrates the inherent potential that lies within the dataset used.

heading: 5 Related Work
text: Supervised Fine-tuning SFT is now a crucial part of constructing a powerful LLM. SODA (Kim et al., 2023) constructs high-quality social dialogues by contextualizing social commonsense knowledge from a knowledge graph. Using the technique of self-instruct (Wang et al., 2023), Alpaca (Taori et al., 2023) is one of the pioneers to leverage ChatGPT to collect SFT data. UltraChat (Ding et al., 2023) utilizes ChatGPT to generate topics in a tree-style structure for the construction of large-scale dialogues. With these efforts,
10https://github.com/OpenBMB/UltraEval
English SFT resources are becoming increasingly rich and effective.
Multilingual SFT Datasets To enhance the global utility of LLMs, numerous multilingual SFT datasets have been created. Lai et al. (2023) employ ChatGPT to translate Alpaca into various languages. Chen et al. (2023) combine ShareGPT with Alpaca and then translate the two datasets. Attardi (2023) and Wei et al. (2023a) extend tasks from Alpaca by introducing filters and rewrites of seed tasks in different languages, generating datasets through multiple iterations.

heading: 6 Conclusion
text: In this work, we propose a knowledge-grounded data augmentation method and a two-stage translation mechanism to construct language-specific and language-agnostic multilingual SFT data, respectively. Experiments demonstrate that the proposed dataset is effective for multilingual LLMs.

heading: 7 Limitations
text: In the paper, our proposed data construction framework is only applied to four language types. Nevertheless, the framework can be easily extended to other languages. We leave it to the future work to include more languages. Moreover, due to constraints imposed by the base model, the multilingual capability still faces several limitations. Notably, the model exhibits significantly better performance in English across many tasks. There is a pressing need to continue constructing high-quality pre-training multilingual datasets, to unlock the full potential of multilingual abilities in LLMs.

heading: Acknowledgements
text: We are grateful to the anonymous reviewers for their valuable feedback. This work is supported by the National Key R&D Program of China (No.2022ZD0116312), the National Natural Science Foundation of China (No. 62236011), and a grant from the Guoqiang Institute, Tsinghua University.

heading: A Data Mixing Strategy
text: The method of mixing the data of different languages could impact the performance of LLM. After getting the volume of different languages, we test three data mixing strategies, including sequential, mixed, and zh-only strategies, to fine-tune the base model. We choose MiniCPM-2.4B11 as the backbone to reduce training costs. “Sequential” means the model is trained on English data first, and then trained on Chinese data. “Mixed” represents that the English and Chinese data are randomly mixed during the training process. “Zh-only” stands for fine-tuning the model only with Chinese data. The evaluation is conducted on the Chinese HumanEval test set
Table 4 shows the results, indicating that mixing multilingual SFT data can result in the best performance. We thus employ the “Mixed” strategy when training UltraLink-LM.

heading: B Evaluation Results
text: This section details the full experimental results of all the baselines on the 6 examined benchmarks.
11https://github.com/OpenBMB/MiniCPM.

heading: Future Work
text: We leave it to the future work to include more languages. Moreover, due to constraints imposed by the base model, the multilingual capability still faces several limitations. Notably, the model exhibits significantly better performance in English across many tasks. There is a pressing need to continue constructing high-quality pre-training multilingual datasets, to unlock the full potential of multilingual abilities in LLMs