Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
Authors: Youmi Ma
Sakae Mizuki
Kazuki Fujii
Taishi Nakamura
Masanari Ohi
Hinari Shimada
Taihei Shiotani
Koshiro Saito
Koki Maeda
Kakeru Hattori
Takumi Okamoto
Shigeki Ishida
Rio Yokota
Hiroya Takamura
Naoaki Okazaki
Venue: COLM 2025 (2025)
Keywords: large language models; instruction tuning; synthetic data generation; cross-lingual datasets
TL;DR: We constructed 4 state-of-the-art datasets in 2 languages for instruction tuning with permissive licenses, by simply appending responses generated by open-weight LLMs to human-written instructions.

Abstract: Instruction tuning is crucial for enabling Large Language Models (LLMs) to solve real-world tasks.
Prior work has shown the effectiveness of instruction-tuning data synthesized solely from LLMs, raising a fundamental question: Do we still need human-originated signals for instruction tuning?
This work answers the question affirmatively: we build state-of-the-art instruction-tuning datasets sourced from human-written instructions, by simply pairing them with LLM-generated responses. 
LLMs fine-tuned on our datasets consistently outperform those fine-tuned on existing ones.
Our data construction approach can be easily adapted to other languages; we build datasets for Japanese and confirm that LLMs tuned with our data reach state-of-the-art performance.
Analyses suggest that instruction-tuning in a new language allows LLMs to follow instructions, while the tuned models exhibit a notable lack of culture-specific knowledge in that language.
The datasets and fine-tuned models will be publicly available.
Our datasets, synthesized with open-weight LLMs, are openly distributed under permissive licenses, allowing for diverse use cases.

Published as a conference paper at COLM 2025
Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
Youmi Ma² Sakae Mizuki Kazuki Fujiit.2 Taishi Nakamura
Masanari Ohil2
Hinari Shimada Taihei Shiotani! Koshiro Saito
Koki Maeda! Kakeru Hattori Takumi Okamoto' Shigeki Ishida'
Rio Yokotal23 Hiroya Takamura? Naoaki Okazakil.23
1 Department of Computer Science, School of Computing, Institute of Science Tokyo
2 National Institute of Advanced Industrial Science and Technology
3 Research and Development Center for Large Language Models, NII ye, swallowenlp., (kazakii)comp..in www.scttcc......
Abstract
Instruction tuning is crucial for enabling Large Language Models (LLMs) to solve real-world tasks. Prior work has shown the effectiveness of instruction-tuning data synthesized solely from LLMs, raising funda- mental question: Do we still need human-originated signals for instruction tuning? This work answers the question affirmatively: we build state-of-the- art instruction-tume siruuniion-tuningg... datasets sourced from human-written instructions, by simply pairing them with LLM-generated responses. LLMs fine-tuned on our datasets consistently outperform those fine-tuned existing ones. Our data construction approach can be easily adapted to other languages; we build datasets for Japanese and confirm that LLMs tuned with our data reach state-of-the-art performance. Analyses suggest that instruction- tuning in new language allows LLMs to follow instructions, while the tuned models exhibit notable lack of culture-specific knowledge in that language. The datasets and fine-tuned models will be publicly available Our datasets, synthesized with open-weight LLMs, are openly distributed under permissive licenses, allowing for diverse use cases.
Introduction
Open-weight Large Language Models (LLMs) have seen rapid advancements over the past year. Models like Llama-3.1 (Grattafiori et al., 2024) and Qwen-2.5 (Qwen et al., 2024) now achieve performance comparable to proprietary models such as GPT-4 (OpenAl et al., 2024) and Gemini (Google et al., 2024b). crucial technology behind this is instruction- tuning (Wei et al., 2022), a supervised fine-tuning process that equips models with abilities to engage in conversations and generate responses that satisfy human needs.
Unfortunately, even for LLMs whose weights are open, data used for instruction tuning are mostly closed. The data, typically consisting of (instruction, response) pairs, are keys to democratizing LLM: it remains an open question what kind of instructions are effective in enhancing LLMs' instruction-followw the instructions directly from the instruction-tuned models by prompting with a pre-query template. Datasets constructed through this process, named Magpie, have demonstrated remarkable effectiveness, being state-of-the- among all publicly available instruction- tuning datasets. The success of the self-synthesized data raises an important question: Do we still need human-originated signals for instruction tuning? In other words, are LLM-generated instructions truly superior to human-written ones?
abilities. Xu et al. (2025) proposed to "extract"
This work seeks an answer to the above question. To this end, we synthesize datasets from human-written instructions, as in Figure 1. For instructions, we extract user utterances

---

Published as a conference paper at COLM 2025
MAGPIE (Xu al., 2025)
This Work
As a Buby developer. what the best practions to while working with legacy code?
How many plansts in the solar systees?
There are eight planats in so system:
<jatars Iuades =d1
write python prugram that creates sqiine3
Python program Unik creates a SQJ rie3
instructions
<lotart header 1dl2
Open, Aligned LLM
W hen working with legacy code, it's resential
Users
Open, Aligned LLM
aapis pintaant
hesder idi>
instrictions
resgiotsss
responses
Figure 1: Overview of how Magpie (Xu et al., 2025), the existing state-of-the-arr dataset and our dataset are collected. Starting from instructions written by humans, we synthesize responses by open-weight LLMs.
Dataset
Instruction
Response
Language
# Instances
Dolly [Conover et 2023)
Human
Human
en.
15,011
OpenAssistan12 (Köpf et al., 2023)
Human
Human
128,575
OpenOrca (Lian et al., 2023)
Human
GPT-355, GPT4
et.
4,233,923
IMSYS-Chat- (Zheng al,, 2024)
Human
25
1,000,000
WildChat (Zhao et al., 2024)
Human
GPT-3.5, GPT-
en.
529,428
Alpaca (Taori al., 2023)
GPT-3.5
GPT-3.5
en.
52,002
Baize (Xu et al., 2023)
GITT-3.5
GPT-355
en.
210,311
UltraChat (Ding et 2023)
GPT-335
GFT-3.5
en. zh.
1,468,352
Evul-Instruct (Xu et al., 2024)
GPTT-3.5
GPT--3.5
70,000
Magpie-Ultna Ultra-v0.1 (Xu et al,, 2025)
Llama-3.1 1atmm.3.---055-nstttrs
1.lma-331 1amm-331-4005--nsttue
e..
49,052
Srspie-Pro-Filtered
et al. 2025)
Llama-3.1-700-nnsmn
Llama-3.1-70B-Isttrr
300,000
Ja-Self-Instruct (Sun et al, 2024)
GPT-4
GPT-4
ja.
52,002
This work
Propused-Llamas
Human
1ama-331-058-Instrae
453,055
Human
Gemma-2-27B..
en.
453,671
Proposed-Llmm--3I--a
Human
1lama-31--0600-Instma
453,802
Proposel-Gemma--Jaa
Human
Gemma-2-27B-rT
ja.
451,430
Table 1: Statistics of existing and our datasets. The column Language shows the major lan- guage in each dataset. We are the first to publish datasets with human-written instructions and responses synthesized by open-weight LLMs.
from LMSYS-Chat-IM (Zheng et al., 2024), a dataset composed of real-world conversations between humans and chatbots. For responses, we synthesize assistant utterances using open-weight LLMs, namely Llama-3.1 (Grattafiori et al., 2024) and Gemma-2 (Google al., 2024a) utilized for synthesizing Magpie (Xu et al., 2025). We fine-tuned four leading base LLMs on our datasets and compared their performance against those fine-tuned on Magpie. Three out of four models demonstrated superiority with our dataset, while the remaining model performed comparably. The results highlight the superiority of human-written instructions compared to LLM-generated ones. Pairing human-written instructions with LLM-generated responses thus leverages the strengths of both humans and LLMs, resulting in state-of-the-an thett-rtrrtttt datasets for instruction tuning.
A practical issue of our proposed method could be the difficulty of collecting large-scale human-written instructions, since existing resources are limited to English. To this end, we verify if our method is still effective in non-English languages based on machine translation. Specifically, we translated English human-written instructions into the target language and then prompted LLMs to synthesize responses. We chose Japanese as the target language and synthesized two datasets. Fine-tuning with the datasets significantly boosted the instruction- following abilities of all five base LLMs with different levels of Japanese proficiency. In total, this work built four datasets, with statistics shown in Table 1. We further analyzed how abilities transfer between languages. From the analyses, LLMs
the instruction-followii trained with Japanese datasets can solve surface-level editing (creative writing and roleplay) and region-agnostic knowledge domain (STEM) but are worse at instructions that require culture-specific knowledge.
In short, the contributions of this work are: (1) We construct and publish state-of-the-art instruction- instruction-tuning datasets with permissive licenses, allowing for wide range of use cases. Fine-tuning LLMss with our datasets yields the best performance among all publicly available
2

---

Published as a conference paper at COLM 2025
datasets. (2) We highlight the importance of human-written instructions collected from real-world human-chatbot interactions. By leveraging both human- and LLM-originated signals we obtain datasets with quality higher than those solely synthesized from LLMs. (3) We demonstrate that instruction-tuning in a new language equips LLMs with abilities to follow instructions, but it does not improve culture-specific knowledge.
Synthesized Data Generation
Starting from human-written instructions, we synthesize responses with open-weight LLMs to construct instruction-tuning datasets. Here we note the dataset as D {(I, R1), (12, R2), ... (In, Rn)), where lę and R₂ (k (1,2, .n)) represent an instruction and its corresponding response.
Instructions. We collected instructions I 12, (11 from LMSYS-Chat-IM (Zheng et al., 2024)2, dataset of one million real-world conversation records between human and LLMs. The dataset was selected as our foundation as it covers a wide range of topics, with a permissive license for both academic and commercial use. Notably, some of the conversations in LMMSSSCCat-IM are flagged as toxic; these conversations were excluded as our focus here is improve LLMs' instruction-folloiing abilities, i.e., how accurately LLMs can fulfiil user requests. After filtering out toxic conversations, we obtained 732,392 instances, from which the first user utterance selected as the instruction. Further removal of duplications left us with 453,889 instructions directly adopted for synthesizing responses.
Responses. Instances in LMSYS-Chat-IM include model responses. Nevertheless, we excluded these responses from our dataset because they were collected from a mixture of models, including both proprietary and open-weight LLMs. To ensure the quality of responses and a permissive license, we employed a leading open-weight LLM Å to generate response R3 corresponding to each instruction Ig. This work synthesized two datasets from the same instruction set, utilizing different LLMs to generate responses: Llama-3.1-405B. Instruct and Gemma-2-277-BB Responses from Llama-3.1-005--Ims.. the API provided by DeepInfra', while responses from Gemma-2-27B-IT were generated using vLLM (Kwon et al., 2023) on a single NVIDIA H100 SXM5 GPU. Responses with over 2,000 tokens were considered repetitions and were evicted from the final dataset.
were generated via
Statistics of our datasets, [Llama-3.1 ( Prop--Pooood--ii are reported Table 1. A dataset D synthesized by an LLM enables models to imitate how ft responds human- written instructions. We hereafter refer to the LLM utilized for response generation as teacher model and the LLM fine-tuned on the dataset as a student model.
Experiments
We validate if LLMs instruction-tunet on the Magpie (Xu et al., 2025) family. Notably, the dataset family has been recognized as state-of-the-art among public instruction-tuning datasets. We expect the results explain whether human-written instructions are still necessary for instruction tuning.
datasets outperformed their counterpart tuned
3.1 Configurations
Student Models. We adopted popular open-weight LLMs our test bed. We experi- mented on Qwen-2.5-7B (Qwen et al., 2024), Llama-3.1-8B (Grattafiori al., 2024), Gemma- 2-9B (Google et al., 2024a), and models from Huggingface Hub.
(OLMo et al., 2024). We used pre-trained
2nttps://huugingfe http:///wwwa..../caaaaaa///////a wttp:raw...mamaaaeeaeaaaeeaeea
1-485B- Instruct

---

Published as a conference paper at COLM 2025
MI-Bench
Base Model
SFT Dataset
Average
1st Turn
2nd Turn
Reference: performance of teacher models
1lama-3.1140511 --55------nstt
_.33
8.39 1.08
8.27
Gemma-2-277B
7.89 05
8.19 1.91.06
7.54
Lllmm-3.1-8B
LMSYS-Chat-IM-Raue
4.10_08
4.77
3.364.14
L.MSS-Chat-IM--Bes
4.15_008
4.89
3.364.13
Ultra-Chat
_.13
5.76
4.43
Magpie-Ultra-00.1
5.61
6.32
1.84
MaggieeGemmaa2-roo-Pe
5.82
6.30
5,24
6,40_06
6,86
6.86._11
5,90
Proposee-Lläma-31--m
6.82_
7.54
6.02..15
Proposed-Gemma-2255)
6.40-10
6.58
6.21
Qwen-2.5-7B
SlagpeeLLaama331-PPPMM-------
5.61_05
6.33
4.82+.07
Proposea-Lllämaa331--h
7.10_.12
7.677.06
6,47-21
Proposed--eemma-22nn
6.92_06
7.59
6.17
Gemma-2-9B
6.47-
7.234.03
5.65-.16
Proposed-Llämia-311--
6.79_006
7.39
6.131.06
Proposed-Gemma-22-En
6.58_06
7.11
5,97
OLMo-2-1124-133
Maspp--Lamm-3311ProMMTT-llll
6.85
7.164.02
6.50
Tülu
5.92_
6,1
5,38
Proposed-Llama-3.1 -lamaa3.1-nn
6.49
7.50
5.39
Soaposed-Gemma-222
5.44
6.95
3.75
Table Model performance evaluated on MT-Bench. t represents the official recipe utilized in the supervised fine-tuning of OLMo-2-1122-1338 with evaluation results of 10
reported. 3 out of LLMs fine-tuned on our datasets outper- form those fine-tuned on Magpie, the previous state-of-the-art dataset family.
Training. We conducted full-parameter Supervised Fine-Tuning (SFT) for every instruction-tuning experiment reported in this paper. Specifically, all model parameters were updated to maximize the log probability of response R₂ given instruction It. The train- ing process utilizes Huggingface's transformers library (Wolf et al., 2020), and distributed across 4 or 8 NVIDIA H100 SXM5 GPUs using DeepSpeed ZeRO (Rajbhandari et al., 2020). All experiments were finished in 24 hours except those with Gemma-2-99 which took 40 hours at maximum. A (Loshchilov & Hutter, 2019) was adopted the optimizer, with 61 = 0.9 and &2 = 0.95. The learning rate was adjusted following a cosine annealing schedule, which linearly increased to the peak at 2.5 x 10 in the first 10% number of steps, then smoothly decreased to 2.5 x 10-6, Each training process ran for epochs with batch size 512 across all experiments.
Evaluation. The performance of trained models evaluated Multi-Turn Bench (MT- Bench), a benchmark known for high agreement with human preferences (Zheng et al., 2023)4. MT-Bench consists of 80 high-quality multi-turn data spanning eight categories. MT-Bench follows
-udgges--- scheme, utilizing GPT-4 score model responses from 1 to 10. We adopted sp1-4-1106-previct as the judge for all experiments. For each instruction in MT-Bench, we sampled five responses from each model, scored each response with the judge, and reported the average score and standard derivation of five runs.
3.2 Main Results
This subsection explores how synthesized datasets improve LLMs' abilities to follow instructions. We compare the performance models fine-tuned using our datasets with two groups of datasets: baselines and strong competitors.
Baselines. Comparison with baselines serves a sanity check. To justify our efforts in synthesizing additional responses, is necessary to show that LLMs fine-tundd on
hhttps: t/gitbbb cottuu..commmmm/ www.l/owws////cceecceecccc
4

---

Published as a conference paper at COLM 2025
our datasets should outperform those fine-tuned on the baseline datasets. We included two baselines derived from the same dataset as ours, ie,, LMSYS-Chat-IM, by using the instruction-response pairs in their original form. For instructions with multiple responses, we either randomly picked one (noted as MASYs-Chat-IM-Ranaae judged by Gemma-2-277-IT (noted LMSYS-Chat-IMM-est))
or chose the best one
Strong Competitors, Comparison with existing datasets provides direct insights into answering the research question. Our major competitor is Magpie (Xu et al., 2025), report- ing state-of-the-art performance among all publicly available instruction-tuning datasets. Here we include three variants:
Magpie-Llama-3. 1llmmm-3.--rroMM Filtered, and Hagpie-Ultra-O0.1 synthesized with Gemma-2-27B-IT Llama-3.1 and Llama-3.114055 -.0.111-0---s-stuctt respectively. Additionally, we include UltraChat (Ding et al., 2023) as the representative of datasets synthesized with proprietary LLMs.
The results are presented in Table 2. We detail the results of Llama-3.1-8B tuned on each baseline, strong competitor, and dataset. For other student models, only the strongest competitor, i.e,,
is included.
Firstly, compared to the baselines, Llama-3.1-8B fine-tuned on Proposed- Proposed-Llama-3... 3.n leads its counterpart fine-tuned on LMSYS-Chat-IM--Bn by over 2.5 points. Notably, the baselines include responses generated by proprietary LLMs such the GPT and Claude families. Despite this, fine-tuning on baselines still underperforms ours. Responses in LMSYS-Chat 1M were collected in mid-2023, nearly one year before the release of the teacher models we adopted. The results show that thanks to the rapid advancements in open-weight LLMs over the past year, updating assistant responses with cutting-edge LLMs effectively enhances the utilization of SMSSYS-Cat--MM,
Next, compared to existing instruction-tuning datasets, Llama-3.1-88 fine-tuned on Proposed-llama-311-- shows clear advantages. Compared to UltraChat, which is synthe- sized using GPT-4, our datasets exhibit better performance across all cases. Against the Mag- pie family synthesized using open-weight LLMs, ie., the state-of-the-an datasets for instruc- tion tuning, the performance benefits are also observable. Note that Sroposed-Gemmaa and
and 1atpii--Utra-vO.1 share the same LLM for synthesized data generation. The difference is instructions: while Magpie datasets employ LLM-generated instructions, datasets utilize human-writted ones. The superiority of our datasets, hence, stems from human-written instructions, answering our research question: human-written instructions remain useful for instruction-tuning. Our approach is thus promising for building instruction-tuniin datasets by leveraging both human and machine resources.
-aaeed--lama-..11.
The superiority of our datasets can also be observed when fine-tuning Gemma-2-9B and Qwen-2.5-7B. For OLMo-2-1124-133,, our datasets slightly lag behind Magpie-Llama-3.1 Pro-MT-Filtered in overall performance but outperform it in the first turn. Notably, Magpie- Llama-3.--Poo-MT-Hiln
a multi-turn dataset, while ours are single-turn. The superior- ity in first-turn performance can thus be considered sufficient validation of the effectiveness of our datasets. In addition, 21-----1124413 fine-tuned on our dataset surpasses the performance of the official model after SFT. As reported in OLMo et al. (2024), the recipe for SFT is a mixture of recent datasets such as WildChat (Zhao et al., 2024) synthesized by GPT-4. Nevertheless, the performance is lower than OLM4--2-1224-13B fine-tuned Proposed-Llame Llama-3.1-mmn suggesting the effectiveness of our datasets.
3.3 Instance-Wise Effectiveness
LLMs fine-tuned our datasets outperform those tuned on existing ones. However, the scales of datasets differ (Table 1): our datasets contain approximately 100K more instances than -ate---laaaa----P---------------
(Xu et al., 2025). Here, decouple the effect from the dataset scale and the quality of each instance by aligning the number of training instances to that of the competitors.
5

---

Published as a conference paper at COLM 2025
SFI Data
# Instances
MT-Bench
Magpie- Llapa3--lam
300,000
6.40
Propose--Lama-3.1--1
300,000
6.54
Magpie-Ultra-v0.1
49,052
5.61
Proposed-Llama-31144
49,052
6.20
Magpie-Gemma2--PM
200,000
5.82
Proposed-Gemma-2-in 200,000
6.50
Table 3: Performance of Llama-3.1-8B evaluated on MT-Bench when the number of training instances is aligned. Fine-tuning on our dataset, even downsampled, outperforms those fine-tuned on the magpie family.
Settings. We downsampled (1) Proposed-LLam--33-- to compare with Magpie-Llama- 3.1-Pro-MT-Filtereed (2) Proposed-LLama-3.11 to compare with Magpie-Ultra, and (3) Proposed-Gemma-22 to compare with Digpie-Gemma-2-Prro
For downsam- pling, we followed the same filtering strategy as Xu et al. (2025) to assign quality score to each synthesized response with LLM, then select the top-N (instruction, response) instances for instruction tuning. For Llama-based datasets, we adopted Llaama3.1-700-Instti automatic scoring and selected the top 300K instances; for Gemma-based datasets, we adopted Gemma-2-277-IT for scoring and selected the top 200K instances, The results Llama-3.1 1--8B- are reported Table
for
Results. Even when the number of training instances is controlled, Llama-3.1 3.BBBB fine- tuned with datasets still outperforms those tuned with Magpie. The performance gap is consistent for all settings, verifying the instance-wise effectiveness of our datasets. Notably, the dataset pairs in the last two groups utilize the same LLM for generating responses, and the number of instances is controlled to be the same: The performance gap thus arises solely from differences the instructions. This further validates the effectiveness of human-written instructions in LMSYS-Chat-IM..
Synthesizing Data in Other Languages
Section witnessed the effectiveness entes of our datasets in instruction tuning. The data con- struction approach, however, is adaptable for synthesizing datasets in other languages. This section examines whether this approach can generate high-quality datasets for non-English instruction-tuning, using Japanese as the representative.
To construct the instruction-tuning dataset, we translated the instruction set I from Section into Japanese with the DeepL translator API Given the translated instructions, Llama-3.1- 405B-Instruce and Gemma-2-27B-II were utilized for response generation. This process yields (Llama-3.1 (emmma-2--Proposede in Table 1.
4.1 Configurations
The configurations are identical to those in Section 3.1, except for the following differences.
Student Models. OLMM----122--33B is excluded due to its limited proficiency in Japanese. (Fujii et al., 2024; Okazaki et al., 2024) and 11m-
Instead, we added Llama-3.1-Saallow- jp-3-13b (Aizawa et al., 2024), where the former undergoes continuous pre-training from Llama-3.1-8B on Japanese corpus, and the latter trained from scratch with a special focus on Japanese capability.
Evaluation. We adopt Japanese MT-Bench, a variant of MT-Bench that reflects Japan's language, education, society, and c Japanese is one of few languages with reliable
chttps::/www.co dewpl. com/ /ttps/sattraaattrrrrrrr hnttps://wandb. wwwdbbaaaaaann 11m- leaderboard/reports//ejiii -LM--No Vml 1dzo2MTkyMTU

---

Published as a conference paper at COLM 2025
Model
SFT Data
MT-Bench
Ja. char. ratio
Reference: performance of teacher models Gemma-2-27B-IT
models
7.10+.06
0.66
1aama-3.1-405B-Insst Llama-3.1-80
6.79+.17
0.67
official
4.65
0.65
la-Self-Inst
3.85
0.77
Proposed-Gemma-ee
5.64
0.65
Hopposed-Llama---
4.42+
0.66
Qwen-2.5-7B
official
6.13+.10
0.82
Ja-Self-Inst
4.24
0.73
Proposed-Gemma-22
6.39
0.67
Proposed- Bapoosdd-Llama-.111
5.40+.09
0.66
official
5.33+.07
0.68
Ja-Self-Inst
4.53
0.73
Proposed-Gemma-22
6.23
0.65
Proposed-Llamee Llama-3.1-ja
5.18+.05
0.69
Gemma-2-9B
official
6.75..10
0.66
Ja-Self-Inst
3.98
0.72
Sypposed-Gemma-2111
6.49
0.66
Saaposed-Llama-333
5.14
0.67
Im-jp-3-133
official
5.18+.08
0.69
Ja-Self-Inst
3.71
0.73
Proposed-Gemmat
5.38.+.07
0.64
Proposed- -laaned-dLa
4.74+.21
0.71
Table 4: Model performance evaluated on Japanese MT-Bench. 4 out of LLMs show better performance when fine-tuned dataset than their official post-trained release.
evaluation baselines, which motivated our choice. In addition, we measure the Japanese character ratio (Ja. char. ratio in Table 4) in the generated outputs to assess the input- output language consistency. A low Japanese character ratio indicates that the model generates output in off-target languages (Sennrich et al., 2024). During pilot experiments, empirically found that Japanese character ratio below 0.60 indicates noticeable presence of content written in off-target languages.
4.2 Results
Competitors. Unlike English, where extensive research has been conducted on instruction tuning, Japanese remains relatively underexplored in this area, with Ja-Self-Insttuct being the most recent research introducing a dataset synthesized with GPT-4 (Sun et al., 2024). We thus compare the performance of LLMs fine-tuned
datasets with (1) their counterparts fine-tuned on Ja-Self-Instruct; and (2) their official post-trained versions. Experimental results are presented in Table including the performance of GPT-3.5, GPT-4, GPT-4o, and the teacher models for reference.
Firstly, models fine-tuned with our datasets outperform those fine-tuned with Ja-Self- Instruct. As Ja-Self-Instruct is synthesized with GPT-4 (gpt-4-0613), a proprietary LLM, this highlights the advancement of open-weight LLMs in the past year. Notably, when com- pared to their officially post-trained counterparts, models tuned on our datasets still exhibit comparable or better performance. Specifically, Gemma-2-90 tuned on our dataset slightly underperforms the official Gemma-2-9B.1 whereas all other models surpass their official counterparts. Therefore, the proposed approach of combining human-written instruc- tions with LLM-generated responses for instruction tuning is effective for non-English languages as well, with instructions translated from English into the target language.
Next, we observe that the dataset synthesized with Gemma-2-27B-I7 functions much better than that synthesized with Llama-3.1-405B- 3.1-4055-Instruct. While (emma-2-22B-11 merely outper- forms
by 0.32 points, the gap between student models fine-tuned on the corresponding synthesized datasets is larger. This is counterintuitive, as we have witnessed the performance of the student model being proportional to that of the teacher

---

Published as a conference paper at COLM 2025
Paapie-Llama-3.-PPro-MTT---
3.1-En
1520
1500
Counts
1000
530
463
500
243
192
184 150 133 97
185 193
176 168 166
125 115 114 article write
128
mathernatics
problem salve
geometry
physic
information request
algebra
measurement
data analysts
word problem
algorithm
code example
information reguest
information retrieval
company introduction
python programming
grammar check
location
mathematics
fact chack
Figure 2: Counts top-10 most frequent topics in Magpie-Llama-3. --r-mm------r------llleeee (Xu et al., 2025) and Sipposed-Llama3.111111 Topics in Magpie are skewed to mathematics, while those in the proposed dataset are more balanced.
model in English. This indicates that the effectiveness of model imitation for non-English languages does not necessarily align with the findings in English.
In terms of the Japanese character ratio, models trained on our datasets consistently maintain safe level above 0.60, matching that of their teacher models. Two native-level Japanese speakers with PhD degrees reviewed the responses generated by the teacher models and found off-target language in fewer than 0.5% of cases. Given that models fine-tuned on our datasets exhibit a similar Japanese character ratio as their corresponding teacher models, we conclude that the risk of these models producing responses in off- fff-tarret languages is low.
Analysis
5.1
Quantitative Analysis
We have shown that human-written instructions in our proposed dataset are superior to the LLM-generated instructions in Magpie (Xu et al., 2025). To better understand their difference, we dive into the instructions in Saaie--aamaa3.--ro-MTT--- 3.1-En. We use InsTag (Lu et al., 2024) to annotate each user instruction with its semantic intent, referred to as the topic, examine differences between Magpie and our dataset.
and Proposed-Llan Llama--
Figure 2 presents the top-10 most frequent topics and their counts. In Magpie, we observe a highly skewed distribution: the most frequent topic, namely mathematics, appears 1, 1,520 nearly three times more than the second most frequent topic (530 times). In contrast, our proposed dataset exhibits a more uniform distribution across the top-10 topics, covering diverse categories such as article writing, grammar checking, and general reasoning. This analysis reveals that Magpie's 's instruction set heavily biased toward mathematical topics, whereas our dataset provides broader and more balanced topic coverage. Such diversity likely contributes to the superior performance of models fine-tuned on our data. We also observe a correlation between the category-wise performance of fine-tuned models and the topic distributions quantified in this section. Further details are provided Appendix E.
5.2 Cross-Lingual Analysis
Our datasets enable cross-lingual analyses of instruction tuning. Here, we analyze how language capabilities are enhanced after instruction-tuning (TT) and distinguish them from those enhanced after continuous pre-training (CPT). Pretraining LLMs per language from scratch demands extensive data and computation resources (Zeng et al., 2023; Aizawa et al., 2024); to avoid this, research has been conducted to adapt LLMs to underrepresented languages via IT (Chen et al., 2024; Shaham et al., 2024; Zhang et al., 2024). Concurrent to this work, Wang et al. (2025) have reported that incorporating CPT before IT better improves the capability in the target language. However, the role of CPT and in enhancing language capabilities remains unclear. We decouple the language capabilities into two categories,
8

---

Published as a conference paper at COLM 2025
Llama+TT(EN) Ulam+ITTJJ)
Liama+ +CPT+IT(EN)
Liama +CPT+IT(A)
Liama Lisma+CPT
(amo
humanities
0.8
(Ia
Uama+GFT+ITTA)
math
extraction
0.62
0.6
0.560.56
0.50
0.490.49
0.43 0.45
1 2
0.4
reasoning
coding
0.2
0.120.13
0.0
roleplay
writing
MT-Bench
Academic Tasks
stem
(a) Scores of MT-Bench are normalized to 0-1. IT(A) enhances performance on MT-Bench but does not help academic tasks.
(b) IT helps writing, stem, and roleplay, while CPT contributes to humanities.
Figure 3: Model performance on each sub-category of Japanese MT-Bench.
namely language understanding and language utilization, and investigate how CPT and IT' contribute to acquiring these capabilities.
Settings. We fine-tuned Llama-3.118) and
with the Gemma-2
versions of the synthesized datasets. Compared Wang et al. (2025), which employs is the state-of-the-ae Japanese LLM obtained through CPT from Llama-3.1-8B, and our datasets are the state-of- the-art for IT. For evaluation, we adopted the metrics from Fujii et al. (2024) to assess LLMs' proficiency in language understanding' and used the Japanese MT-Bench to evaluate their ability in language utilization. The results are presented in Figure 3a.
LlaMA-2 as the test bed, our setup is strong: Alama-3.1-Swallowee
Language Understanding. On academic tasks, observe significant performance im- provement in models that underwent CPT. In contrast, models without CPT show only subtle changes in performance, even after IT with the Japanese dataset. These results suggest that CPT is crucial in equipping LLMs with fundamental language understanding abilities.
Language Utilization. On MT-Bench, dataset (Lam==CCT) scores the highest. Interestingly, Llama-3.1 fine-tuned on the Japanese dataset (Llama+IT(A)) and Llama-3.1-Swallow fine-tuned the English dataset ILlama+CPT+ITTER achieves comparable performance on MT-Bench, while the former underperforms the latter the academic task. The result indicates a lack of factual knowledge about the Japanese language in Llama-3.1; nevertheless, the model follows Japanese instructions as well -awwalw. fine-tuned the English cltasett.
fine-tuned on the Japanese
Task-Wise Performance. The performance of fine-tuned models on each sub-category of the Japanese MT-Bench as in Figure 3b. The figure portrays clear contribution of CPT in the humanities category, which includes topics about Japanese history, finance, and social manners. This indicates that following instructions related to cultural-specific topics requires CPT. The findings correspond to those reported in Romanou et al. (2025), saying that multilingual LLMs struggle with cultural questions, especially languages not present in pre-training. Meanwhile, for categories that rely on global knowledge already present in the pre-trained LLM (i.e., stem) or those categories that require no factual knowledge (i.e., roleplay, writing, and extraction), IT works sufficiently well on its own. Capabilities to follow instructions such categories are likely transferrable across languages.
The benchmark covers wide range of tasks such as Japanese question-answering summarization, and translation. We only report the average across all tasks.

---

Published as a conference paper at COLM 2025
6 Related Work
Instruction-tuning datasets are typically sourced from three origins: humans, human-LLM interactions, and LLMs (Zhang et al., 2023).
Humans. Wei et al. (2022) introduced the first instruction tuning dataset as a collection of academic tasks in NLP research. They constructed a dataset from existing datasets, covering tasks such as question answering, translation, and summarization. These language resources are of high-quality, but do not represent the real-world use cases of current ÁI assistants. Recently, efforts have been made to construct instruction-tuning datasets that better reflect real-world needs. Köpf et al. (2023) recruited crowd-source workers worldwide to collect a conversation corpus simulating human-assistant conversations. To ensure the expertise of annotators, Conover al. (2023) employed experts working in the data science domain. These datasets face challenges as being simulations, resulting in a gap between constructed data and real-world needs.
Human-LLM Interactions. To build datasets that fit into real-world settings, efforts have been made gather chatlogs between humans and AI assistants in the wild. Zheng et al. (2024) collects human-AT conversations from web service called Chatbot Arena with 25 LLMs. Zhao et al. (2024) is a concurrent work with the Al assistant restricted to the GPT family. While these datasets consist of diverse user instructions, the quality of assistant response could not be ensured. Moreover, responses generated from proprietary LLMs could not be utilized for LLM development due to the restrictiveness, in particular around the licensing of model outputs. This work values the diversity of user instructions in chatlogs by pairing them with responses generated from state-of-the-art open- LLMs.
LLMs. The community has been focusing on automatically constructing data with pro- prietary LLMs. LLMs can generate instruction-tuning datasets based on minimum set of high-quality human-written (instruction, response) pairs (Taori et al., 2023; Wang et al., 2023; Sun et al., 2024). The generated dataset can be improved by utilizing LLMs to rewrite the instructions (Xu et al., 2024) or uttllizing multiple LLMs to chat with each other (Ding et al., 2023; Xu al., 2023). However, synthesized with proprietary LLMs, the usage of these datasets is restricted due to license issues. More recently, Xu et al. (2025) synthesized datasets using open-weight LLMs by generating completions from system prompts. The datasets are purely "extracted" from LLMs, with limited investigation into the appropriateness and quality of the LLLM-generater instructions. Moreover, whether this approach is effective for mon-English languages often underrepresented in LLM training data remains uncertain. We compare the effectiveness of LLM-generated instructions with human-writted ones and propose strategy for synthesizing cittrucctiottuuinn data for non-English languages.
7 Conclusion
Our work constructed datasets by leveraging open-weight LLMs to generate responses to human-written instructions. Three out of four LLMs fine-tuned with our datasets out- performed those fine-tuned with existing datasets built by self-synthesis, validating the usefulness of human-written instructions. Platforms like Chatbot Arena are thus essential for LLM development, serving not only as sites for manual evaluations but also as sources of real-world conversational data.
We applied the same strategy and synthesized state-of-the-art datasets for Japanese. With the datasets, we explore the cross-lingual transferability of instruction tuning. Abilities transferrable across languages turn out to be those requiring no factual knowledge, global knowledge already present in the pre-trained models. Fundamental knowledge of target language can only be enhanced by continuous pre-training.
While not explored in this study, we plan to enhance the utility of our datasets by incor- porating support for safety and multi-turn conversations. Another future direction is to construct and evaluate the utility of synthesized datasets with instructions derived from Human-writted instructions that have been empirically validated as effective.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
This paper focuses more on improving the usefulness than the safety of LLMs. As result, although we filtered out instructions flagged as toxic in LMSYS-Chat-IM, we acknowledge that some toxic contents still remain in datasets. LLMs trained with our datasets should undergo further fine-tuning or reinforcement learning to ensure their safety and fairness. Derived from LMSSS-Chat-IMM (Zheng et al., 2024), all instructions in our datasets have tokens that could reveal Personally Identifiable Information (PII) masked out. For LLM-generated responses, we leave them as-is, as the synthetic data generation process does not introduce any additional information about real individuals.
Acknowledgments
This work was supported by JSPS KAKENHI Grant Number 25H01137. This work was also supported by JST K Program Japan Grant Number JPMJKP24C3. The research and development of the large language model Swallow has been supported by the AIST Project "Research and Development on Generative Al Foundation Models in the Physical Domain". It was also supported by a project from the Ministry of Education, Culture, Sports, Science, and Technology (MEXT) aimed at "establishment of research and development centers to ensure the transparency and reliability of generative AI models", along with other contributions. This research was conducted using the ABCI large-scale generative AI research and development support program, and the TSUBAME 4.0 supercomputer at Institute of Science Tokyo. We thank Sangwhan Moon for his valuable suggestions on improving the manuscript.
References
Akiko Aizawa et al. LLM-jp: A cross-organizational project for the research and develop- ment of fully open Japanese LLMs. arXiv:2407.03963 2024. URL https://doi. org/10. 48550/arXiv.. 2407.03966
Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. Monolingual or multilingual instruction tuning: Which makes better alpaca. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Compu- fational Linguistics: EACL 2024, pp. 1347-1356, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https: /atp//aaaaaaaaaa....aa/aa org/ 2024. findings-eacl. 90/.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's d's first truly open instruction-tuned 11m, 2023. URL https://www/ databri wiw.brik....................a... www.blog/ http///lgg22222////// 04/
httpr/uccciiiotune-----e- 11m.
et al DeepSeek-AL. Deepseek-rl: Incentiivizing reasoniin capabiiity in llms via reinforcement learning, 2025. URL https://arxiv..er httpp:://aarii..rr////////..111 12948.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https: //tpnrevi//peeeiiewwe
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki lida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. First Conference on Language Modeling (COLM), 2024.
Gemma Google et al. Gemma 2: Improving open language models at a practical size. arXiv:2408.0011 2024a. URL https:ht//xiiiiii abs/2408. 00118.
11

---

Published as a conference paper at COLM 2025
Team Google et al. Gemini: A family of highly capable multimodal models, 2024b. URL www.:///g////a/t/e 2312. 11805.
Aaron Grattafiori et al. The Llama 3 herd of models. arXiv:2407.2178, 2024. URL https: //arxiv.org/abs//2000 21783.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick. Openas- sistant conversations -
democratizing large language model aligmment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL /ttps:://ppereviiii net/forum2id=vSSogee
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:: http://hhggggggeeeeeeeee wwtppooo//poeeeeee ///pppnnOOrr//ppeOrraa.a 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Ninth International Conference on Learning Representations (ICLR), 2019. URL http:tttts//opeeeiiiiiii
net//forumiddBkkBRRccQyy..
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httt:::://ppeneeiiiee
Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki lida, Masanari Ohi, Kazuki Fujüi, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. Building large Japanese web corpus for large language models. In First Conference on Language Modeling (COLM), 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Ji- acheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2024. URL
https://arxiv. wwwpaaawwi..oogrr///aa/aaaaaaaae 2501.00033
OpenAl et al. GPT-4 technical report. ArXiv:2306.02707,,00 2024. URL http::////arri .87777
Qwen al. Qwen2.5 technical report. ArXiv:2412.55115, 2024. URL http:::://axxitooorr abs/2412. 15115.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networkingg Storage and Analysis, 2020. ISBN 9781728199986.
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karisson, Eldar Khalilov,
12

---

Published as a conference paper at COLM 2025
Christopher Klamm, Fajri Koto, Dominik Krzemiñski,, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia soltani moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut. INCLUDE: Evaluating multilingual language understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //wp///evei..w...///mmo wttpeviwwwwweeteeeee meet/forumiidkk3giieeeee
Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. Mitigating hallucinations and decoding. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 21-33, St. Julian's, s, Malta, March 2024. Association for Computational Linguistics. doi: 22116653/1122224.cmm
off-target machine translation with source-contrastive and Language-contrasee
URL https: / /ttp://hoaaaaaoooycccooo.o org/2022
t.hort.4.. 4/.
Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. Multilingual instruction tuning with just pinch of multilinguality, In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, Pp. 2304-2317, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/vi ww/////1.22/////// /tt////...iniiinss-aa... //aclanthology, org/ 2024. findings- acl. 136/.
URL https:
Yikun Sun, Zhen Wan, Nobuhiro Ueda, Sakiko Yahata, Fei Cheng, Chenhui Chu, and Sadao Kurohashi. Rapidly developing high-quality instruction data and evaluation benchmark for large language models with minimal human effort: A case study on japanese. In 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLIN)), 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following 2023.
LLaMA model. https://githuu. http:////gituuu.commameeeeee www wwwpssuuaaaassttooooarooooeo
Shumin Wang, Yuexiang Xie, Bolin Ding, Jinyang Gao, and Yanyong Zhang. Language adaptation of large language models: An empirical study on LLaMA2. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Lingnistics, pp. 7195-720208 Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. wwwphhhllgy.oooooooooooooo wtrg/2225.com cowing main.480/.
URL
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 13484-13508. Association for Computational Linguistics, 2023. doi: 10. (0.18633/v1/2000 acl-long.754. URL https: httpss:///cllahhllogg/// org//023....// acl-long. 754/-
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Tenth International Conference on Learning Representations (ICLR), 2022. URL https: //openreview. wwwpeerrvieenneee/fooo www..eeeoooruuiddgggeroooooooe
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020. URL https: org/abs/1910.0377
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardL Empowering large pre-trained language
13

---

Published as a conference paper at COLM 2025
models to follow complex instructions. In Twelfth International Conference Learning Representations (ICLR), 2024. URL https:/ /ttpp//eeeeiie.eee/ooo/r///eeee/
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6268-6278. Association for Computational Linguistics, 2023. doi: main.385/
URL https::/aclanntoology
org/ 2tg/2023.emme
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran,, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. In Thirteenth International Conference on Learning Representations hetfforummdd=Pkkk7
(ICLR), 2025. URL
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang, GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttps:///perreview.. wwwprreewwweeeeeorunm.... aee//forumtd===wwrrrr
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et Instruction tuning for large language models: A survey, arXiv preprint ar wrXiv:2308.10792. 2023.
Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. PLUG: Leveraging pivot language in cross-lingual instruction tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7025- 7046, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 /v1 www.024.cllooog.....
URL https://aclantthlll
wwt.orgg//2222 acl long. 379/.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- chat: Im chatGPT interaction logs in the wild. In Twelfft International Conference on Learning Representations (ICLR), 2024. URL https: httpaarreeeeeeeeteooorrrm//-----
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2023. URL https: /ttp::://ppmmee /ttp///eeeee/eeeeee/eeeeeee wwwperreieee.ne/fforeoe
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ton Stoica, and Hao Zhang. LMSYS-chat-Imm large-scale real-world LLM conversation dataset. In Twelfth International Conference Learning Representations (ICLR), 2024. URL h //penreiie.....ww..
net/forummiddBOFDXXfwrr
A License Information
Our datasets are granted for both research and commercial uses, in compliance with the LMSYS Dataset License Agreement, together with the Llama 3.1 Community
License Agreement for Llama-based datasets and the Gemma Terms of Use for Gemma- based datasets.
hhttps://wwe caamaa.commlaaa comm/lamma_111 1/1icens//1ccee/////e 9htpp: /etttp::/ai.googgee.. google. dev/ gemma/ terms
14

---

Published as a conference paper at COLM 2025
MI-Bench
Base Model
SFT Dataset
Average
1st Turn
2nd Turn
Reference: performance of teacher models 1lama-3.1-405PP -nnt-suuttt
_.33
8.39 1.08
8.27
Gemmma---27Bb
7.89 05
8.19 8.19
7.54:.10
Qwen-2.5-7B
Magpie-Ultra-vO.1
3.56_.19
4.21
2.964.26
Haspiee-Gemma2-Pro-iilteeee
6.00_00
6.30
5.664.14
Daapie-Llama-31-PPo-MMTE
5.61 5.61-05
6.33
4.82.107
Proposed-Llana-3))
7.10
77.67
1.06
6.47 1.21
Proposed-Gemma-22Enn
6.92
7.59 4.7
6.17
Gemma-2-9h
Magpie-UUlra-v001
5.99_.10
6.89
5.09
Moggie-Gemma2-1Pee
5.81-07
6.63
4.83
6.47
7.23+.03
5.654.16
6.79_.06
7.39
6,13
Proposed-Gemma-2-En
6.58_06
7.11
5,97
OLM--2-1124-13B
Magpie-Ultra-ve
3.14
4.42+.06
1.794.06
Srapii-Gemmm22-to--P-ee
4.20-.15
5.664.08
2.55..27
Soapie-Llamaa3.-PPo--ME--
6.85
7.164.02
6,50
Tülu
5.92
6.41+11
5.38
Proposed-Lláma-33
6.49
7.504.04
5,394.18
Semposed-Gemma-2-11
5.44-.12
6.95
3.754-21
Table 5: Model performance evaluated on MT-Bench. t represents the official recipe utilized in the supervised fine-tuning of OLMo-2-1124-113 with evaluation results of allenna/OLMMo22-124-111-- reported. This table is complementary to Table 2.
Teacher
Gemma-2
OLMo-2
Teacher
Gemma-2 Qwen-2.5
LLM-jp-3
Llama-3.1
Qwen-2.5
Llama-3.1
Llama- 3.1 3ww-Swollow
humanities
humanilies
extraction
math
extraction
1234
2
reasoning
coding
reasoning
coding
roleplay
writing
roleplay
writing
stem
stem
(a) Results on English MT-Bench. The teacher
(b) Results on Japanese MT-Bench. The teacher
model is 1lama-3.1-405--mmar
model is Gemma-2-27B-N2
Figure 4: Radar chart of the performance of models for each task category in MT-Bench.
B Additional Experiments
This section presents additional experiments not included in Table 2, shown in Table 5. Results for MB-yS-ChattMMMRanddae LMSYS-Chat-IMMBB (Zheng et al., 2024), and Ultra- Chat (Ding et al., 2023) are omitted, they serve weak baseline intended for sanity checks; Models fine-tuned on these datasets significantly underperform those fine-tuned on Magpie (Xu et al., 2025) and proposed datasets.
Table shows trends consistent with Table 2, confirming the superiority of our proposed datasets over Magpie.
15

---

Published as a conference paper at COLM 2025
SFT Data
Instances
MT-Bench
base model: Llama-3.1-8B
Ja-Self-Instruct
52,002
3.85
Proposed-Gemma-22
52,002
base model: 1lm-jp-3-13b
Ja-Self-Instruce
52,002
4.91
Proposed-Gemma-
52,002
5.64
base model: Llama-3.1-Swallow-B88
Ja-Self-Instruct
52,002
4.53
Proposed-Geema--JJa
52,002
5.71
Table 6: Performance of LLMs evaluated on MT-Bench when the number of training in- stances is aligned. Fine-tuning on our dataset, even downsampled, outperforms those fine-tuned on Ja-Self-Instruct.
C Category-Wise Effectiveness
MT-Bench includes instructions across eight categories. Here, examine how our collected datasets enhance performance in each category. Evaluation results for LLMs tuned on Pooposed-Llama-..11 and Proposed-Gemma-22 tively. The performance of teacher models, i.e., LLMs utilized for generating responses, is also included to see how the student models inherited the capabilities.
shown in Figure 4a and 4b, respec-
Firstly, we observe a similar tendency in both radar charts that our datasets effectively enhance LLMs' abilities in writing, coding, and extraction. Meanwhile, a common difficulty has been observed in inheriting the ability to reason. The results indicate that imitating the response generated from stronger LLMs can be insufficient equip weaker models with reasoning abilities.
On the other hand, in the math category for both languages, we observe that Qwen-2.5 leads in performance. The base models in the Qwen series are known for their strength in math tasks. Similarly, in the Japanese humanities category (Figure 4b), I.lma-3.1-Swalll outperforms Llama-3,1 and demonstrates the best performance among all student models. Llama-3.1-Swallov which was continuously pre-trained with Japanese corpus on Llama-3.1, gains deeper understanding of Japanese local culture than Llama-3.1 (Okazaki et al., 2024; Fujii et al., 2024). These results suggest that the inherent advantages in reasoning abilities and knowledge from base models persist even after instruction-tuninng
Based on the above observations, we conclude that instruction-tuning by imitating the responses of strong L.MSs is effective for tasks related to writing reformatting. However, its effectiveness is limited for tasks that require deep knowledge and reasoning. these cases, the advantages of the base models remain even after instruction-tuning
D Instance-Wise Effectiveness: Japanese
Settings. To match the dataset scale of Ja-Self-Instruct, we randomly sampled 52K (instruc- tion, response) pairs from Proposed-Gemma-2JJ. LLMs were fine-tuned this subset and compared with those fine-tuned on Ja-Self-Instruct. The results are shown in Table 6.
LLMs fine-tuned on our dataset outperform those fine-tuned on even when the number of training instances is aligned to the same. Compared to English settings, the performance gap between datasets and the existing ones is more pro- nounced in the Japanese settings, underscoring a significant shortage of Japanese resources for instruction-tuning.
Results.
16

---

Published as a conference paper at COLM 2025
Category-wise Score
Magpie-Llama-311 -araaaaaaa----------.-.- -r-----eeeed
Proposed 3.1EEn3
Score
10
6.406.82
7.52
8.30
7.81
7.88
5.72
5.90
6.41
7.76
6.72
7.30
8.52
4.90
3.93
Average
Coding
Extraction
Humanities
Math
Reasoning
Roleplay
STEM
Writing
Category
Figure 5: Category-wise scores of models fine-tuned on -aappi----amaa--Proo---e- and hoppoedd-Llama-1 The performance gap is larger in categories underrepresented in Magpie's topic distribution.
E Category-Wise Quantitive Analysis
Corresponding to Section 5.1, here we report the category-wise performance of Llama-3.1-8B -aaee--Llaaaa3.-----M------
trained
and Proposed-Llama-3.111 in Figure 5.
As shown the plot, models trained on our dataset outperform those trained on Magpie in the Humanities and Roleplay categories, while the advantage diminishes in Reasoning and Math. This trend may reflect the topic distributions discussed in Section 5.1, where our dataset is more balanced but Magpie is heavily skewed toward mathematics. The skew likely boosts performance in Reasoning and Math, consistent with recent findings on the close relationship between mathematical and reasoning abilities (DeepSeek-A) 2025).
17


Reviews:
[{"review_id": "axyUPKrI0p", "reviewer": "Reviewer_7i5D", "summary": "The authors introduce a method to generate instruction-tuning datasets using human\ninstructions from a real-world dataset. The method is tested in several LLMs and performs\nwell. The method is evaluated in English and Japanese.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written and is easy to follow. The authors test on a variety of\nLLMs and test in English and Japanese. The method compares favorably with Magpie,\na SOTA instruction-tuning dataset.", "reasons_to_reject": "A relatively minor weakness is that only one alternative instruction-tuning dataset,\nMagpie, is examined.", "questions_to_authors": "Line 106 – Should this say “maximize log probability” instead of minimize?\nLine 212 – Should be “one of few”\nLine 315 – “Four” should not be capitalized", "extra_scores": {}}, {"review_id": "Lr9g1S8b0h", "reviewer": "Reviewer_WhHW", "summary": "The main research question that is explored in the paper is whether \"LLM-generated instructions are better than human-written ones\" . The paper presents a dataset that is created using human-written instructions from an instruction dataset coupled with generated responses from open LLMs. The results of fine-tuning are compared to existing synthetic datasets such as Magpie (Xu et al) among other baselines. The paper shows that instruction tuning with the proposed dataset achieves superior or comparable performance to the baselines and prior work. A controlled experiment in terms of the dataset size by downsampling the proposed dataset to the same size of Magpie is also conducted and shows superior results. It is observed that finetuning with the proposed dataset also has gains to Japanese.\n\nFurthermore, machine translation is employed to translate the instruction datasets to Japanese to evaluate performance of the method is Japanese.\n\nHowever, the key research question remains largely unanswered. A more controlled experiment with the human-written instructions of the same dataset substituted with its synthetic counter part which could be generated using the procedure adopted in Magpie would have been a more appropriate baseline.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The research question is interesting. It is valuable to the community to know the tradeoffs of various approaches to generate synthetic instruction tuning datasets that are critical for aligning language models.\n\n* The exploration of an additional language beyond English in the experiments is a value add.", "reasons_to_reject": "* The main research question remains largely unanswered since the baselines are derived from other data sources. A more controlled experiments using the same seed LM-Sys-Chat dataset with synthesised instructions is needed to answer the research question of whether human-written instructions are more valuable that machine generated instructions.", "questions_to_authors": "* Adding more languages beyond Japanese would  enable authors to make more generalized conclusions regarding finetuning on non-English languages.", "extra_scores": {}}, {"review_id": "Nhbd2gsoqu", "reviewer": "Reviewer_DRap", "summary": "This work contribution is two-fold: 1) it studies the performance of instruction-tunining of open-source LLMs on synthetically generated data by large open-source teacher model. It gets inspiration from Magpie [1] work that demonstrates that it is possible to extract instructions and responses from pretrained open-source LLMs by prompting with  basic templates.  This work suggest to use human-generated instructions, and extract only responses from pretrained open-source LLMs. Authors demonstrate that relying on human generated instructions lead to superior performance comapred to Magpie approach, and can also sometimes lead to better performance that models tuned on human-generated responses.  Second part of this work extends such synthetic data generation to Japanese language and demonstrates that it is possible to build on strong teacher models distillation to create competitive japanes instruciton-tuned models.  It also perform some interesting analysis on the importance of continous pretraining and instruction finetuning in non-english language (Japanese in this case). \n\n\n[1] Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing Xu et al. ICLR 2025", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work reports results outperforming Magpie work, which might be of interest for future LLMs improvements with synthetic datasets\n- Results on Japanses instruction finetuning are interesting, as well as analysis on the continout pretraining + instruction tuning across different domains in japanese.\n\n---\nScore updated after clarifications provided by the authors", "reasons_to_reject": "- It seems to me that the findings of the first part of this  work are quite marginal compared to previous studies in synthetic data generation [1].  More in-depth discussion and analysis of the results comparing Magpie to the proposed method would help to strengthen this contribution: eg perform some manual analysis of the examples where proposed method outperforms magpie and vice versa would be interesting. It could also be possible to trace back the predicitons to the training data similar to OlmoTrace work. That would allow to perform even more rich analysis. \n\n- The finetuning on Japanese synthetic data and analysis is interesting, but clearly lacks more in depth analysis and comparison to previous studies in multilingual instruction tuning [3,4,5]. Assessment of reliability of LLM-as  a judge for Japanese seems important as well: eg. how significant would be the difference of 0.2-1 in scores if we take llms-as-a-judge potential error into account?\n \n\n[1] Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing Xu et al. ICLR 2025\n[2] Multilingual Instruction Tuning With Just a Pinch of Multilinguality Shaham et al. Findings of ACL 2024\n[3] Zero-shot cross-lingual transfer in Instruction Tuning of large language models. Chirkova et al. INLG 2024\n[4] Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed? Kew et al. EMNLP 2024", "questions_to_authors": "- Did you assess how good gpt-4 is at assessing answers in japanese? \n- You mention to skip some of the results due to page limit. I believe good practice is to put those results in the appendix for the interested reader. It allows to get more complete picture of the work.\n- discussion about discrepancy in student ant teacher models behaviour for English and Japanes (lines 232-234): did you manually examine the performance of both teacher models? Did you observe presence of code-switching? How does gpt4-based judge behave on code-switching data? In our previous experiments we observed that and answer  would be judged as correct by gpt4o, even if it is not generated in the same language as question. If one of the teachers tends to generate more code-switched answers than other it won't necessarily be reflected in overal score computed by gpt4 judge, but can have an important impact on the performance of instruction tuned model on such dataset.", "extra_scores": {}}, {"review_id": "Fsx4jMKSZ8", "reviewer": "Reviewer_UvMA", "summary": "This paper investigates whether human-written instructions are still necessary for instruction tuning of large language models (LLMs), despite recent success of fully synthetic datasets like Magpie. The authors build new instruction-tuning datasets by pairing real user-written instructions from LMSYS-Chat-1M with responses generated by state-of-the-art open-weight LLMs (Llama-3.1 and Gemma-2). Fine-tuning on these datasets consistently outperforms existing instruction-tuning datasets—including Magpie—on MT-Bench evaluations across multiple base models. The authors further demonstrate that their approach generalizes to other languages, such as Japanese, by translating instructions and generating corresponding responses. They show that instruction-tuning improves instruction-following ability, while culturally specific knowledge still requires language-specific pretraining. The paper concludes that combining human-written instructions with LLM-generated responses yields superior datasets and presents a scalable and open-source method for instruction-tuning across languages.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper demonstrates that datasets built by pairing human-written instructions with responses from open-weight LLMs consistently outperform existing state-of-the-art datasets (e.g., Magpie) across multiple base models (Llama, Gemma, Qwen, OLMo) and in multiple languages (English and Japanese). This highlights the robustness and generalizability of their approach.\n\n2. The authors present a simple yet effective and reproducible pipeline for instruction tuning: using real-world human-chatbot conversations (LMSYS-Chat-1M) for instructions and generating responses using publicly available LLMs. Their datasets are openly licensed, allowing for broader adoption in both academic and commercial settings.\n\n3. The paper goes beyond just performance metrics and provides thoughtful analysis of cross-lingual instruction tuning. It distinguishes between capabilities gained through instruction tuning and those requiring continuous pretraining, offering valuable insights into the limitations of current models in handling culture-specific knowledge.", "reasons_to_reject": "1. The primary evaluation relies heavily on MT-Bench, which, while strong, may not comprehensively capture all facets of instruction-following (e.g., safety, robustness, or factual accuracy). The paper could be strengthened by including human evaluations or a broader range of benchmarks to support its claims more holistically.\n\n2. The proposed datasets are primarily single-turn, while many real-world applications of LLMs involve multi-turn interactions. Although the paper acknowledges this and cites it as future work, the lack of multi-turn data may limit the practical applicability of the datasets in dialog-heavy tasks.\n\n3. The cross-lingual extension to Japanese relies on machine-translating English instructions. This approach may introduce translation artifacts or overlook linguistic and cultural nuances, potentially affecting instruction quality. A more thorough analysis of translation quality or a comparison with native Japanese instructions would strengthen the multilingual claims.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates whether instruction-tuning datasets built from human-written instructions paired with LM-generated responses can outperform fully synthetic datasets like Magpie. Using real instructions from LMSYS-Chat-1M and responses from open-weight LMs (e.g., Llama-3.1, Gemma-2), the authors construct new datasets and demonstrate superior or comparable performance across multiple base models and in both English and Japanese. 

Pros:
- The pipeline is simple, reproducible, and publicly available. 
- The authors explore cross-lingual instruction tuning by translating instructions to Japanese, demonstrating generalization, and analyze the limits of tuning versus pretraining for handling cultural knowledge.
- The approach outperforms or matches state-of-the-art datasets like Magpie (a state of the art instruction tuning dataset) across multiple models.
- The approach uses real human-written instructions, improving instruction diversity and realism.
- The paper provides insightful analysis on instruction tuning vs. pretraining needs.

Cons:
- The work relies heavily on MT-Bench; lacks broader or human evaluations.
- The experiments focus on single-turn data, limiting applicability to multi-turn tasks.
- Japanese extension uses machine-translated instructions, which may introduce artifacts.
- The paper lacks direct comparison using synthetic instructions generated from the same seed data.
- Reviewers point out that some analyses (e.g., Japanese performance, LM judge reliability) could be deeper.

One reviewer (WhHW) identified a discrepancy between the paper's stated research claims and the experimental setup, requesting a more controlled experiment.  The meta-reviewer encourages the authors to more carefully scope the claims of the paper and, in the paper, identify the missing experiment as a possibility for future work.  Several reviewers asked for content to be moved into the paper (or appendix), and the authors are advised to take these recommendations seriously.
Decision: Accept