Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Authors: Haoran Li
Qingxiu Dong
Zhengyang Tang
Chaojun Wang
Xingxing Zhang
Haoyang Huang
Shaohan Huang
Xiaolong Huang
Zeqiang Huang
Dongdong Zhang
Yuxian Gu
Xin Cheng
Xun Wang
Si-Qing Chen
Li Dong
Wei Lu
Zhifang Sui
Benyou Wang
Wai Lam
Furu Wei
Venue: Accepted by TMLR ()

Abstract: We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction-tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy. While promising, our approach may inherit biases or inaccuracies from LLM-generated data as in other synthetic data work and is primarily evaluated on exam-style benchmarks. Broader evaluations and data quality control are left for future work.

Published in Transactions on Machine Learning Research (07 (07/2025)
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Haoran Li 1, Qingxiu Dong* Zhengyang Tang* Chaojun Wang"; Xingxing Zhang" Haoyang Huang",, Shaohan Huang', Xiaolong Huang', Zeqiang Huang', Dongdong Zhang', Yuxian Gu Xin Cheng", Xun Wang', Si-Qing Chen', Li Dong', Wei Lu², Zhifang Sui Benyou Wang", Wai Lam', Furu Weite
Microsoft
Singapore University of Technology and Design
a Peking University A Tsinghua University
The Chinese University of Hong Kong, Shenzhen
The Chinese University of Hong Kong
Reviewed OpenReview: http:p///paaaareiieeoo//////ttta
Abstract
We introduce Generalized Instruction Tuning (called GLAN), general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction-tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across disciplines. Specifically, spired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, facilitated by LLMs. Subsequently,
distinct disciplines semi-automaticalle comprchensive list of subjects for every discipline and proceed to design syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in ev- ery class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (c.g., Mistral) demonstrate that GLAN excels in multiple dimen- sions from mathernttcca reasoning, coding, academic exams, logical reasoning to general instruction following without task-specific training data. In addition, GLAN allows for easy customization and new fields skills can be added by simply incorporating node into our taxonomy. While promising, our approach may inherit biases or inaccuracies from L.M-gennrated data as other synthetic data work and is primarily evaluated on exam-style benchmarks. Broader evaluations and data quality control are left for future work.
generate
1 Introduction
Large Language Models (LLMs) have enabled unprecedented capabilities to understand and generate text like humans. By scaling up model size and data size (Kaplan et al., 2020; Hoffmann et al., 2022), LLMs are better at predicting next tokens and prompting to perform certain tasks with few demonstrations ions (Brown et al., 2020). However, these capabilities do not directly translate to better human instruction -ooeetiiiooooooiiinnggg
"Equal contribution. Corresponding author.

---

Published in Transactions on Machine Learning Research (07 (025)
GLAN
Taxonomy of Human Knowledge Capability
FLAN
Self-Instruct
Evolve-Instruct
Many NLP Datasets
Seed Examples
Existing Dataset
LLM
Break Down
Described Instructions
LLM
LLM
Few-shot Expansion
Data Augmentation
Sample
LLM
Instruction
Instruction
Instruction Dataset
Generate
Dataset
Dataset
Instruction
Dataset
Figure 1: Comparing GLAN with FLAN (Longpre el al., 2023), Self-Instruct (Wang el al., 2023) and Evolve- Instruct (Xu et al., 2023a). The inputs of FLAN, Self- Instret and Instre- are either seed examples or existing datasots, which limits the scope of domains of instructions that these methods can generate. GLAN takes the taxonomy of human knowledge & capabilitics as input to ensure the broad coverage of generated instructions various domains. This taxonomy then broken down into smaller pieces and recombined to generate diverse instruction data.
ability (Ouyang et al., 2022). Instruction luning (Wei el al., 2022) bridges this gap by fine-tuning LLMs on instructions paired with human- responses.
Previous work has constructed instruction tuning data using seed examples or existing datasets (Xu et al., 2023a; Wang et al., 2023). For example, FLAN (Wei et al., 2022) aggregates traditional NLP datasets into an instruction-following set. However, the availability of only a few thousand NLP tasks (Wang et al., 2022; Longpre et al., 2023) restricts the generalization capabilities Recently, the Self- Self-instruct approach (Wang et al., 2023) has generated synthetic instruction tuning datasets from limited pool of human-written seed instructions. Evolve-Instruct (Xu al., 2023a) further enhances this by augmenting existing instruction tuning datascts through rewriting operations using LLMs. Despite these advancements, these strategies primarily rely on data augmentation, meaning the range of domains or tasks covered by the augmented datasets remains constrained by the original input datasets.
LLMs trained on FLAN (Xu et al., 2023a).
How to create a general instruction tuning dataset? We draw inspiration from the systematic structure in human education system. The structure human education includes several levels, starting from early childhood education up higher education and beyond (Wikipedia contributors, 2023). Within each level, student acquires knowledge, skills, and values in systematic process. The courses student learns from primary school to college cover broad range of knowledge and skills, which facilitates the development of diverse array of abilities. We believe that the systemic framework of the human education system has the potential to help the generation of high-quality and general instruction data, which spans a diverse range of disciplinary areas.
In this paper, we introduce a generalized instruction tuning paradigm GLAN (shorthand for Generalized Instruction-Tuning for Large LANguage Models) to generate synthetic instruction tuning data almost from scratch. As shown in Figure 1, unlike existing work (Xu et al., 2023a; Luo et al., 2023b;a; Mukherjee et al., 2023), GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale instruction data systematically and automatically across all disciplines. Specifically, inspired by the structure of the human education system, the input laxonomy is constructed by decomposing human knowledge and capabilities to various fields, sub-fields, and, ultimately, distinct disciplines semi- automatically, facilitated by LLMs and human verification. The cost of human verification process is low due the limited number of disciplines in the taxonomy. As shown in Figure 1, we then further break down these disciplines into even smaller units. We continue to generate comprehensive list of subjects for
2

---

Published in Transactions on Machine Learning Research (07/2025)
every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we can first sample from them and then generate diverse instructions with broad coverage across the entire spectrum of human knowledge and skills. The process described above mirrors the human educational system, system, where educators in each discipline craft a series of subjects for student learning. Instructors then develop a syllabus for each subject, breaking down the content into specific class sessions. These sessions are then further divided into core concepts that students must comprehend and internalize. Based on these detailed core concepts outlined in the syllabus, teaching materials and exercises are subsequently created, which are instruction tuning data.
GLAN is general, scalable and customizable. GLAN is general method, which is task-agnostic and is capable of covering a wide range of domains. GLAN is scalable. Similar to Wang et al. (2023); Xu et al. (2023a), GLAN generates instructions using LLMs, which can produce instructions on a massive scale. Morcover, the input of GLAN taxonomy, which generated by prompting an LLM and human verification, requiring minimal human effort. GLAN allows for casy customization. New fields or skills can be added by simply incorporating a new node into our laxonomy. Note that each node of the taxonomy can be expanded independently, which means that only need apply our method to the newly added nodes without re-gencrating the entire datasct. Extensive experiments on large language models (c.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, and logical reasoning to general instruction following without using task-specific training data of these tasks. However, our approach depends on the quality of LLM-generated taxonomies and data, and may inherit their biases or inaccuracies. Furthermore, we primarily evaluate on exam-style benchmarks, and broader assessment standard instruction-following benchmarks is left for future work.
2 GLAN: Generalized Instruction-Tunan Language Models
GLAN aims to create synthetic instruction data covering various domains of human knowledge and capa- bilities on a large scale. As shown in Algorithm 1, we first build taxonomy of human knowledge and capabilitics using frontier LLMs (i.c., GPT-4) and human verification. The taxonomy naturally breaks down human knowledge and capabilities to fields, sub-fields, and ultimately different disciplines (see Section 2.1). The following steps are fully autonomously facilitated by GPT-4 (or GPT-3.5). Then for each discipline, we again instruct GPT-4 further decompose into list of subjects within this discipline (Section 2.2). Similar to an instructor, GPT-4 continues to design a syllabus for cach subject, which inherently breaks subject into various class sessions with key concepts that students need to master (Section 2.3). With the obtained class sessions and key concepts, we are ready to construct synthetic instructions. We prompt GPT-4 to generate homework questions based on randomly sampled class sessions and key concepts as well as the syllabus (Section 2.4). We recursively decompose human knowledge and capabilities into smaller units until atomic-level components (i.e., class sessions and key concepts). We expect to randomly combine these class sessions and key concepts ensure the coverage and diversity of synthetic instructions.
Algorithm GLAN Instruction Generation
D + buwlduutilooooeee (aild_taxxnomy(
build a taxonomy and return a list of disciplines (Section 2.1)
L
for each discipline d ID do
S + cenerate_subjecter for each subject E do
Obtain a list of subjects in d (Section 2.2)
A + sneerate_syllabusss d)
D Return syllabus A for (Section 2.3)
C, K f axtract_class_dettiee
> Extract class scssions and key concepts (Section 2.3)
Q - generate_instructione ins.iincissssss/// (A, C, K, d) D Generate instructions by sampling class sessions and
key concepts (Section 2.4)
L L+ LUQ
end for end for return L
3

---

Published in Transactions on Machine Learning Research (07 2025)
2.1 Taxonomy of Human Knowledge and Capabilities
We build laxonomy of human knowledge and capabilities to guide the generation of synthetic instructions. Therefore, its coverage is important. On the other hand, it is also essential to make the taxonomy highly extensible, since the preferred capabilities of LLMs may change over time. In the first step, we propose to generate the taxonomy by prompting GPT-4 with a set of different instructions (e.g,, list all fields of human knowledge and capabilities). Then, we do human post-editing to ensure its correctness and completeness. Due to the limited number of fields, sub-fields, and disciplines in our taxonomy, the cost of human verification reasonably low. Another advantage of hurnan post-editing that we can easily add new fields or disciplines to the taxonomy as needed.
Our taxonomy currently covers diverse range of knowledge and capabilities in both academic education and vocational training. The top level of the taxonomy contains fields such as Natural Sciences, Humanilies, or Services (vocational training). These fields branch out to various sub-fields and/or disciplines such as Chemistry, Sociology or Retading. We keep breaking down nodes of the taxonomy until disciplines, and we leave the breaking down of disciplines to automatic methods described in the following sections. By collecting the leaf nodes of the taxonomy, we obtain a list of disciplines D = {di, d2,, das).
2.2 Subject Generator
As Algorithm 1, for each discipline d, we aim extract the list of subjects in through prompt engineering. Specifically, we instruct GPT-4 to act as an education expert of discipline d and design a list of subjects a student should learn. The completion of GPT-4 contains a comprehensive list of subjects and their meta data (e.g., level, introduction and subtopics of the subject) in unstructured text format, which can not be directly used in subsequent steps. We therefore used another round of prompting to convert the completion to JSONL format:
Prompt
Transform the above JSONL format that it is casier for computer to understand. Enclose the JSONL output between two sets of triple backticks. For each JSONL object, use the keys "subject name", "level" and "subtopics"
It is worth noting that generating a subject list in JSONL format using a single prompt is feasible. However, we refrain to do so, because we observe that incorporating additional formatting instructions directly into the prompt can compromise the quality of the resulting subject. list. These extracted subjects (as well as their meta data) = {s1, ,,,..., can be subsequently used in next steps. For each s E S, let n.mam, level and s. subtopics denote the name, grade level and subtopics of subject s, respectively. We can apply the above prompts multiple times to ensure better coverage of subjects within this discipline.
2.3 Syllabus Generator
For each subject s, have already extracted its name (s. (s. name), grade level (s. level), and a small set of included sub-topics (s subtopics) in structured format. In this section, we aim to further segment each subject into smaller units, making them more suitable for creating homework assignments. We consult GPT-4 to design a syllabus for this subject. We opt for syllabus generation for the following reasons. Firstly, a syllabus essentially breaks down the main topic of a subject into smaller segments in a hierarchical manner. Specifically, each subject comprises several class sessions, and each session covers a variety of sub-topics and key concepts. Secondly, a syllabus provides an introduction, objectives, and expected outcomes of a subject, which are inherently useful for formulating homework questions. We instruct GPT-4 to 1) design syllabus based its meta data (s. level, name and subtopics); 2) break the subject into different class sessions; 3) provide details for each class session with a description and detailed key concepts students need to master.
Let A denote the generated syllabus. The resulting syllabus A is in unstructured text format. However, class session names and key concepts of each class are required in the instruction generation stop (sce Algorithm 1). Similar the process of subject list extraction Section 2.2 we again extract these meta data of each class session by prompting GPT-4. As a result, we obtain a list of class sessions C {cr, C2.-. . and
4

---

Published in Transactions on Machine Learning Research (07/2025)
their corresponding key concepts IK (k1, kg..................... ...,.... k,j,) The detailed prompt for syllabus generation is in Appendix A.2.
2.4 Instruction Generator
Given a syllabus A as well list of its class sessions C and their associated key concepts K, we are ready to generate homework questions and their answers. To generate diverse homework questions, we first sample one or two class session names from C and one to five key concepts under these selected class sessions. Let C denote the selected class session names and IK the selected key concepts. Then we prompt GPT-4 (or GPT-3.5) to generate a homework question given the selected class sessions C and key concepts K as well as the syllabus A. We intend to give GPT-4/3.5 more context (e.g., what students have already learned in previous sessions) when creating assignments. Therefore, we additionally instruct GPT to consider that students have learned up to class sessions C when crafting homework and try to leverage multiple key concepts across different class sessions. See details of our prompt for instruction generation in Appendix A.3.
Sampling Class Sessions and Key Concepts In single syllabus, there are numerous class sessions and koy concepts. We have two strategics to sample from them. In the first strategy, we generate assignments from a single class session. Therefore, we have only one class session name. Suppose we have m key concepts total in this session. We randomly sample one five key concepts from the m key concepts, which means we have totally -n unique combinations. In this strategy, we focus on creating basic homework questions. To make the resulting questions more challenging (combine knowledge from multiple class sessions), we propose a second strategy to combine key concepts from two class sessions in the second strategy. We intend to generate questions leverage knowledge from two different class sessions. Supposc we have my and m2 key concepts in the first and second class sessions, respectively. We can have Σ (m*+*a) Di (*i)) - -i ("²²) different combinations, which is significantly more than that of the first strategy. We use both strategies to ensure our created questions are diverse in difficulty levels.
Answer Generation After we generate questions in previous steps, we simply send these questions to G and collect answers. We use 5 for answer generation, because we find the quality of generated answers from GPT-3.5 5 is sufficiently good and using G is significantly faster than GPT-4. The resulting question-answer pairs are our instruction tuning data. With a huge amount of question-aa ansserr pairs ranging from different disciplines with various difficulty levels, we expect the resulting LLM can excel in wide range of tasks.
3 Experiments
3.1 Data Generation
Taxonomy Creation By asking GPT-4 to create taxonomy of human knowledge and capabilities, end up with set of fields, sub-fields, and disciplines that cover broad range of domains in human knowledge and capabilities. Next, we ask human annotators to decide whether these elements in the taxonomy should be kept or not in order to reduce the redundancy of the taxonomy while maintaining its correctness. Note that if a field or sub-field is marked as remove, we remove its descendant as well. We kept 126 disciplines after majority voting (provided in supplementary materials). Note that it is feasible to manually add extra disciplines, sub-fields, or fields whenever necessary.
Subject and Syllabus Generation During the subject list and syllabus generation, we prompt GPT-4 and employ nucleus sampling (Holtzman et al., 2020) with temperature T 1.0 and top-p 0.95 to encourage diversity. We do not use OPT-3.5-turbe since some subjects belong to the long-tail distribution which may not be effectively modeled by SPP-3.5-turbo. To ensure diversity and completeness of the generated subjects, we query GPT-4 10 times for each discipline (Section 2.2). There are 100 to 200 subjects for each discipline on average. It worth noting that the same subjects may appear in different disciplines. For instance, the subject calculus both in physics and mathematics. We do not de-duplicate those subjects, since it may reflect their importance in human knowledge. Given a subject in specified discipline, we query
5

---

Published in Transactions on Machine Learning Research (07/2025)
GPT-4 for only one time to design syllabus (see details in section 2.3). The temperature and top-p are still set to 1.0 and 0.95, respectively. The number of class sessions contained in each syllabus varies from 10 to 30 and each class session contains around five key concepts.
Instruction Data Generation Each instruction data consists of a question and its answer. We choose to generate questions and answers separately since we observed that separate generations lead to higher quality outputs. After question generation with GPT-4, each question is then answered by GPT-3.5-turbe with lemperature T = 0.7, top-p 0.95 (we use lower temperature in order to make the resulting answers more accurate). We use GPT-3. 5-turbo instead of GPT-4 for answer generation, because significantly faster with reasonably good results. According to the calculation method outlined in Section 2.4, we have over 500 million unique combinations of class sessions and key concepts, which guarantees the diversity of the generated data. In this paper, we generate 10 million instruction--eeponse pairs in total and then we do training data decontamination. Specifically, the training instruction--esponne pairs are decontaminated by removing pairs that contain questions or input prompts from the test and training (if any) sets of benchmarks we evaluate. We exclude the training set of benchmarks we evaluate to verify the generalization capability of our synthetic data.
is
Inference Cost The inference cost of GLAN is closely tied to the models used for data generation. Note that GLAN is not limited to GPT-4 or GPT-3.5, 5; it can be applied to any open-source or closed- source models. To best showcase GLAN's performance, we selected the top available models at the time GPT-4 and GPT-3.5. We queried GPT-4 approximately 26,000 times for taxonomy, subject, and syllabus generation combined. For instruction generation, we queried GPT-4 10 million times, and for answer generation, we queried GPT-3.5 also 10 million times. For more details, please refer to Appendix A.4.
of
3.2 Model Training
We employ Mistral 7B (Jiang al., 2023) as our base model. During training, we concatenate each instruc- tion and response pair to single sequence and only compute loss on response tokens. We train our model for 3 epochs with a learning rate of 3c-6. The batch size is set to approximately 512 instruction-respons pairs. We employ a dynamic batch size to ensure a constant total number of lokens per batch. We use a cosine learning rate schedule and we start with a linear warm-up of 1000 steps and the final learning rate is reduced to O. The training requires approximately days using 32 A100 GPUs.
3.3 Benchmark Evaluation
The instruction data GLAN generated spans a wide range of subjects. We evaluate its effectiveness in mathematical reasoning, coding, logical reasoning, and academic exams.
Mathematics is a common subject in many different disciplines. Hence, it is necessary to test the math reasoning ability of GLAN. We choose the two popular benchmarks for evaluation (i.e., GSMSK (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b)). GSM8K (Cobbe et al., 2021) is a high-quality math problem dataset that measures the basic multi-step mathematical reasoning ability. It contains around 7k problems for training and 1K problems for test. MATH (Hendrycks et al., 2021b) is a challenging math dataset that contains mathematics competition-level problems from AMC, AIME, etc. The 7.5k training and 5K test problems cover seven math subjects, i.c., Prealgebra, Precalculus, Algebra, Intermediate Algebra, Number Theory, Counting and Probability, and Geometry. Note that GLAN does not use any examples in the training set of GSMSK MATH. Following Luo et al. (2023a), we report O~shot setting results for GLAN. Coding: To evaluate the coding capability of GLAN, we opt for two coding benchmarks HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). We employ 0-shot setting for HumanEval and 3-shot, setting for MBPP following prior art (Chen et al., 2021; Luo el al., 2023b). BBH: The instruction dataset we generated covers many disciplines, which can potentially enhance the reasoning ability of GLAN. Therefore, we evaluate GLAN on the BIG-Bench Hard datasct (BBH (Suzgun et al., 2023)), which contains 23 challenging tasks from Big- Bench (Srivastava et al., 2023). We employ the standard 3-shot setting with chain-of-thought demonstrations. Academic Exams: We also evaluate GLAN
Mathematical Reasoning:
6

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 1: Main results on Mathematical Reasoning, Coding, Logical Reasoning, and Academic Exam bench- marks. Best results are in boldface, while the second best results are underscored.
Model
HumanE MBPP GSM8K MATH BBH ARC-E ARC-C MMLU
GPT-4
88.4
80.0
92.0
52.9
86.7
95.4
93.6
86.4
GPT-3.5-turbo
72.6
70.8
74.1
37.8
70.1
88.9
83.7
70.0
LLaMA2
7B
12.8
36.2
15.4
4.2
39.6
74.6
46.3
45.9
Orca
7B
17.1
28.4
55.7
10.1
42.8
87.8
78.4
53.9
WizardLM v1.2
13B
31.7
47.9
46.8
9.0
48.4
74.2
50.2
52.7
Mistral
7B
28.0
50.2
43.4
10.0
56.1
79.5
53.9
62.3
Mistral Instruct
7B
46.7
31.7
24.4
8.2
46.0
76.9
52.0
53.7
MetaMath Mistral
7B
35.4
48.6
77.7
28.2
55.7
77.3
51.0
61.0
WizardMath v1.1
7B
51.2
54.1
83.2
33.0
58.2
79.8
53.2
60.3
Mistral CodeAlpaca
7B
35.4
50.2
34.6
8.3
56.1
79.1
54.2
60.9
GLAN
7B
48.8
57.6
80.8
32.7
60.7
90.7
81.1
62.9
Table 2: Detailed Results on Academic Exam benchmarks.
MMLU
Model
ARC-E
ARC-C
STEM
Humanities
Social Sciences
Other
Mistral
79.5
53.9
52.0
56.5
73.3
70.1
GLAN
90.7
81.1
60.1
54.9
71.8
68.6
on different academic benchmarks to verify whether GLAN is capable of solving exam questions. We choose two benchmarks (i.e., ARC (Clark et al., 2018) and MMLU (Hendrycks et al., 2021a)). Both benchmarks are composed of multi-choice questions. AI2 Reasoning Challenge (ARC (Clark al., 2018)) contains grade- school level, aullii-hoice science questions. It contains two sub-sets, which are ARC-Challenge (ARC-C) and ARC-Easy (ARC-E). Massive Multitask Language Understanding (MMLU (Hendrycks et al., 2021a)) consists of set of multiple-choice questions about 57 subjects ranging in difficulty from elementary levels to professional levels. It covers various of domains of knowledge, including humanities, STEM and social sciences. Note that there is a training set for ARC. However, we have excluded it from our training set during the decontamination process described in Section 3.1. Previous models mostly leverage probability- based methods on ARC and MMLU, which returns the best. option based on the probabilities of the four options conditioned on the corresponding multi-choice question. We observe that after training on 10 million instructions, GLAN is able to generate its predicted options and analysis of multi-choice questions in plain text GPT-3.5 does. We therefore opt for 0-shot setting for GLAN and extract predictions using rules based on its completions as in Mitra et al. (2023).
Results Our main results are shown in Table We compare GLAN against general domain models (Orca 2 (Mitra al., 2023), Mistral Instruct (Jiang et al., 2023) and WizardLM (Xu et al., 2023a)), math optimized models (MetaMath (Yu et al,, 2024) and WizardMath (Luo et al., 2023a)) and coding optimized models (CodeAlpaca (Chaudhary, 2023)). We also report results of base LLMs (i.e., LLaMA2 (Touvron et al., 2023) and Mistral (Jiang al., 2023)) references. GLAN either obtains the best results results close to the best across all benchmarks. We observe that capabilities of math or coding optimized models increase on math or coding benchnarks while usually not others. After instruction luning, GLAN excels on multiple dimensions from mathenatical reasoning, coding, reasoning, and academic exams with a systematical data gencration approach. Also note that our method does not use any task-specific training data such training of GSM8K, MATH, or ARC as in Orea 2, MetaMath, and WizardMath, which indicates the general applicability of GLAN.

---

Published in Transactions on Machine Learning Research (07/2025)
HumanEval
BBH
GSMBK
MATH
80-
48-
47-
Exact Manth
60.0
75-
40-
45-
59.5
70-
26
24
64-
65
22-
$8.5
20
5e4
205
565
166
1c7
5e4
265
565
166
167
5c4
2e5
565
1e6
1c7
5e4
205
5c5
1:66
167
Figure 2: The scaling curve of GLAN on downstream tasks. The æ-axis denotes GLAN data size (in log10 scale following (Kaplan et al., 2020)), and the y-axis denotes the task performance.
A Closer Look at Academic Exams ARC and MMLU are all multi-choice based benchmarks on academic exams. However, we observe that improvements of GLAN over Mistral on ARC are much larger than these on MMLU (sco Table 1). By grouping the 57 subjects in MMLU into four categories (i.c., STEM, Humanities, Social Sciences, and Other (business, health, misc.)), we observe GLAN wildly improves on STEM in MMLU while not in other categories (TTable 2). This is consistent with recent findings that Chaun-of-Thought (CoT) primarily aids in symbolic reasoning problems rather than other types of questions (Sprague et al., 2024). Also note that ARC is composed of high school science problems, which are also STEM questions. GLAN is good at STEM subjects may be because responses of our dataset are from GPT-3.5 turbo, which by default generates responses with CoT reasoning. Indeed, we observe that GLAN generales solutions with CoT for multi-choice questions.
3.4 Scaling Property of GLAN
We investigate the scaling property of GLAN by training Mistral on different numbers of examples (i.c., 50K, 200K, 500K, 1M, and 10M) we generated. The results on downstream tasks are shown in Figure 2. It can be observed that overall task performance tends to increase as increase the data size. It's important to note the performance drop observed in the 200K to 1M data range for both HumanEval and BBH benchmarks. This regression might be attributed to the relatively small average number of data points per discipline at these scales. Our dalaset encompasses 126 disciplines, with an average of approximately 2,000 examples per discipline at the 200K total, increasing to about 8,000 examples per discipline at the 1M total. Interestingly, we observe a significant performance boost when scaling from 1M 10M examples on both HvnanEval and BBH. This improvement. suggests that the increase in data points per domain crosses a threshold where it becomes substantial enough to positively impact model performance. Note that none of the curves have reached plateau, indicating the potential for further improvement through continued scaling of GLAN, We leave further exploration on the scaling property of GLAN to future work.
3.5 Task-specific Training Data
GLAN is a generalized method to create synthetic data for instruction tuning. In order to evaluate the generalization capabilities of this synthetic data, we deliberately exclude task-specific training sets from all benchmarks on which we conduct our assessments. Similar to Wei et al. (2023), we explore whether models have been trained on task-specific in-domain data. We compute the training loss Turrin and test loss Trest on ARC Challenge (ARC-C), GSM8K, and MATH for GLAN and other models in comparison. We choose these datasets because among all benchmarks evaluated in Section 3.3, these benchmarks contain training sets. Intuitively, the larger A Ltest Lirain is, the more likely the training set is exposed. To make A Table casier to interpret, we additionally compute the relative difference A = (Lrest- - Ltrain)/Lee.. 3 shows the losses of the training and test splits for GLAN are nearly identical (or A is negative). This suggests that GLAN has not been exposed to in-domain data during training and tuning procedures. Please refer to the detailed losses of Lerain and Lreet Table (in Appendix). Additionally, as shown in Table 8, we observe that GLAN obtains higher losses on both test and training splits on GSMSK, MATH, and ARC compared to other models, while performances of GLAN these datasets are high (see Table 1). This
8

---

Published in Transactions on Machine Learning Research (07/2025)
Table 3: The evaluation of loss values between the test data and training data. Large positive A (or A indicates task-specific in-domain training data might be exposed to the model during training.
Wenchmark/Lose Losss
Orea2-7B L.AMA2-7B
Mistral-7B- Instrruct
Whardd.M-13B--1.22
GLAN-7B
A
-0.01
0.05
-0.01
-0.01
-0.03
ARC-C
(%)
-0.5%
2.10%
-0.43%
-0.47%
-0.74%
A
-0.02
0.04
-0.03
-0.02
-0.01
ARC-E
A (%)
-0.95%
1.61%
-1.19%
-0.91%
-0.23%
0.13
0.05
GSM8K
(%)
0%
11.4%
0%
4.30%
0.92%
A
-0.03
0.03
-0.03
-0.02
-0.03
MATH
A (%6
-2.70%
2.54%
-2.67%
-1.63%
-1.79%
Table Instruction following capability evaluation on IFEval.
Model
Prompt-level strict-accuracy
Instruction-level strict-accuracy
Prompt-level strict-accuracy
Instruction-level
loose-accuracy
GPT-3.5-turbo
53.8
64.7
56.6
67.5
GPT-4
77.1
83.7
79.7
85.6
LLaMA2-7B
14.8
27.1
16.6
Orca2-7B
19.4
28.9
26.1
34.7
32.0
42.8
37.7
WizardLM-33--V
23.1
33.5
26.6
37.6
GLAN-7B
34.0
44.8
41.2
51.6
might imply that synthetic data generated by GLAN is diverse and our resulting model avoids convergence any specific domain or style present existing benchmarks.
3.6 Instruction Following Evaluation
IFEval We assess the instruction- fostruction-oolooinng capabilities GLAN utilizing the Instruction Following Eval- uation dataset (IFEval (Zhou et al., 2023b)). 1FEval consists of collection of "verifiable instructions" encompassing 25 distinct types of instructions (around 500 prompts in total). Each prompt comprises one more verifiable instructions. The evaluation involves four types of metrics at both prompt level and instruction level, evaluating strict and loose accuracies. As shown in Table 4, GLAN demonstrates superior instruction-foldee following capabilities in both prompt-level and instruction-leva evaluations. However, there is still a considerable gap compared to GPT-3.5 5-turbo and GPT-4.
Evol-Instruct Test Evol-Instruct testset (Xu et al., 2023a) contains real-world human instructions from diverse sources, and it consists of 218 instances with 29 distinct skills. Each instruction is associated with difficulty level from 1 to 10. The responses mark is a necessary supplement to IFEval (answers to their Instructions are "verifiable"). Following Xu et al. (2023a) and Chiang et al. (2023), we adopt a GPTT-4-baee automatic evaluation method to conduct a pairwise comparison between GLAN and other models. Specifically, GPT-4 is instructed to assign a score between and 10 overall score w.r.t. the helpfulness, relevance, accuracy, and level of detail of responses gen- erated by two different models for given input question. A higher score indicates better overall performance. To mitigate potential order bias, we perform bidirectional comparisons for each response pair and determine their average score. The average score difference to GLAN (i.e., avg_score (GLAN) wrrg_score(s) serves the final metric. Table 5 presents the results of pairwise comparisons across various levels of instruction difficulty. GLAN showcases superior performance compared to LLaMA-2, Orca 2, Mistral Instruct, and even WizarLLM-13B (note that GLAN contains only 7B parameters) on most difficulty levels and overall scores. This suggests that GLAN demonstrates improved ability to process diverse instructions, regardless of their difficulty or complexity. Also, note that GLAN falls behind UPT-3.5-tures as other modcls in com- parison. Additionally, we group Evol-Instruct test according the 29 skills and observe the same trends. Detailed results are listed in Appendix (Table 9 and 10). GLAN demonstrates strong performance most
often open-ended descriptions, and we believe this bench-

---

Published in Transactions on Machine Learning Research (07/2025)
skills, especially in Math, Coding, and Reasoning. However, it slightly falls short in common-sense related tasks. We also created G-AAN similar to the Evol-Instruct Test but much larger in size, where GLAN outperforms other models as well (see Appendix A.8).
Table 5: Pairwise comparison on various difficulty levels between GLAN and other models on Evol-Instruct testset. The scores are the average gap of scores assigned by GPT-4, calculated as score (GLAN) avg_ avg_ (core()..
Difficulty Ratio
LLaMA2-7B
Orca2-7B
Bisstal-7B-Inssruee Wizard-13B-V12 GPT--35-tuubb
(1-5) Easy 41.00%
5.46
2.19
1.13
1.32
-1.22
(6-10) Hard
59.00%
5.38
2.28
1.68
0.99
-0.68
4 Related Work
Recent literature has extensively explored the collection of various human-made resources for instruction tuning. An intuitive direction is to collect existing NLP datasets and corresponding task descriptions (Sanh et al., 2022; Wang et al., 2022; Zhou et al., 2023a), typical LLMs such as BLOOMZ (Muennighoff et al., 2023) and FLAN (Wei et al., 2022) are trained on this type of instruction tuning data. However, with only tens to thousands of existing datasets available, the scope and diversity of instruction tuning are inevitably limited. Another common practice to implement instruction tuning with real-world human user prompts. For instance, Instruct (Ouyang el. al., 2022) was trained on high-quality human prompts submitted by real-world users to OpenAl GPT APIs. Vicuna (Chiang et al., 2023) leverages user-shared prompts along with ChatGPT responses for instruction tuning, and Dolly(Conover et al., 2023) was trained on simulated human-user interactions written by over 5k employees. Nevertheless, acquiring instructional data from human users typically involves high costs and involves privacy concerns.
As LLM capabilitics improve, instruction tuning with LLM-generated data exhibits better scalability and potential in addressing the auperaalgnmmen problem (Shen et al., 2023). Leveraging the in-context learning ability of LLMs,, Unnatural instructions (Honovich et al., 2023) and Self-instruct (Wang et al., 2023) sampled seed instructions as examples to elicit LLMs to generate new instructions. Taking advantage of the rephrasing ability of LLMs, WizardLM (Xu et al., 2023a) and WizardMath (Luo et al., 2023a) were trained using Evol-unstruct. Evol-Instruct iteratively employs ChatGPT to rewrite seed instructions into increasingly complex instructions. Similar to generation from sced instructions, carefully selected seed topics are used for generating textbook-like synthetic data (Li el al., 2023b) or self-chat multi-turn dialogues (Xu et al., 2023b; Ding et al., 2023) for instruction tuning. However, models trained on these -Leerreee data only work well in specific domains such as math (Luo et al., 2023a; Yu et al., 2024), dialogue (Xu et al., 2023b; Ding et al., 2023) open-ended question answering (Taori et al., 2023; Xu et al., 2023a). These methods encounter challenges in generalization (Gudibande et al., 2024), as the data diversity is restricted by seed instructions or seed topics.
5 Limitations
While our work demonstrates the potential of fully LLM ILM-generated instruction tuning datasets, several limitations must be acknowledged. Firstly, the accuracy and reliability the taxonomy and synthetic answers generated by these models can be compromised by potential biases and inaccuracies inherent in LLMs, impacting the dataset's quality. Secondly, our evaluation predominantly utilizes exam-style benchmarks For general instruction following, we evaluated our method on IFEval (Zhou et al., 2023b) and Evol-Instruce (Xu et al., 2023a). Broader evaluations on standard benchmarks such as AlpacaEval (Li et al., 2023a) and MT- Bench (Zheng et al., 2023) would provide a more comprehensive understanding of our dataset's utility. Additionally, the high cost associated with data generation using proprietary APIs raises questions about scalability and practicality. Future work should consider using open-source models to improve affordability and reproducibility. Finally, our current synthetic data generation does not incorporate mechanisms for filtering inaccuracies hallucinations, which could propagate errors within the dataset. Detecting errors
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
and hallucinations in synthetic data remains open challenge (Tang et al., 2024), and we plan to explore this in future work. Addressing these limitations would enhance real-world model deployment.
6 Conclusions
We propose GLAN, general and scalable method for synthesizing instruction data. Experiments show that GLAN can help large language models improve their capabilities in multiple dimensions, from mathematical reasoning, coding, academic exams, and logical reasoning to general instruction following. Currently, our synthetic data are based on the taxonomy of human knowledge and capabilities, and there are other types of useful data that have not been covered. We are interested in designing methods with border coverage. Our current instruction data are mostly question-answer pairs, and in the next step, we plan to generate synthetic data of multi-turn conversations and long documents.
Acknowledgments
We would like to thank the anonymous reviewers and meta-reviewer for their constructive foedback on this work. This research/projes is partially supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative, (AISG Award No: LISG-NMLP22024-000 Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No. MOE-T2EP20122-0011) The work described in this paper is substantially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200620).
and
References
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint ar Xiv:22108..0 07732, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 2020.
Sahil Chaudhary. Code alpaca: An instruction-followinn llama model for code generation. https://github hoa//sah11200114/ooeeeel
2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. XrXiv preprinl ar Xiv:2100.00377, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL attps://1msy:: htt.//ssssor//bbl. org/blog/ 2023-03-30-vicuma//
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 2018.
Karl Cobbe, Vineot Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXic:2110.14 14168, 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tumed 1lm, 2023. URL https://www https://www.databrick. databricks com//blog/023/04//222 dolly-first- open- www.errrally---aalle--ttttttoooo
11

---

Published in Transactions on Machine Learning Research (07/2025)
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint Krv:22305 14233, 2023.
Meta GenAT. Mlama3. https://laana.mmtttccoo///aaaaa wtt.mmtt...mmamma../...
2024.
Arnay Gudibande, Eric Wallace, Charlie Victor Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary language models. In International Conference on Learning Representations, 2024. URL attps://openrett http:://pppppaeeeeaaaaoooooorraa k2y
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems, 2021b.
Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- alion. In International Conference on Learning Regresenlations, 2020. URL https: //tp///eppeeeeeeeee/e/ee / forum?id forumiid-rygQDyrvvt
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics ( (Volume 1: Long Papers), 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar XrX::223100066 06622, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ai preprint arXiv:2001.00 00866161,,..0,,,..... 2020.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cntrruction-ooloowiii models. https: //github. com/tatsu-lab/alle laaab//lpaca_eeee eval, 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Text- books are all you need ii: 1hi-1.5 technical report. arXiv preprint Xrx::22309.054 2023b.
S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for elfective instruction luning. In International Conference on Machine Learning, 2023. URL https: //api. s/api.semanticscholla org/CorpusID: 256415991.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ar preprint arXi::2308.090 09588, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint ar Xiv:23306. 08568, 2023b.
12

---

Published in Transactions on Machine Learning Research (07 2205))
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Ravdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.110/5. 2023.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanic, Zaid Alyafcai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Anneual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.00 02707, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Informalion Processing Systems, 2022.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaf- fin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint 1.Ki:2339.15025, 2023.
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. 2024. URL /tttpp///appssmmmmicccccooo.... wtt.c/ccchoaaooooooorrprssllllla 272708032.
semanticscholar
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-lonsso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, et al. Beyond the imitation game: Quantilying and extrapolating the capabilities of language models. Transac- tions on Machine Learning Research, 2023. URL /ttps:// h/tppr/////peeeviieeee net f wwwfffrmmiiioooo idduyyLLBBoo...
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Lênguis- tics: ACL 2023, 2023.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. MathScale: Scaling instruction tuning for mathematical reasoning, In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47885-47900. PMLR, Satess/v35//aas t/ag5// html.
21 27 Jul 2024. URL https:/ httpe/////coccceegg..mm mlr, wwwpp/eee333
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloingg llama model. https://githut: com/ datsu-labbstaanfod__lleeee 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xix preprint arXiv:2307.09280 2023.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjum Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Kara- manolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kun- tal Kumar Pal, Maitreya Patel, Mchrad Moradshahi, Mihir Parmar, Mirali Purohit, Necraj Varshney, Phani Rohitha Kaza, Pulkit. Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shaïlaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super- Naturallnstructions:: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self- genereteetd instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https: httpes//ppppeeeeeeeeee net
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou. Skywork: A more open bilingual foundation model, 2023.
Wikipedia contributors. Education, 2023. URL http:///w....aaaaaaaa////a/aaaaa Last editod 24 March 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprinl arXiv:230,.12244 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter- efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023b.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference Learning Represenlations, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a- judge with mt- bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 46595 46623. Cur- ran Associates, Inc., 2023. URL https::/procceeeet newrips....pp.. httpsscc/papppp cc/paper htlllpppppleepe_pllllllllllaaeee 1t1le//paprr2223//1122 91
_nd Benchmarks .df.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followigg evaluation for large language models. arXiv preprint orXiv:2311.079. 2023b.
14

---

Published in Transactions on Machine Learning Research (07 (07/202)) 2025)
A
Appendix
A.1 Broader Impacts
Data synthesizing is crucial for the continual scaling of large language models, especially as we exhaust available human data. GLAN demonstrates the potential to generale vast amounts of synthetic data from scratch, paving the way for even larger-scale data syuthesis efforts. While GLAN has shown the effectiveness of synthetic data, we must point out that synthetic data may inherit and even amplify social biases present in the frontier LLMs for generation. Future research should focus on developing techniques identify and correct biases in the generated datasets and models trained them.
A.2 Prompt for Syllabus Generator
The prompt template for syllabus generation in Table
Table 6: Prompt template for Syllabus Generator.
You are expert in (s.name).
Using given data, design a syllabus for teaching students at the specified level. Note that example subtopics or descriptions are just give you impression of what this class like. Feel to add extra subtopics if needed (remember you are expert in (a (s.name]).
Data:
Level: (s.level)
Main Topic: [s.name)
Description Example Subtopics: {s subtopcpi
### # Syllabus Design Guide
1. Introduuttoon **: Start with an overview of the primary topic for the syllabus.
2. ** Class Details**; For each class session, provide:
*Description** Briefly describe the focus of the session.
** **Knoweedge Points**, Enumerate key concepls or lopics.
These will be used to craft homework questions.
** *eaaring Outcomes Activities**; Offer expected learning results and suggest related exercises activities.
A.3 Prompt for Instruction Generator
The prompt template for instruction generator is in Table 7.
A.4 Detailed Inference Cost
In this paper, we pair GLAN with the closed-source models GPT-4 and GPT-3.5. Since the architectures of these models are not publicly disclosed, we report API costs instead of actual computational costs (i.e., FLOPs). We estimate the API cost for generating 10 million data points to be approximately 360K USD when using GPT-4 and GPT-3.5 for answer generation.
At the time of submission, we recommend using GPT-4o and GPT-4o-mini (for answer generation). re- ducing the cost to about 66K USD. This is based on the consistent performance of GPT-4o over GPT-4 and CPT-4o-mini over GPT-3.5. Additionally, leveraging Mistral Large 2 and Mistral 8x7B (for answer generation) can further reduce costs to around 42K USD.
Notably, API costs have significantly decreased over the past year, from 30/60 USD per million input/outpus output lokens to 2.5/10 USD per million inpul/ /ouppt lokens. We anticipate that these costs will continue to decline.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 7: Prompt template for Instruction Generator.
## Background
You are an expert in (a name) education and you have designed syllabus (i.e., '## Syllabus")
We invite you (again) design ONE homework question for given class sessions and knowledge points.
The student have already learned all class sessions up the current sessions
(i.e., *# Current Session(s)").
There might multiple class session ## Current Session(s)'
The designed homework question should focus the topics in ## Current Session(s) and you should
try to cover the given knowledge points ### Given Knowledge Points'
We prefer homework questions leveraging multiple knowledge points and across different topics
## Syllabus
{A}
## Current Session(s) (C)
# # Given Knowledge Points
[k)
Morcover, open-source models, such as LLaMA-3 (GenAI, 2024), present powerful alternatives. The inference cost of GLAN when paired with these open-source models can be further reduced, making the application of GLAN more feasible.
A.5 Task-specific Training Data
We provide the specific train/test values of different models on different benchmarks in Table 8.
The evaluation of loss values between the test data and training data. Large positive A (or A(%)) indicate task specific in-domain training data may be exposed to the model during training.
Table 8:
Benchmark/Lon Loss
LLaMA2-7B
Orca2-7B
Mistral-7B-Instruee
WizardLM-13B-V11
GLAN-7B
Liest
2.02
2.39
2.32
2.11
4.03
ARC-C
Lirain
2.03
2.34
2.33
2.12
4.06
A
-0.01
0.05
-0.01
-0.01
-0.03
(%)
-0.5%
2.10%
-0.43%
-0.47%
-0.74%
Lisst Lirain
2.10
2.47
2.51
2.18
4.31
ARC-E
2.12
2.43
2.54
2.20
4.32
A A (%)
-0.02
0.04
-0.03
-0.02
-0.01
-0.95%
1.61%
-1.19%
-0.91% %
-0.23%
1.38
1.26
1.14
2.17
GSM8K
Lirain
1.38
1.26
1.09
2.15
0.13
0.05
A (%)
0%
11.4%
0%
4.30%
0.92%
Ltest
1.11
1.18
1.12
1.22
1.67
MATH
Tavuin
1.14
1.15
1.15
1.24
1.70
A
-0.03
0.03
-0.03
-0.02
-0.03
(%)
-2.70%
2.54%
-2.67%
-1.63%
-1.79%
A.6 Evol-Instruct Test Results on Different Difficulty Levels
The concrete Evol-Instruct test results different difficulty levels shown in Table 9.
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 9: Pairwise comparison on various difficulty levels between GLAN and other models on Evol-Instruct testset. The scores are the average gap of scores assigned by GPT-4, calculated as avo_ (GLAN) avg_ vcg_sccr
Difficulty
LLaMA2-7B Orca2-7B
Wisard-13--Vi CPT-3.5-turbo
Ratio
5.1%
5.41
2.23
-0.37
-0.21
-2.41
8.7%
3.87
1.74
1.06
1.41
-1.18
3
12.4%
3.72
2.35
1.04
-1.14
4
10.5%
3.61
1.34
1.52
1.54
-0.22
5
4.1%
4.67
3.31
2.39
2.5
-0.45
G
19.3%
4.43
2.42
0.74
1.54
-1.36
11.0% 7
4.97
1.26
1.62
1.36
-0.41
8 17.9%
6.02
3.58
3.17
1.7
0.15
9
6.0%
6.35
4.2
1.36
0.9
-0.92
10
5.1%
5.14
-0.05
1.53
-0.54
-0.83
(1-5) Easy (6-10) Hard
41.00%
5.46
2.19
1.13
1.32
-1.22
59.00%
5.38
2.28
1.68
0.99
-0.68
A.7
Evol-Instruct Test Results Different Skills
The concrete Evol-Instrue test results on different skills are shown in Table 10.
Table 10: Pairwise comparison on various skills between GLAN and other models on Evol-Instruct testset.
The scores are the average gap of scores assigned by GPT-4, calculated as avg_ (GLAN) avg_score(2
Skill
LLaMA2-7B
Orca2-7B
Mistral-71B-Instruct
Wizar--13B-11.2
GPT-3,5-turbo
Ratio
8.7%
6.58
2.16
2.41
-1.42
Code Generation
8.3%
6.16
3.87
4.22
2.59
-0.25
Writting
8.3%
3,2
0.79
-0.22
0.24
-1.1
Computer Science
6.9%
7.1
4.4
0.83
1.22
0.02
Reasoning
6.0%
6.3
2.52
3.38
3.02
0.62
Complex Format
5.3%
3.13
3.5
-0.17
2.41
-1.96
Code Debug
4.6%
5.85
2.3
1.4
0.2
-2.5
Common-Seusse
4.1%
6.5
3.19
-133
-0.92
-2.78
Counterfactual
3.7%
7.06
2.15
1.5
0.72
Multilingual
3.2%
7.35
0.79
1.71
-0.88
-2.75
Roleplay
2.8%
7.08
2.25
-0.59
Biology
6.66
2.75
1.46
-0.09
1.38
Tochnology
2.8%
-0.08
2.54
-3
-1.5
-2.75
Ethics
2.8%
6.59
3.38
2.41
5.42
-0.21
TruthfulQA
2.3%
3.1
3.7
-1.05
-1.3
-0.85
Sport
2.3%
4.3
0.55
-0.2
4.8
-0.3
Law
2.3%
7.7
4.65
5.85
1.7
0.2
Medicine
2.3%
3.9
-2.05
1.9
0.15
-1.25
Literature
2.3%
6.3
1.9
0.2
1.45
-0.15
Entertaiument
2.3%
4.5
2.7
-3
1.9
-3.2
2.3%
-0.85
-2.05
Music
4.1
-2.
Toxicity
7.25
3.12
3.75
Beonomy
2.3%
0.15
1.9
0
Physics
2.3%
6.8
2.5
4.35
3.65
-1
History
1.8%
4.12
-0.56
3.76
-0.31
0.12
Academic Writing
1.8%
6.76
6.37
2.44
1.37
0.62
Chemistry
0.9%
9.5
0.63
5.25
2.5
0.75
Philosophy
0.5%
11
-0.25
0.25
-0.25
0.5
Avg. Avg.(29 (29 skills)
100%
5.42
2.24
-0.95
17

---

Published in Transactions on Machine Learning Research (07 (07))
A.8 GLAN-Tes Overall Results
GLAN-Test There are only hundreds of instructions in In IFEval and Evol-Instrucl Test and we believe the domains or skills they can cover are rather limited. Therefore, we propose heldout test set using GLAN data and we call it GLAN-Test. It contains 6,300 instructions on 126 disciplincs (50 instructions for each discipline). We further categorize the 126 disciplines to distinct fields (i.e., Academic-Humanitiiss Academic-Social Science, Academic-Natural Science, Academic-Appliet Science, Academic-Formal Science, Industry-Services and Andustay-Agricuultur)) We believe that the extensive domain coverage of GLAN-Test renders it effective test bed for the assessment of generalization capabilities in LLMs. We adopt the same GPT-4 based evaluation protocol as in Evol-Instruct Test (previous paragraph). We prompt GPT-4 to do pairwise ranking of GLAN and other models in comparison. The overall results and results across the 8 fields are presented in Table 11, where GLAN obtains higher GPT-4 scores than Orca2-7B, Mistral-7B Instruct and WizardLM-13B. despite using only 7B parameters. GLAN still lag be- hind GPT-4. Detailed results for the 126 fine-grained disciplines can be found in Appendix A.9 (see Table for more details). GLAN demonstrates its effectiveness on multiple domains (or disciplines) such Math- ematics, Physics, Chemistry, Computer science, Electrical, Mechanical, etc., indicating that smaller models may yield general improvements various domains through strategic fine-tuning. Furthermore, it is noted that GLAN demonstrates less-than-ideal performance across distinct disciplines such as American history, Divinity, or Radiology. This observation underscores the polential for further refinement and development of our methodology within these domains.
cedustry-Manufacturrii
Table 11: Pairwise comparison between GLAN and other models on GLAN-Test (the 126 disciplines are calegorized into 8 fields for clarily of the illustration). The scores are the average gap of scores assigned by GPT-4, calculated as avg_ (GLAN) - avg_ _co_scc
Field (Ratio)
Orca2-7B
Mistral-77BIIntrruet
WizardMM-13B-V1. GPT-4
Academic-Humaniticc (15.9%)
0.79
0.25
0.02
-0.62
Academic-Sode -Sciill Science (7.9%)
1.22
0.21
0.09
-0.63
Academic-Naturee Science (4.0%)
1.73
1.23
0.53
-0.5
Academie-Applied Science (42.1%)
1.58
0.32
0.08
-0.58
Academic-Forman Science (3.2%)
3.87
2.48
2.32
-0.55
Mndustry-Mamufactuurii (12.7%)
2.26
0.56
0.33
-0.43
Industry-Servicer (11.9%)
1.82
0.23
0.09
-0.5
Industry-Agricultus (2.4%)
0.46
Overall (100.0%)
1.61
0.43
0.19
-0.55
A.9 GLAN- Test Results on Different Disciplines
18

---

Published in Transactions on Machine Learning Research (07 2225))
Table 12: Pairwise comparison across 126 disciplines (or domains) on GLAN-Test, The scores are generated from the average gap between GLAN and other model in assessment scores assigned by GPT-4, calculatod as (GLAN) avg_ score()..
Discipline
Orca-2-77
WiradLMM-13B-112 GPT-4
Avg.
1.61
0.43
0.19
-0.55
Advertising
1.92
0.46
0.21
-0.04
Aeraspace industry
3.24
1.24
0.6
-0.42
Agriculture
2.44
0.04
-0.05
-0.48
American history
-0.49
-027
-0.76
-0.83
American polities
1.23
-0.3
-0.4
-0.87
Anthropology
0.59
0.17
0.06
-0.7
Applied mathematics
3.75
2.6
2.74
-0.77
Archaeology
2.59
-0.11
-056
Architecture and design
2.63
0.34
-0.77
Astrouomy
1.01
0.03
-0.4
Automotive industry
1.27
0.71
0.46
-0.6
Biblical studies
-0.05
0.33
00.47
-0.5
Biology
1.09
0.22
-0.09
-0.17
Business
3.61
1.14
0.88
-026
Chemical Engincering
3.15
1.6
1.18
-0.77
Chemistry
3.06
2.09
0.8
-0.77
Civil Engineering
1.94
0.74
0.75
-025
Clinical laboratory sciences
1.32
0.94
-0.11
-0.77
Clinical neuropeychology
2.15
0.29
0.25
-0.4
Clinical physiology
2.07
0.41
0.51
-0.08
Communication studies
0.26
-0.15
-0.3
Computer science
4.29
1.45
1.9
-0.33
Cultural industry
3.15
0.44
0.05
-0.36
Dance
2.11
0.21
0.4
-0.47
Dentistry
1.67
0.66
0.48
0.01
Dermatology
2.12
0.55
-0.55
-0.65
Divinity
-0.34
-0.17
-0.48
-0.89
Earth science
0.39
0.44
-.088
-0.33
Economics
2.62
0.96
0.62
-0.4
Education
2.67
0.42
-0.84
Education industry
2.19
0.4
0.56
-1.33
Electric power industry
3.23
1.31
0.39
-0.79
Electrical Engineering
3.81
1.26
1.41
-0.4
Emergency medicine
2.04
0.44
-0.18
-0.86
Energy industry
3.59
0.98
0.54
-0.22
Environmental studies and forestry
0.12
0.41
0.1
-0.45
Epidemiology
3.02
0.52
0.33
-0.46
European history
0.14
0.62
0.15
-0.18
Fashion
2.5
0.66
0.47
-0.33
Film
0.76
0.45
0.16
-0.78
Film industry
1.58
0.46
0.25
-0.59
Fishing industry
1.67
0.57
Floral
1.92
0.89
0.58
-0.09
Food industry
3.64
0.12
0.14
-0.42
Foreign policy
2.4
0.49
0.16
-0.46
Geography
0.88
0.6
0.28
-0.66
Geriatries
2.19
-0.32
-0.56
-0.71
Gynaccology
1.05
-0.27
-0.26
-0.67
Healtheare industry
1.62
-0.25
0.14
-0.5
Hematology
0.35
0.32
-0.05
0.72
History
0.75
0.54
-0.4
-0.8
Holistic medicine
0.85
0.48
0.26
-0.77
Hospitality industry
2.36
0.48
0.28
0007
Housing
4.04
0.15
-0.22
Industrial robot industry
3.84
1.22
0.84
-0.71
Infections disease
1.76
0.14
0.18
-0.56
Insurance industry
2,67
0.42
0.61
-0.4
Intensive care medicine
1.11
0.56
0.08
-0.33
Internal medicine Journalism
1.02
0.45
00.01
-0.42
2.77
-0.13
-0.21
-0.69
Languages and literature
0.45
0.05
-0.39
-0.84
Law
0.42
0.39
0.04
-0.49
Lcisure industry
L.49
19
0.12
-0.09
-0.49
Library and museum studies
1.52
0.5
0.33
-0.32

---

Published in Transactions on Machine Learning Research (07/2025)
Discipline
Orca-2-7b
Mistral-7BB-Instruut-hhb.
WizardLM-13BB-11.2 GPT-4
Linguistics
0.39
0.38
-0.12
-0.96
Logic
2.95
1.56
1.62
-0.79
Materials Science and Engineering
1.71
0.97
0.54
-0.91
Mathematics
4.69
3.81
2.73
-0.61
Mechanical Engineering
2.25
1.71
1.15
-0.95
Medical toxicology
0.62
0.11
-1.01
Medicine
1.49
0.93
0.36
-0.37
Military sciences
0.42
0.53
0.17
-0.45
Mining
3.17
0.32
0.41
-0.61
Music
2.85
0.38
1.07
-0.05
Music industry
2.05
-0.03
-0.08
-0.8
Nursing
1.49
0.14
-0.12
-0.59
Nutrition
1.15
-0.2
-0.13
-0.65
Obstetrics
1.49
0.08
-0.43
-0.53
Ophthalmology
0.97
0.01
-0.47
-0.97
Otolaryngology
1.51
-0.44
-0.29
-1.11
Pathology
0.23
0.35
0.19
-0.72
Pediatrics
1.62
0.55
-0.34
-0.47
Performing arts
0.38
0.09
-036
1.06
Petroleum industry
3.12
0.44
0.08
-0.44
Pharmaceutical industry
2.75
0.41
0.4
-0.46
Pharmaceutical sciences
0.77
0.19
0.16
-0.8
Philosophy
0.51
0.25
0.49
-0.64
Physies
3.15
2.67
2.05
-0.73
Political science
0.04
-0.05
-0.31
-0.91
Prehistory
0.35
0.19
0.05
-0.41
Preventive medicine
2.69
0.57
0.09
-0.36
Psychiatry
2.93
0.27
-0.07
-0.32
Psychology
0.53
-0.02
-0.3
-0.96
Public administration
0.94
-0.27
0.1
-1.2
Public health
1.21
0.07
0.22
-0.56
Public policy
0.78
-0.06
-0.28
-0.92
Pulp and paper industry
1.13
0.63
0.57
-0.25
Radiology
-0.17
-0.19
-0.82
-0.62
Real estate industry
1.01
0.02
-0.12
-0.5
Religious Studies
0.38
-0.32
-0.63
Retail industry
1.1
-0.25
-0.37
-0.6
Semiconductor industry
1.49
0.64
0.71
-0.42
Sexology
1.81
-0.44
-0.37
-0.96
Shipbuilding industry
1.54
0.37
0.42
-0.32
Social work
0.93
-0.42
-0.53
-0.77
Sociology
1.49
0.21
0.76
-0.3
Steel industry
0.88
0.45
0.09
-0.31
Surgery
0.86
-0.02
-0.35
-0.73
Systems science
1.9
0.56
0.41
-0.45
Telecommuniattions industry
1.81
0.4
0.39
-0.27
Television
0.37
-0.33
-0.69
-1
Textile industry
0.82
-0.26
-0.68
-0.59
Theatre
0.31
-0.27
-0.34
-1.07
Theology
-0.38
0.37
-0.45
-0.54
Tobacco industry
0.59
-0.13
-0.48
-0.67
Transport industry
1.19
-0.33
-0.36
-0.56
Transportation
1.74
0.26
0.17
-0.74
Urology
0.05
-0.29
-0.36
-0.64
Veterinary medicine
-0.14
0.36
-0.31
-0.62
Video game industry
1.67
0.2
-0.24
-0.62
Visual arts
0.98
0.22
0.26
-0.6
Water industry
0.9
-0.11
-0.09
-0.51
Wood industry
1.36
0.5
0.31
-0.25
20


Reviews:
[{"review_id": "rGPe6Ct0S9", "reviewer": "Reviewer_BjiJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper demonstrates that it is possible to generate effective instruction tuning datasets from scratch using powerful models like GPT-4, without relying on seed data. Models trained on these synthetic datasets outperform those trained on instruction data bootstrapped from existing sources.\n- By proposing a scratch-from-LLM approach to dataset construction—an area not explored in prior work—the authors open up a promising new direction for synthetic data generation, particularly for training instruction-following models capable of solving a wide range of academic and knowledge-based tasks.\n\n## Weaknesses\n\n- The taxonomy created by GPT-4 and the answers generated by GPT-3.5 are not guaranteed to be accurate or trustworthy. While the dataset appears effective for instruction tuning, the paper does not discuss this limitation. Incorporating retrieval-augmented generation could potentially improve reliability.\n- Most experiments focus on exam-style benchmarks, and the generated dataset predominantly consists of exam-type data. As a result, the proposed approach may appear more effective than it actually is in general-purpose instruction tuning. The lack of evaluation on more standard instruction-tuning benchmarks such as AlpacaEval 2 [1] or MT-Bench [2] makes it difficult to assess the practical utility of the resulting instruction-tuning dataset. The paper does not discuss this limitation, which weakens its broader applicability and takeaway.\n- Although discussed briefly in Appendix A.5, the data generation cost appears excessively high. It is unclear whether the gains from such an expensive data generation process justify the cost. While the idea is academically interesting, its practical value and scalability remain questionable.\n\n[1] Li et al., AlpacaEval: An automatic evaluator of instruction-following models.\n\n[2] Zheng et al., Judging LLM-as-a-judge with MT-Bench and Chatbot Arena, NeurIPS Datasets and Benchmarks 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores a method for constructing a general-purpose instruction tuning dataset without relying on any seed data.\n\nSpecifically, the authors take inspiration from human education systems to build a taxonomy tree that captures a broad range of human knowledge. Based on this taxonomy, they propose a method for generating instruction tuning data.\n\nThe taxonomy tree is built by prompting GPT-4, beginning with a high-level prompt like \"list all fields of human knowledge and capabilities.\"\nThe tree is then expanded by generating subfields for each main category. Subsequently, GPT-4 and GPT-3.5 are used to generate subject descriptions, syllabi, instructions, and answers.\n\nModels tuned with the resulting synthetic data outperform those trained on datasets derived from existing instruction tuning corpora in a variety of knowledge-intensive tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing the concerns listed in the Weaknesses section would greatly improve the paper.\n\nIn particular, it would be beneficial if the authors explicitly acknowledged the limitations regarding (1) the reliability of GPT-4/GPT-3.5-generated data, and (2) the narrow evaluation scope—focused primarily on exam-style tasks—which constrains the generalizability of their method.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oyj3sBLowa", "reviewer": "Reviewer_Tbup", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- taxonomy-driven design ensures systematic coverage across hundreds of  disciplines\n- fully automated pipeline removes dependency on human‐written seeds, enhancing scalability over SELF-INSTRUCT’s reliance on initial task seeds \n- customizable expansion: new knowledge nodes can be added without regenerating existing data \n- extensive zero-shot evaluation on diverse benchmarks demonstrates broad applicability\n\n# Weaknesses\n- novelty may be incremental given parallels to Instruction Tuning with Human Curriculum and Unnatural Instructions \n- high API cost (~$360 K for 10 M queries) and lack of open-source LLM ablation limit reproducibility \n- no automated filtering for LLM hallucinations or factual errors, risking error propagation \n- generates only single-turn Q&A, omitting multi-turn conversational dynamics essential for dialogue systems \n- broader impacts section lacks concrete bias detection or mitigation strategies", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents GLAN, a curriculum-inspired pipeline that, without any human-written seed examples, automatically constructs a taxonomy of human knowledge (fields → subfields → disciplines), derives subjects and detailed syllabi (class sessions + key concepts), and samples combinations of these to generate over 10 million synthetic instruction–response pairs via LLM prompting, then fine-tunes Mistral 7B on this data to achieve competitive zero‐shot performance across GSM8K, MATH, HumanEval, MBPP, BBH, ARC, MMLU, and IFEval benchmarks", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "provide open-source LLM ablation (e.g., LLaMA 3) to validate model-agnostic claims and affordability (critical) \n\nintegrate automated data‐quality filters (fact-checking, perplexity thresholds) to remove low‐quality or hallucinatory samples (critical) \n\npublish full taxonomy and human-verification criteria (with examples of pruned vs. retained nodes) for reproducibility (critical)\n\ninclude direct performance comparison to Instruction Tuning with Human Curriculum and Unnatural Instructions to clarify novelty \n\ndemonstrate extension to multi-turn dialogues, e.g., via multi-agent conversational data synthesis frameworks (strengthening)", "broader_impact_concerns": "synthetic data may propagate and amplify social biases present in GPT-4 \n\nlarge-scale API usage incurs significant energy consumption, raising sustainability concerns\n\nreliance on proprietary APIs exacerbates resource disparities among researchers", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0vPVtL2A9X", "reviewer": "Reviewer_zfPc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The paper is well-written and easy to follow.\n- The proposed instruction tuning data generation method makes sense and the authors provide a rationale for each design choice.\n- The constructed instruction tuning dataset can benefit the community if published.\n\n\nWeakness\n- It would be great to showcase what the constructed taxonomy looks like.\n- The cost of this method seems to be very high. Can we use some open-source LLMs to replace GPT-4 and GPT-3?\n- The experiments seem to be only conducted on the Mistral 7B base model. It would strength the paper if further experiments are conducted on more base models, e.g., Llama-base and Gemma-base.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces generalized instruction tuning (GLAN), a general and scalable method for LLM instruction tuning. GLAN proposes to have a pre-curated taxonomy of human knowledge and capabilities as input, which serves as guidelines for large-scale synthetic instruction data generation. Furthermore, they propose to decompose human knowledge into various fields, sub-fields, and distinct disciplines semi-automatically. The diverse instructions are then generated based on the fine-grained key concepts from the class session of the syllabus. The authors conduct experiments on Mistral-7b models and demonstrate the effectiveness of the proposed GLAN method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper looks good to me overall. There is no clear request for major revision. However, I would recommend the author to add some experiments on other LLM base models, e.g., Llama-base and Gemma-base, in addition to Mistral models.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)