Title: Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Venue: NeurIPS 2024
Accepted: no
Keywords: Synthetic data; large language models
Abstract: We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). 
Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, 
GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale 
synthetic instruction data across all disciplines.
Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. 
Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs.
With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. 
Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.

Abstract

We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction-tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.

1 Introduction

Large Language Models (LLMs) have enabled unprecedented capabilities to understand and generate text like humans. By scaling up model size and data size [17, 13], LLMs are better at predicting next tokens and prompting to perform certain tasks with a few demonstrations [2]. However, these capabilities do not directly translate to better human instruction following [25]. Instruction tuning [34] bridges this gap by fine-tuning LLMs on instructions paired with human-preferred responses.

Prior work constructs instruction tuning data from seed examples or existing datasets. Initially, natural 25

language processing (NLP) datasets described via instructions are used to fine-tune LLMs and the 26

27

resulting LLMs can generalize on unseen (NLP) tasks [34]. However, there are only thousands of

NLP tasks [33, 19] available, which limits the tuned LLMs to generalize in real-world scenarios [39]. 28

Self-instruct [32] is a cost-effective method for creating synthetic instruction tuning datasets, which 29

starts from a small pool of human-written seed instructions and generates new instructions by few30

shot prompting an LLM (e.g., text-davinci-002 ) with randomly selected instructions from the 31

pool. Unfortunately, the diversity of generated instructions is still an issue, since few-shot prompting 32

tends to generate new instructions similar to its demonstrations. In addition, the process of creating 33

high-quality seed instructions requires considerable human effort and expertise. Evolve-Instruct [39] 34

improves self-instruct by augmenting existing instruction tuning datasets with different rewriting 35

operations using LLMs, which is essentially data argumentation. Consequently, the scope of domains 36

GLAN

Taxonomyof

Human Knowledge

FLAN

Self-lnstruct

Evolve-lnstruct

&Capability

Many

Seed

Existing

LLM

NLP Datasets

Examples

Dataset

BreakDown

Described via

LLM

LLM

Instructions

Few-shot

Data

Sample

Expansion

Augmentation

LLM

Instruction

Instruction

Instruction

Generate

Dataset

Dataset

Dataset

Instruction

Dataset

or tasks that these augmented datasets can cover is limited by the original input datasets. See Figure 1 37 for illustrations of these methods described above. There are also studies concentrated on developing 38 instruction-tuning datasets tailored to particular domains or tasks. For instance, [20] creates datasets 39 targeting mathematical reasoning. In contrast, [3] and [21] focus on coding-related tasks. All of the 40 above methods cannot produce instruction datasets that are generally applicable to a wide range of 41 domains. 42

How to create a general instruction tuning dataset? We draw inspiration from the systematic structure 43 in human education system. The structure of human education includes several levels, starting 44 from early childhood education up to higher education and beyond [37]. Within each level, a 45 student acquires knowledge, skills, and values in a systematic process. The courses a student learns 46 from primary school to college cover a broad range of knowledge and skills, which facilitates the 47 development of a diverse array of abilities. We believe that the systemic framework of the human 48 education system has the potential to help the generation of high-quality and general instruction data, 49 which spans a diverse range of disciplinary areas. 50

In this paper, we introduce a generalized instruction tuning paradigm GLAN (shorthand for 51 G eneralized Instruction-Tuning for L arge L AN guage Models) to generate synthetic instruction 52 tuning data almost from scratch. Unlike existing work [39, 21, 20, 24], GLAN exclusively utilizes 53 a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale 54 instruction data systematically and automatically across all disciplines. Specifically, inspired by 55 the structure of the human education system, the input taxonomy is constructed by decomposing 56 human knowledge and capabilities to various fields, sub-fields, and, ultimately, distinct disciplines 57 semi-automatically, facilitated by LLMs and human verification. The cost of human verification 58 process is low due to the limited number of disciplines in the taxonomy. As shown in Figure 1, 59 we then further break down these disciplines into even smaller units. We continue to generate a 60 comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each 61 subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session 62 of the syllabus, we can first sample from them and then generate diverse instructions with broad 63 coverage across the entire spectrum of human knowledge and skills. The process described above 64 mirrors the human educational system, where educators in each discipline craft a series of subjects 65 for student learning. Instructors then develop a syllabus for each subject, breaking down the content 66 into specific class sessions. These sessions are then further divided into core concepts that students 67 must comprehend and internalize. Based on these detailed core concepts outlined in the syllabus, 68 teaching materials and exercises are subsequently created, which are our instruction tuning data. 69

GLAN is general, scalable and customizable. GLAN is a general method, which is task-agnostic 70 and is capable of covering a wide range of domains. GLAN is scalable. Similar to [32, 39], GLAN 71 generates instructions using LLMs, which can produce instructions on a massive scale. Moreover, the 72 input of GLAN is a taxonomy, which is generated by prompting an LLM and human verification, 73 requiring minimal human effort. GLAN allows for easy customization. New fields or skills can be 74 added by simply incorporating a new node into our taxonomy. Note that each node of the taxonomy 75 can be expanded independently, which means that we only need to apply our method to the newly 76 added nodes without re-generating the entire dataset. Extensive experiments on large language 77 models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical 78 reasoning, coding, academic exams, and logical reasoning to general instruction following without 79 using task-specific training data of these tasks. 80

81

2 GLAN: Generalized Instruction-Tuned Language Models

GLAN aims to create synthetic instruction data covering various domains of human knowledge 82 and capabilities on a large scale. As shown in Algorithm 1, we first build a taxonomy of human 83 knowledge and capabilities using frontier LLMs (i.e., GPT-4 ) and human verification. The taxonomy 84 naturally breaks down human knowledge and capabilities to fields , sub-fields , and ultimately different 85 disciplines (see Section 2.1). The following steps are fully autonomously facilitated by GPT-4 (or 86 GPT-3.5 ). Then for each discipline, we again instruct GPT-4 to further decompose it into a list of 87 subjects within this discipline (Section 2.2). Similar to an instructor, GPT-4 continues to design 88 a syllabus for each subject, which inherently breaks a subject into various class sessions with key 89 concepts students need to master (Section 2.3). With obtained class sessions and key concepts, we 90 are ready to construct synthetic instructions. We prompt GPT-4 to generate homework questions 91 based on randomly sampled class sessions and key concepts as well as the syllabus (Section 2.4). 92 We recursively decompose human knowledge and capabilities into smaller units until atomic-level 93 components (i.e., class sessions and key concepts). We expect to randomly combine these class 94 sessions and key concepts to ensure the coverage and diversity of synthetic instructions. 95

96

2.1 Taxonomy of Human Knowledge and Capabilities

We build a taxonomy of human knowledge and capabilities to guide the generation of synthetic 97 instructions. Therefore, its coverage is important. On the other hand, it is also essential to make 98 the taxonomy highly extensible, since the preferred capabilities of LLMs may change over time. 99 In the first step, we propose to generate the taxonomy by prompting GPT-4 with a set of different 100 instructions (e.g., list all fields of human knowledge and capabilities ). Then, we do 101 human post-editing to ensure its correctness and completeness. Due to the limited number of fields, 102 sub-fields, and disciplines in our taxonomy, the cost of human verification is reasonably low. Another 103 advantage of human post-editing is that we can easily add new fields or disciplines to the taxonomy 104 as needed. 105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

Our taxonomy currently covers a diverse range of knowledge and capabilities in both academic education and vocational training. The top level of the taxonomy contains fields such as Natural Sciences , Humanities , or Services (vocational training). These fields branch out to various sub-fields and/or disciplines such as Chemistry , Sociology or Retailing . We keep breaking down nodes of the taxonomy until disciplines , and we leave the breaking down of disciplines to automatic methods described in the following sections. By collecting the leaf nodes of the taxonomy, we obtain a list of disciplines D = { d 1 , d 2 , . . . , d M } .

2.2 Subject Generator

As in Algorithm 1, for each discipline d , we aim to extract the list of subjects in it through prompt engineering. Specifically, we instruct GPT-4 to act as an education expert of discipline d and design a list of subjects a student should learn . The completion of GPT-4 contains a comprehensive list of subjects and their meta data (e.g., level, introduction and subtopics of the subject) in unstructured text format, which can not be directly used in subsequent steps. We therefore used another round of prompting to convert the completion to JSONL format:

Awesome! Transform the above to JSONL format so that it is easier for a computer to understand. Enclose the JSONL output between two sets of triple backticks. For each JSONL object, use the keys 'subject_name', 'level' and 'subtopics'.

It is worth noting that generating a subject list in JSONL format using a single prompt is feasible. However, we refrain to do so, because we observe that incorporating additional formatting instructions directly into the prompt can compromise the quality of the resulting subject list. These extracted subjects (as well as their meta data) S = { s 1 , s 2 , . . . , s N } can be subsequently used in next steps. For each s ∈ S , let s.name , s.level and s.subtopics denote the name, grade level and subtopics of subject s , respectively. We can apply the above prompts multiple times to ensure better coverage of subjects within this discipline.

2.3 Syllabus Generator

For each subject s , we have already extracted its name ( s.name ), grade level ( s.level ), and a small set of included sub-topics ( s.subtopics ) in a structured format. In this section, we aim to further segment each subject into smaller units, making them more suitable for creating homework assignments. We consult GPT-4 to design a syllabus for this subject. We opt for syllabus generation for the following reasons. Firstly, a syllabus essentially breaks down the main topic of a subject into smaller segments in a hierarchical manner. Specifically, each subject comprises several class sessions, and each session covers a variety of sub-topics and key concepts. Secondly, a syllabus provides an introduction, objectives, and expected outcomes of a subject, which are inherently useful for formulating homework questions. We instruct GPT-4 to 1) design a syllabus based on its meta data ( s.level , s.name and s.subtopics ); 2) break the subject into different class sessions; 3) provide details for each class session with a description and detailed key concepts students need to master.

Let A denote the generated syllabus. The resulting syllabus A is in unstructured text format. However, class session names and key concepts of each class are required in the instruction generation step (see Algorithm 1). Similar to the process of subject list extraction in Section 2.2, we again extract these meta data of each class session by prompting GPT-4 . As a result, we obtain a list of class sessions C = { c 1 , c 2 , . . . , c | C | } and their corresponding key concepts K = { k 1 , k 2 , . . . , k | C | } . The detailed prompt for syllabus generation is in Appendix A.3.

2.4 Instruction Generator

Given a syllabus A as well as a list of its class sessions C and their associated key concepts K , we are ready to generate homework questions and their answers. To generate diverse homework questions, we first sample one or two class session names from C and one to five key concepts under these selected class sessions. Let ˆ C denote the selected class session names and ˆ K the selected key concepts. Then we prompt GPT-4 (or GPT-3.5 ) to generate a homework question given the selected class sessions ˆ C and key concepts ˆ K as well as the syllabus A . We intend to give GPT-4/3.5 more

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

context (e.g., what students have already learned in previous sessions) when creating assignments. Therefore, we additionally instruct GPT to consider that students have learned up to class sessions ˆ C when crafting homework and try to leverage multiple key concepts across different class sessions. See details of our prompt for instruction generation in Appendix A.4.

Sampling Class Sessions and Key Concepts In a single syllabus, there are numerous class sessions and key concepts. We have two strategies to sample from them. In the first strategy, we generate assignments from a single class session. Therefore, we have only one class session name. Suppose we have m key concepts in total in this session. We randomly sample one to five key concepts from the m key concepts, which means we have totally ∑ 5 i =1 ( m i ) combinations. In this strategy, we focus on creating basic homework questions. To make the resulting questions more challenging (combine knowledge from multiple class sessions), we propose a second strategy to combine key concepts from two class sessions in the second strategy. We intend to generate questions leverage knowledge from two different class sessions. Suppose we have m 1 and m 2 key concepts in the first and second class sessions, respectively. We can have ∑ 5 i =2 ( m 1 + m 2 i ) -∑ 5 i =2 ( m 1 i ) -∑ 5 i =2 ( m 2 i ) different combinations, which is significantly more than that of the first strategy. We use both strategies to ensure our created questions are diverse in difficulty levels.

Answer Generation After we generate questions in previous steps, we simply send these questions to GPT-3.5 and collect answers. We use GPT-3.5 for answer generation, because we find the quality of generated answers from GPT-3.5 is sufficiently good and using GPT-3.5 is significantly faster than GPT-4 . The resulting question-answer pairs are our instruction tuning data. With a huge amount of question-answer pairs ranging from different disciplines with various difficulty levels, we expect the resulting LLM can excel in a wide range of tasks.

3 Experiments

3.1 Data Generation

Taxonomy Creation By asking GPT-4 to create a taxonomy of human knowledge and capabilities, we end up with a set of fields, sub-fields, and disciplines that cover a broad range of domains in human knowledge and capabilities. Next, we ask human annotators to decide whether these elements in the taxonomy should be kept or not in order to reduce the redundancy of the taxonomy while maintaining its correctness. Note that if a field or sub-field is marked as remove , we remove its descendant as well. We kept 126 disciplines after majority voting (provided in supplementary materials). Note that it is feasible to manually add extra disciplines, sub-fields, or fields whenever necessary.

Subject and Syllabus Generation During the subject list and syllabus generation, we prompt GPT-4 and employ nucleus sampling [14] with temperature T = 1 . 0 and topp = 0 . 95 to encourage diversity. We do not use GPT-3.5-turbo since some subjects belong to the long-tail distribution which may not be effectively modeled by GPT-3.5-turbo . To ensure diversity and completeness of the generated subjects, we query GPT-4 10 times for each discipline (Section 2.2). There are 100 to 200 subjects for each discipline on average. It is worth noting that the same subjects may appear in different disciplines. For instance, the subject calculus is both in physics and mathematics. We do not de-duplicate those subjects, since it may reflect their importance in human knowledge. Given a subject in a specified discipline, we query GPT-4 for only one time to design a syllabus (see details in section 2.3). The temperature and topp are still set to 1.0 and 0.95, respectively. The number of class sessions contained in each syllabus varies from 10 to 30 and each class session contains around five key concepts.

Instruction Generation Each instruction data consists of a question and its answer. We choose to generate questions and answers separately since we observed that separate generations lead to better quality. After question generation with GPT-4 , each question is then answered by GPT-3.5-turbo with temperature T = 0 . 7 , topp = 0 . 95 (we use a lower temperature in order to make the resulting answers more accurate). We use GPT-3.5-turbo instead of GPT-4 for answer generation, because GPT-3.5-turbo is significantly faster with reasonably good results. We generate 10 million instruction-response pairs in total and then we do training data decontamination. Specifically, the training instruction-response pairs are decontaminated by removing pairs that contain questions or

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

input prompts from the test and training (if any) sets of benchmarks we evaluate. We exclude the training set of benchmarks we evaluate to verify the generalization capability of our synthetic data.

3.2 Model Training

We employ Mistral 7B [16] as our base model. During training, we concatenate each instruction and response pair to a single sequence and only compute loss on response tokens. We train our model for 3 epochs with a learning rate of 3 e -6. The batch size is set to approximately 512 instruction-response pairs. We employ a dynamic batch size to ensure a constant total number of tokens per batch. We use a cosine learning rate schedule and we start with a linear warm-up of 1000 steps and the final learning rate is reduced to 0. The training requires approximately 8 days using 32 A100 GPUs.

3.3 Benchmark Evaluation

The instruction data GLAN generated spans a wide range of subjects. We evaluate its effectiveness in mathematical reasoning, coding, logical reasoning, and academic exams.

Mathematical Reasoning : Mathematics is a common subject in many different disciplines. Hence, it is necessary to test the math reasoning ability of GLAN. We choose the two popular benchmarks for evaluation (i.e., GSM8K [7] and MATH [12]). GSM8K [7] is a high-quality math problem dataset that measures the basic multi-step mathematical reasoning ability. It contains around 7k problems for training and 1K problems for test. MATH [12] is a challenging math dataset that contains mathematics competition-level problems from AMC, AIME, etc. The 7.5k training and 5K test problems cover seven math subjects, i.e., Prealgebra, Precalculus, Algebra, Intermediate Algebra, Number Theory, Counting and Probability, and Geometry. Note that GLAN does not use any examples in the training set of GSM8K or MATH. Following [20], we report 0-shot setting results for GLAN. Coding : To evaluate the coding capability of GLAN, we opt for two coding benchmarks HumanEval [4] and MBPP [1]. We employ 0-shot setting for HumanEval and 3-shot setting for MBPP following prior art [4, 21]. BBH : The instruction dataset we generated covers many disciplines, which can potentially enhance the reasoning ability of GLAN. Therefore, we evaluate GLAN on the BIG-Bench Hard dataset (BBH [29]), which contains 23 challenging tasks from Big-Bench [28]. We employ the standard 3-shot setting with chain-of-thought demonstrations. Academic Exams : We also evaluate GLAN on different academic benchmarks to verify whether GLAN is capable of solving exam questions. We choose two benchmarks (i.e., ARC [6] and MMLU [11]). Both benchmarks are composed of multi-choice questions. AI2 Reasoning Challenge (ARC [6]) contains grade-school level, multi-choice science questions. It contains two sub-sets, which are ARC-Challenge (ARC-C) and ARC-Easy (ARC-E). Massive Multitask Language Understanding (MMLU [11]) consists of a set of multiple-choice questions about 57 subjects ranging in difficulty from elementary levels to professional levels. It covers various of domains of knowledge, including humanities, STEM and social sciences. Note that there is a training set for ARC. However, we have excluded it from our

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

5e4

2e5

5e5 1e6

1e7

42

43

44

45

46

47

48

49

Pass@1

HumanEval

5e4

2e5

5e5 1e6

1e7

58.5

59.0

59.5

60.0

60.5

Exact Match

BBH

5e4

2e5

5e5 1e6

1e7

60

65

70

75

80

Exact Match

5e4

2e5

5e5 1e6

1e7

20

22

24

26

28

30

32

Exact Match

MATH

training set during the decontamination process described in Section 3.1. Previous models mostly leverage probability-based methods on ARC and MMLU, which returns the best option based on the probabilities of the four options conditioned on the corresponding multi-choice question. We observe that after training on 10 million instructions, GLAN is able to generate its predicted options and analysis of multi-choice questions in plain text as GPT-3.5 does. We therefore opt for 0-shot setting for GLAN and extract predictions using rules based on its completions as in [22].

Results Our main results are shown in Table 1. We compare GLAN against general domain models (Orca 2 [22], Mistral Instruct [16] and WizardLM [39]), math optimized models (MetaMath [40] and WizardMath [20]) and coding optimized models (CodeAlpaca [3]). We also report results of base LLMs (i.e., LLaMA2 [31] and Mistral [16]) as references. GLAN either obtains the best results or results close to the best across all benchmarks. We observe that capabilities of math or coding optimized models increase on math or coding benchmarks while usually not others. After instruction tuning, GLAN excels on multiple dimensions from mathematical reasoning, coding, reasoning, and academic exams with a systematical data generation approach. Also note that our method does not use any task-specific training data such as training sets of GSM8K, MATH, or ARC as in Orca 2, MetaMath, and WizardMath, which indicates the general applicability of GLAN.

A Closer Look at Academic Exams ARC and MMLU are all multi-choice based benchmarks on academic exams. However, we observe that improvements of GLAN over Mistral on ARC are much larger than these on MMLU (see Table 1). By grouping the 57 subjects in MMLU into four categories (i.e., STEM, Humanities, Social Sciences, and Other (business, health, misc.)), we observe GLAN wildly improves on STEM in MMLU while not in other categories (Table 2). Also note that ARC is composed of high school science problems, which are also STEM questions. GLAN is good at STEM subjects may be because responses of our dataset are from GPT-3.5-turbo , which by default generates responses with Chain-of-Thoughts (CoT) reasoning. Indeed, we observe that GLAN generates solutions with CoT for multi-choice questions. CoT may help the multi-step reasoning in STEM multi-choice questions [35], while humanities and social sciences questions involve more memorization and single-step reasoning, where CoT may introduce additional errors.

3.4 Scaling Property of GLAN

We investigate the scaling property of GLAN by training Mistral on different numbers of examples 271 (i.e., 50K, 200K, 500K, 1M, and 10M) we generated. The results on downstream tasks are shown in 272 Figure 2. It can be observed that overall task performance tends to increase as we increase the data size. 273 Notably, the curve has not reached a plateau, indicating the potential for further improvement through 274 the continued scaling of the data size of GLAN. However, we defer further scaling experiments to 275

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

3.5 Task-specific Training Data

GLAN is a generalized method to create synthetic data for instruction tuning. In order to evaluate the generalization capabilities of this synthetic data, we deliberately exclude task-specific training sets from all benchmarks on which we conduct our assessments. Similar to [36], we explore whether models have been trained on task-specific in-domain data. We compute the training loss L train and test loss L test on ARC Challenge (ARC-C), GSM8K, and MATH for GLAN and other models in comparison. We choose these datasets because among all benchmarks evaluated in Section 3.3, these benchmarks contain training sets. Intuitively, the larger ∆ = L test -L train is, the more likely the training set is exposed. To make ∆ easier to interpret, we additionally compute the relative difference ∆(%) = ( L test -L train ) /L test . Table 3 shows the losses of the training and test splits for GLAN are nearly identical (or ∆ is negative). This suggests that GLAN has not been exposed to in-domain data during training and tuning procedures. Please refer detailed L train and L test losses in Table 8 (in Appendix). Additionally, as shown in Table 8, we observe that GLAN obtains higher losses on both test and training splits on GSM8K, MATH, and ARC compared to other models, while performances of GLAN on these datasets are high (see Table 1). This might imply that synthetic data generated by GLAN is diverse and our resulting model avoids convergence to any specific domain or style present in existing benchmarks.

3.6 Instruction Following Evaluation

IFEval We assess the instruction-following capabilities of GLAN utilizing the Instruction Following Evaluation dataset (IFEval [42]). IFEval consists of a collection of 'verifiable instructions', encompassing 25 distinct types of instructions (around 500 prompts in total). Each prompt comprises one or more verifiable instructions. The evaluation involves four types of metrics at both prompt level and instruction level, evaluating strict and loose accuracies. As shown in Table 4, GLAN demonstrates superior instruction-following capabilities in both prompt-level and instruction-level evaluations. However, there is still a considerable gap compared to GPT-3.5-turbo and GPT-4 .

Evol-Instruct Test Evol-Instruct testset [39] contains real-world human instructions from diverse sources, and it consists of 218 instances with 29 distinct skills. Each instruction is associated with a difficulty level from 1 to 10. The responses are often open-ended descriptions, and we believe this benchmark is a necessary supplement to IFEval (answers to their instructions are 'verifiable'). Following [39] and [5], we adopt a GPT-4-based automatic evaluation method to conduct a pairwise comparison between GLAN and other models. Specifically, GPT-4 is instructed to assign a score between 1 and 10 overall score w.r.t. the helpfulness, relevance, accuracy, and level of detail of

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

responses generated by two different models for a given input question. A higher score indicates better overall performance. To mitigate potential order bias, we perform bidirectional comparisons for each response pair and determine their average score. The average score difference to GLAN (i.e., avg_score( GLAN ) -avg_score ( x ) ) serves as the final metric. Table 5 presents the results of pairwise comparisons across various levels of instruction difficulty. GLAN showcases superior performance compared to LLaMA-2, Orca 2, Mistral Instruct, and even WizardLM-13B (note that GLAN contains only 7B parameters) on most difficulty levels and overall scores. This suggests that GLAN demonstrates improved ability to process diverse instructions, regardless of their difficulty or complexity. Also, note that GLAN falls behind GPT-3.5-turbo as other models in comparison. Additionally, we group Evol-Instruct test according to the 29 skills and observe the same trends. Detailed results are listed in Appendix (Table 9 and 10). GLAN demonstrates strong performance on most skills, especially in Math, Coding, and Reasoning. However, it slightly falls short in commonsense related tasks. We also created GLAN-Test, similar to the Evol-Instruct Test but much larger in size, where GLAN outperforms other models as well (see Appendix A.8).

4 Related Work

Recent literature has extensively explored the collection of various human-made resources for instruction tuning. An intuitive direction is to collect existing NLP datasets and corresponding task descriptions [26, 33, 41], typical LLMs such as BLOOMZ [23] and FLAN [34] are trained on this type of instruction tuning data. However, with only tens to thousands of existing datasets available, the scope and diversity of instruction tuning are inevitably limited. Another common practice is to implement instruction tuning with real-world human user prompts. For instance, InstructGPT [25] was trained on high-quality human prompts submitted by real-world users to OpenAI GPT APIs. Vicuna [5] leverages user-shared prompts along with ChatGPT responses for instruction tuning, and Dolly[8] was trained on simulated human-user interactions written by over 5k employees. Nevertheless, acquiring instructional data from human users typically involves high costs and involves privacy concerns. As LLM capabilities improve, instruction tuning with LLMgenerated data exhibits better scalability and potential in addressing the super-alignment problem [27]. Leveraging the in-context learning ability of LLMs, Unnatural instructions [15] and Self-instruct [32] sampled seed instructions as examples to elicit LLMs to generate new instructions. Taking advantage of the rephrasing ability of LLMs, WizardLM [39] and WizardMath [20] were trained using EvolInstruct. Evol-Instruct iteratively employs ChatGPT to rewrite seed instructions into increasingly complex instructions. Similar to generation from seed instructions, carefully selected seed topics are used for generating textbook-like synthetic data [18] or self-chat multi-turn dialogues [38, 9] for instruction tuning. However, models trained on these LLM-generated data only work well in specific domains such as math [20, 40], dialogue [38, 9] or open-ended question answering [30, 39]. These methods encounter challenges in generalization [10], as the data diversity is restricted by seed instructions or seed topics.

5 Conclusions

We propose GLAN, a general and scalable method for synthesizing instruction data. Experiments 347 show that GLAN can help large language models improve their capabilities in multiple dimensions, 348 from mathematical reasoning, coding, academic exams, and logical reasoning to general instruction 349 following. Currently, our synthetic data are based on the taxonomy of human knowledge and 350 capabilities, and there are other types of useful data that have not been covered. We are interested in 351 designing methods with border coverage. Our current instruction data are mostly question-answer 352 pairs, and in the next step, we plan to generate synthetic data of multi-turn conversations and long 353 documents. 354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

References

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 2020.

S. Chaudhary. Code alpaca: An instruction-following llama model for code generation. https: //github.com/sahil280114/codealpaca , 2023.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023.

N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233 , 2023.

A. Gudibande, E. Wallace, C. V. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine, and D. Song. The false promise of imitating proprietary language models. In International Conference on Learning Representations , 2024.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021.

D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems , 2021.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems , 2022.

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In International Conference on Learning Representations , 2020.

O. Honovich, T. Scialom, O. Levy, and T. Schick. Unnatural instructions: Tuning language 397 models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the 398 Association for Computational Linguistics (Volume 1: Long Papers) , 2023. 399

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, 400 G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023. 401

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad402 ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint 403 arXiv:2001.08361 , 2020. 404

405

406

Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 , 2023.

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , 2023.

H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 , 2023.

Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 , 2023.

A. Mitra, L. Del Corro, S. Mahajan, A. Codas, C. Simoes, S. Agarwal, X. Chen, A. Razdaibiedina, E. Jones, K. Aggarwal, et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045 , 2023.

N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. Le Scao, M. S. Bari, S. Shen, Z. X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023.

S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707 , 2023.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems , 2022.

V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V. Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Févry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022.

T. Shen, R. Jin, Y. Huang, C. Liu, W. Dong, Z. Guo, X. Wu, Y. Liu, and D. Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025 , 2023.

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research , 2023.

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023 , 2023.

R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca , 2023.

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 2023.

Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat, S. Mishra, S. Reddy A, S. Patro, T. Dixit, and X. Shen. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022.

J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chainof-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , 2022.

T. Wei, L. Zhao, L. Zhang, B. Zhu, L. Wang, H. Yang, B. Li, C. Cheng, W. Lü, R. Hu, C. Li, L. Yang, X. Luo, X. Wu, L. Liu, W. Cheng, P. Cheng, J. Zhang, X. Zhang, L. Lin, X. Wang, Y. Ma, C. Dong, Y. Sun, Y. Chen, Y. Peng, X. Liang, S. Yan, H. Fang, and Y. Zhou. Skywork: A more open bilingual foundation model, 2023.

Wikipedia contributors. Education, 2023. Last edited on 24 March 2023.

C. Xu, D. Guo, N. Duan, and J. McAuley. Baize: An open-source chat model with parameterefficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 , 2023.

L. Yu, W. Jiang, H. Shi, J. YU, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations , 2024.

C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. YU, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems , 2023.

J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction487 following evaluation for large language models. arXiv preprint arXiv:2311.07911 , 2023. 488

A Appendix 489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

A.1 Limitations

While GLAN presents significant advancements in academic benchmarks. However, there may still have several limitations in real world deployment. The resulting LLMs train on generated data using GLAN may occasionally produce factual incorrect (or even toxic) responses. Further training for refusal, hallucination reduction as well as toxic content reduction should be performed before deployment.

A.2 Broader Impacts

Data synthesizing is crucial for the continual scaling of large language models, especially as we exhaust available human data. GLAN demonstrates the potential to generate vast amounts of synthetic data from scratch, paving the way for even larger-scale data synthesis efforts. While GLAN has shown the effectiveness of synthetic data, we must point out that synthetic data may inherit and even amplify social biases present in the frontier LLMs for generation. Future research should focus on developing techniques to identify and correct biases in the generated datasets and models trained on them.

A.3 Prompt for Syllabus Generator

The prompt template for syllabus generation is in Table 6.

You are an expert in { s.name }. Using the given data, design a syllabus for teaching students at the specified level. Note that example subtopics or descriptions are just give you an impression of what this class like. Feel free to add extra subtopics if needed (remember you are the expert in { s.name }). Data: - Level: { s.level } - Main Topic: { s.name } - Description or Example Subtopics: { s.subtopics } ### Syllabus Design Guide 1. **Introduction**: Start with an overview of the primary topic for the syllabus. 2. **Class Details**: For each class session, provide: - **Description**: Briefly describe the focus of the session. - **Knowledge Points**: Enumerate key concepts or topics. These will be used to craft homework questions. - **Learning Outcomes & Activities**: Offer expected learning results and suggest related exercises or activities.

A.4 Prompt for Instruction Generator 506

The prompt template for instruction generator is in Table 7. 507

A.5 Task-specific Training Data 508

We provide the specific train/test values of different models on different benchmarks in Table 8. 509

A.6 Evol-Instruct Test Results on Different Difficulty Levels 510

The concrete Evol-Instruct test results on different difficulty levels are shown in Table 9. 511

A.7 Evol-Instruct Test Results on Different Skills 512

The concrete Evol-Instruct test results on different skills are shown in Table 10. 513

514

A.8 GLAN-Test Overall Results

GLAN-Test There are only hundreds of instructions in In IFEval and Evol-Instruct Test and 515 we believe the domains or skills they can cover are rather limited. Therefore, we propose a held516 out test set using GLAN data and we call it GLAN-Test. It contains 6,300 instructions on 126 517 disciplines (50 instructions for each discipline). We further categorize the 126 disciplines to 8 518 distinct fields (i.e., Academic-Humanities, Academic-Social Science, Academic-Natural Science, 519 Academic-Applied Science, Academic-Formal Science, Industry-Manufacturing, Industry-Services 520 and Industry-Agriculture). We believe that the extensive domain coverage of GLAN-Test renders 521 it an effective test bed for the assessment of generalization capabilities in LLMs. We adopt the 522 same GPT-4 based evaluation protocol as in Evol-Instruct Test (previous paragraph). We prompt 523 GPT-4 to do a pairwise ranking of GLAN and other models in comparison. The overall results and 524 results across the 8 fields are presented in Table 11, where GLAN obtains higher GPT-4 scores than 525 Orca2-7B, Mistral-7B Instruct and WizardLM-13B, despite using only 7B parameters. GLAN still 526

lag behind GPT-4 . Detailed results for the 126 fine-grained disciplines can be found in Appendix 527 A.9 (see Table 12 for more details). GLAN demonstrates its effectiveness on multiple domains (or 528 disciplines) such as Mathematics, Physics, Chemistry, Computer science, Electrical, Mechanical, etc., 529 indicating that smaller models may yield general improvements on various domains through strategic 530 fine-tuning. Furthermore, it is noted that GLAN demonstrates less-than-ideal performance across 531 distinct disciplines such as American history, Divinity, or Radiology. This observation underscores 532 the potential for further refinement and development of our methodology within these domains. 533

A.9 GLAN-Test Results on Different Disciplines 534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 5 and Appendix A.1

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 2 and 3.1, we provide a detailed description of the data generation process. Although we haven't shared the original prompts yet, they are quite simple and customizable. Besides, we are actively working to gain authorization to release them as soon as possible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: While we are temporarily unable to provide open access to the data and code, we are actively working to gain the necessary authorization to release these resources. Once obtained, we will ensure that all data and code, along with detailed instructions, are made available to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 3.2, 3.3

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We did not include error bars in the experiments due to the high computational demands.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We included compute resources in Section 3.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This study strictly adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Appendix A.2

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: To ensure future responsible release, we are still in the process of implementing comprehensive safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used in this paper are properly credited. The license and terms of use are properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Once authorization is obtained, we will ensure that comprehensive documentation is provided alongside the assets to facilitate their proper use and understanding.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

846 847 848 849 850

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces generalized isntruction tuning (GLAN), an approach for synthesizing instruction tuning data using a taxonomy-based approach. GLAN generates synthetic instruction data from pre-curated taxonomy of human knowledge and capabilities and aims to create diverse and broad-ranging instruction dataset.\n\nSTRENGTHS:\n1. Comprehensive Coverage of Evaluation: The paper presents extensive experiments demonstrating that GLAN outperforms various popular instruction-tuned LLMs across multiple dimensions, including mathematical reasoning, coding, logical reasoning, and general instruction following.\n2. Minimization of Human Involvement: The generation process significantly reduces human involvement, requiring human verification only at the taxonomy construction stage. This makes the approach scalable and less labor-intensive.\n3. Customizability and Extensibility: The taxonomy-based approach allows for easy customization and extension. New fields or skills can be incorporated by simply adding new nodes to the taxonomy.\n\nWEAKNESSES:\n1. While the paper addresses generalization, there is a risk that the generated synthetic data might overfit to the taxonomy's structure, potentially missing out on more nuanced, real-world instructions.\n\nQUESTIONS:\n1. Are there any measures in place to ensure the generated synthetic data's diversity and prevent redundancy?\n2. what is the whole taxonomy of the human knowledge and capabilities? And can each task (e.g., gsm8k, arc) be categorised into any sub category?\n3. Are the effects of each category orthogonal to each other? i.e., ablating data from a child category does not effect tasks in another child category. It would be beneficial if the authors could provide some preliminary results.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper introduces GLAN, a general and scalable method for instruction tuning of Large Language Models (LLMs). GLAN employs a top-down approach to generate high-quality instruction tuning datasets. Experiments across various benchmarks demonstrate that GLAN performs comparably to other existing methods.

STRENGTHS:
1. This paper focuses on the alignment of Large Language Models, which is a trendy and important topic. If the dataset is released, it will be beneficial for the community.
2. This method is easy to follow. The process is highly scalable, leveraging LLMs like GPT-4 for generating instructions on a massive scale.
GLAN allows for easy customization. New fields can be added by incorporating new nodes into the taxonomy.

WEAKNESSES:
The novelty is limited as similar top-down designs have been utilized in many previous works. Besides, the main experimental results in Table 1 appear mediocre compared to other methods.

QUESTIONS:
Refer to the weakness.

LIMITATIONS:
Refer to the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalized way of creating instruction data. The high-level motivation is to take inspiration from how curriculum is designed for human learning into a taxonomy of subjects and use the same to prompt an off-the-shelf LLM to create data. GLAN does not need seed examples, or pre built-taxonomy like prior work. Human verification is also performed post the building of taxonomy to weed out unimportant or inaccurate divisions. The overall process is High level taxonomy -> subjects -> syllabus -> instructions.\n\nSTRENGTHS:\n1. Overall strong Performance: Extensive experiments show GLAN's effectiveness in various tasks, outperforming or matching state-of-the-art models in several benchmarks (Table 1)\n2. Figure 2 on scaling properties of GLAN: I found this figure quite interesting. It suggests a log linear scaling trend in performance as GLAN data is scaled up. This is quite promising.\n3. Section 3.5 on Task-specific overfitting: Another great analysis section that discusses how GLAN does not particularly overfit to the training data. This ensures that the synthetic data remains generalizable across different domains.\n4. Modularity of the pipeline: The modular nature of the GLAN pipeline allows for easy customization and extension by incorporating new nodes into the taxonomy without re-generating the entire dataset.\n\nWEAKNESSES:\n1. No use of actual human curriculum: The paper set the expectation right in the abstract of using/getting strongly inspired from human curriculum. I was disappointed that the method does not utilize existing human curriculum structures, potentially missing out on years of insights in developing the same. Generating synthetic data, and in this case entire taxonomies from pre-existsing models can lead to extremely large amounts of bias. I would have much rather seen the authors delegate only lower level questions to LLMs than high level abstractions, which would lead to a trickle down effect on every single node in the taxonomy. This study, in my opinion, is incomplete without using either human generated taxonomies, and/or a comparison between how different the taxonomies are.\n2. Computation cost not compared: The paper does not provide a comparison of computational costs with similar methods, such as WizardLM. For instance, GLAN training required approximately 8 days using 32 A100 GPUs to generate 10 million instructions, but no direct comparisons are made to illustrate the efficiency or cost-effectiveness relative to other approaches.\n3. The method is limited by the performance of GPT-3.5/4: The quality of the generated taxonomy and syllabus heavily depends on the capabilities of the underlying LLMs used in the process, namely GPT-3.5 and GPT-4. In general, GLAN does not inform how we can improve capabilities of models beyond GPT4. But also, does not consider the cost of generating 10 million instructions.\n4. High variability in results (Table 2): There is significant variability in GLAN's performance across different categories, with particularly weaker results in humanities and social sciences compared to STEM fields. The authors should address this, also discuss the document proportion of each taxonomy, and potentially see if there is a correlation between the data size and performance.\n\nQUESTIONS:\n**Did you perform an ablation to verify this hypothesis?**: \"For instance, the subject calculus is both in physics and mathematics. We do not de-duplicate those subjects, since it may reflect their importance in human knowledge.\"\n\nPlease see Weaknesses.\n\nLIMITATIONS:\nPlease see Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Overall summary\n- This paper introduces GLAN, a method for enhancing LLMs by generating synthetic instruction data using a taxonomy of human knowledge and capabilities. GLAN constructs this taxonomy by decomposing knowledge into fields and disciplines, leveraging LLMs for generating a comprehensive syllabus for each subject. \n- GLAN’s scalable and customizable framework allows for easy integration of new fields of skills, highlighting its potential for ongoing improvement and adaptation.\n\n## My opinion of the paper\n- I think this is a really interesting approach to generate data that can allow LLMs to be potentially smarter. However, I am wondering if there are newer topics, for example (within the medical area, we have the new topic called \"Covid-19\".) Since GLAN is very dependent on LLMs, the main area of concern would be ensuring that the LLMs that GLAN depends on remains updated.\n\nSTRENGTHS:\n## Originality\n- The approach is quite interesting. The authors made use of real life scenarios, which is to use the structure of human education systems to build the taxonomy. This approach mimics the systematic acquisition of knowledge and skills in education, providing a framework for generating instruction data.\n## Clarity\n- Pseudo Algorithm provided and figures are easy to understand.\n## Significance\n- By creating a general and scalable method for instruction tuning, GLAN has the potential to improve the performance of LLMs across a wide range of tasks and domains.\n\nWEAKNESSES:\n## Quality\n- While the paper claims scalability, there is limited discussion on the computational resources required for generating the synthetic data at scale. Practical constraints related to computational costs and time could be a potential weakness. It was mentioned in the checklist that it is very computationally expensive to repeat experiments.\n\nQUESTIONS:\n- Line 269: Why do you say that the reason why errors coming from humanities and social science questions is due to CoT? Could it be due to the lack of knowledge, because it was not trained on that knowledge?\n- Figure 2: Any reason why for HumanEval and BBH datasets, the scores dropped even though the GLAN data size increase?\n\nLIMITATIONS:\nIndicated in the appendix (do consider placing it in main paper), but did not mention about computation cost like what was mentioned in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-31"}