Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:52:38

Abstract: We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). 
Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, 
GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale 
synthetic instruction data across all disciplines.
Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. 
Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs.
With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. 
Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have enabled unprecedented capabilities to understand and generate text like humans. By scaling up model size and data size [17, 13] , LLMs are better at predicting next tokens and prompting to perform certain tasks with a few demonstrations [2] . However, these capabilities do not directly translate to better human instruction following [25] . Instruction tuning [34] bridges this gap by fine-tuning LLMs on instructions paired with human-preferred responses.Prior work constructs instruction tuning data from seed examples or existing datasets. Initially, natural language processing (NLP) datasets described via instructions are used to fine-tune LLMs and the resulting LLMs can generalize on unseen (NLP) tasks [34] . However, there are only thousands of NLP tasks [33, 19] available, which limits the tuned LLMs to generalize in real-world scenarios [39] .Self-instruct [32] is a cost-effective method for creating synthetic instruction tuning datasets, which starts from a small pool of human-written seed instructions and generates new instructions by fewshot prompting an LLM (e.g., text-davinci-002) with randomly selected instructions from the pool. Unfortunately, the diversity of generated instructions is still an issue, since few-shot prompting tends to generate new instructions similar to its demonstrations. In addition, the process of creating high-quality seed instructions requires considerable human effort and expertise. Evolve-Instruct [39] improves self-instruct by augmenting existing instruction tuning datasets with different rewriting operations using LLMs, which is essentially data argumentation. Consequently, the scope of domains Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Figure 1 : Comparing GLAN with FLAN, Self-Instruct and Evolve-Instruct. The inputs of FLAN, Self-Instrct and Eovlve-Instruct are either seed examples or existing datasets, which limits the scope of domains of instructions that these methods can generate. GLAN takes the taxonomy of human knowledge & capabilities as input to ensure the broad coverage of generated instructions in various domains. This taxonomy is then broken down into smaller pieces and recombined to generate diverse instruction data.or tasks that these augmented datasets can cover is limited by the original input datasets. See Figure 1 for illustrations of these methods described above. There are also studies concentrated on developing instruction-tuning datasets tailored to particular domains or tasks. For instance, [20] creates datasets targeting mathematical reasoning. In contrast, [3] and [21] focus on coding-related tasks. All of the above methods cannot produce instruction datasets that are generally applicable to a wide range of domains.How to create a general instruction tuning dataset? We draw inspiration from the systematic structure in human education system. The structure of human education includes several levels, starting from early childhood education up to higher education and beyond [37] . Within each level, a student acquires knowledge, skills, and values in a systematic process. The courses a student learns from primary school to college cover a broad range of knowledge and skills, which facilitates the development of a diverse array of abilities. We believe that the systemic framework of the human education system has the potential to help the generation of high-quality and general instruction data, which spans a diverse range of disciplinary areas.In this paper, we introduce a generalized instruction tuning paradigm GLAN (shorthand for Generalized Instruction-Tuning for Large LANguage Models) to generate synthetic instruction tuning data almost from scratch. Unlike existing work [39, 21, 20, 24] , GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale instruction data systematically and automatically across all disciplines. Specifically, inspired by the structure of the human education system, the input taxonomy is constructed by decomposing human knowledge and capabilities to various fields, sub-fields, and, ultimately, distinct disciplines semi-automatically, facilitated by LLMs and human verification. The cost of human verification process is low due to the limited number of disciplines in the taxonomy. As shown in Figure 1 , we then further break down these disciplines into even smaller units. We continue to generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we can first sample from them and then generate diverse instructions with broad coverage across the entire spectrum of human knowledge and skills. The process described above mirrors the human educational system, where educators in each discipline craft a series of subjects for student learning. Instructors then develop a syllabus for each subject, breaking down the content into specific class sessions. These sessions are then further divided into core concepts that students must comprehend and internalize. Based on these detailed core concepts outlined in the syllabus, teaching materials and exercises are subsequently created, which are our instruction tuning data.GLAN is general, scalable and customizable. GLAN is a general method, which is task-agnostic and is capable of covering a wide range of domains. GLAN is scalable. Similar to [32, 39] , GLAN generates instructions using LLMs, which can produce instructions on a massive scale. Moreover, the input of GLAN is a taxonomy, which is generated by prompting an LLM and human verification, requiring minimal human effort. GLAN allows for easy customization. New fields or skills can be added by simply incorporating a new node into our taxonomy. Note that each node of the taxonomy can be expanded independently, which means that we only need to apply our method to the newly added nodes without re-generating the entire dataset. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, and logical reasoning to general instruction following without using task-specific training data of these tasks. 
glan: generalized instruction-tuned language models: 
 GLAN aims to create synthetic instruction data covering various domains of human knowledge and capabilities on a large scale. As shown in Algorithm 1, we first build a taxonomy of human knowledge and capabilities using frontier LLMs (i.e., GPT-4) and human verification. The taxonomy naturally breaks down human knowledge and capabilities to fields, sub-fields, and ultimately different disciplines (see Section 2.1). The following steps are fully autonomously facilitated by GPT-4 (or GPT-3.5). Then for each discipline, we again instruct GPT-4 to further decompose it into a list of subjects within this discipline (Section 2.2). Similar to an instructor, GPT-4 continues to design a syllabus for each subject, which inherently breaks a subject into various class sessions with key concepts students need to master (Section 2.3). With obtained class sessions and key concepts, we are ready to construct synthetic instructions. We prompt GPT-4 to generate homework questions based on randomly sampled class sessions and key concepts as well as the syllabus (Section 2.4).We recursively decompose human knowledge and capabilities into smaller units until atomic-level components (i.e., class sessions and key concepts). We expect to randomly combine these class sessions and key concepts to ensure the coverage and diversity of synthetic instructions. L ← L ∪ Q end for end for return LWe build a taxonomy of human knowledge and capabilities to guide the generation of synthetic instructions. Therefore, its coverage is important. On the other hand, it is also essential to make the taxonomy highly extensible, since the preferred capabilities of LLMs may change over time.In the first step, we propose to generate the taxonomy by prompting GPT-4 with a set of different instructions (e.g., list all fields of human knowledge and capabilities). Then, we do human post-editing to ensure its correctness and completeness. Due to the limited number of fields, sub-fields, and disciplines in our taxonomy, the cost of human verification is reasonably low. Another advantage of human post-editing is that we can easily add new fields or disciplines to the taxonomy as needed.Our taxonomy currently covers a diverse range of knowledge and capabilities in both academic education and vocational training. The top level of the taxonomy contains fields such as Natural Sciences, Humanities, or Services (vocational training). These fields branch out to various sub-fields and/or disciplines such as Chemistry, Sociology or Retailing. We keep breaking down nodes of the taxonomy until disciplines, and we leave the breaking down of disciplines to automatic methods described in the following sections. By collecting the leaf nodes of the taxonomy, we obtain a list of disciplines D = {d 1 , d 2 , . . . , d M }.As in Algorithm 1, for each discipline d, we aim to extract the list of subjects in it through prompt engineering. Specifically, we instruct GPT-4 to act as an education expert of discipline d and design a list of subjects a student should learn. The completion of GPT-4 contains a comprehensive list of subjects and their meta data (e.g., level, introduction and subtopics of the subject) in unstructured text format, which can not be directly used in subsequent steps. We therefore used another round of prompting to convert the completion to JSONL format:Awesome! Transform the above to JSONL format so that it is easier for a computer to understand. Enclose the JSONL output between two sets of triple backticks. For each JSONL object, use the keys "subject_name", "level" and "subtopics".It is worth noting that generating a subject list in JSONL format using a single prompt is feasible. However, we refrain to do so, because we observe that incorporating additional formatting instructions directly into the prompt can compromise the quality of the resulting subject list. These extracted subjects (as well as their meta data) S = {s 1 , s 2 , . . . , s N } can be subsequently used in next steps.For each s ∈ S, let s.name, s.level and s.subtopics denote the name, grade level and subtopics of subject s, respectively. We can apply the above prompts multiple times to ensure better coverage of subjects within this discipline.For each subject s, we have already extracted its name (s.name), grade level (s.level), and a small set of included sub-topics (s.subtopics) in a structured format. In this section, we aim to further segment each subject into smaller units, making them more suitable for creating homework assignments. We consult GPT-4 to design a syllabus for this subject. We opt for syllabus generation for the following reasons. Firstly, a syllabus essentially breaks down the main topic of a subject into smaller segments in a hierarchical manner. Specifically, each subject comprises several class sessions, and each session covers a variety of sub-topics and key concepts. Secondly, a syllabus provides an introduction, objectives, and expected outcomes of a subject, which are inherently useful for formulating homework questions. We instruct GPT-4 to 1) design a syllabus based on its meta data (s.level, s.name and s.subtopics); 2) break the subject into different class sessions; 3) provide details for each class session with a description and detailed key concepts students need to master.Let A denote the generated syllabus. The resulting syllabus A is in unstructured text format. However, class session names and key concepts of each class are required in the instruction generation step (see Algorithm 1) . Similar to the process of subject list extraction in Section 2.2, we again extract these meta data of each class session by prompting GPT-4. As a result, we obtain a list of class sessions C = {c 1 , c 2 , . . . , c |C| } and their corresponding key concepts K = {k 1 , k 2 , . . . , k |C| }. The detailed prompt for syllabus generation is in Appendix A.3.Given a syllabus A as well as a list of its class sessions C and their associated key concepts K, we are ready to generate homework questions and their answers. To generate diverse homework questions, we first sample one or two class session names from C and one to five key concepts under these selected class sessions. Let Ĉ denote the selected class session names and K the selected key concepts. Then we prompt GPT-4 (or GPT-3.5) to generate a homework question given the selected class sessions Ĉ and key concepts K as well as the syllabus A. We intend to give GPT-4/3.5 more context (e.g., what students have already learned in previous sessions) when creating assignments. Therefore, we additionally instruct GPT to consider that students have learned up to class sessions Ĉ when crafting homework and try to leverage multiple key concepts across different class sessions.See details of our prompt for instruction generation in Appendix A.4.Sampling Class Sessions and Key Concepts In a single syllabus, there are numerous class sessions and key concepts. We have two strategies to sample from them. In the first strategy, we generate assignments from a single class session. Therefore, we have only one class session name. Suppose we have m key concepts in total in this session. We randomly sample one to five key concepts from the m key concepts, which means we have totally 5 i=1 m i combinations. In this strategy, we focus on creating basic homework questions. To make the resulting questions more challenging (combine knowledge from multiple class sessions), we propose a second strategy to combine key concepts from two class sessions in the second strategy. We intend to generate questions leverage knowledge from two different class sessions. Suppose we have m 1 and m 2 key concepts in the first and second class sessions, respectively. We can have5 i=2 m1+m2 i - 5 i=2 m1 i - 5 i=2 m2 idifferent combinations, which is significantly more than that of the first strategy. We use both strategies to ensure our created questions are diverse in difficulty levels.Answer Generation After we generate questions in previous steps, we simply send these questions to GPT-3.5 and collect answers. We use GPT-3.5 for answer generation, because we find the quality of generated answers from GPT-3.5 is sufficiently good and using GPT-3.5 is significantly faster than GPT-4. The resulting question-answer pairs are our instruction tuning data. With a huge amountof question-answer pairs ranging from different disciplines with various difficulty levels, we expect the resulting LLM can excel in a wide range of tasks.3 Experiments 
data generation: 
 Taxonomy Creation By asking GPT-4 to create a taxonomy of human knowledge and capabilities, we end up with a set of fields, sub-fields, and disciplines that cover a broad range of domains in human knowledge and capabilities. Next, we ask human annotators to decide whether these elements in the taxonomy should be kept or not in order to reduce the redundancy of the taxonomy while maintaining its correctness. Note that if a field or sub-field is marked as remove, we remove its descendant as well. We kept 126 disciplines after majority voting (provided in supplementary materials). Note that it is feasible to manually add extra disciplines, sub-fields, or fields whenever necessary.During the subject list and syllabus generation, we prompt GPT-4 and employ nucleus sampling [14] with temperature T = 1.0 and top-p = 0.95 to encourage diversity. We do not use GPT-3.5-turbo since some subjects belong to the long-tail distribution which may not be effectively modeled by GPT-3.5-turbo. To ensure diversity and completeness of the generated subjects, we query GPT-4 10 times for each discipline (Section 2.2). There are 100 to 200 subjects for each discipline on average. It is worth noting that the same subjects may appear in different disciplines. For instance, the subject calculus is both in physics and mathematics. We do not de-duplicate those subjects, since it may reflect their importance in human knowledge. Given a subject in a specified discipline, we query GPT-4 for only one time to design a syllabus (see details in section 2.3). The temperature and top-p are still set to 1.0 and 0.95, respectively. The number of class sessions contained in each syllabus varies from 10 to 30 and each class session contains around five key concepts.Instruction Generation Each instruction data consists of a question and its answer. We choose to generate questions and answers separately since we observed that separate generations lead to better quality. After question generation with GPT-4, each question is then answered by GPT-3.5-turbo with temperature T = 0.7, top-p = 0.95 (we use a lower temperature in order to make the resulting answers more accurate). We use GPT-3.5-turbo instead of GPT-4 for answer generation, because GPT-3.5-turbo is significantly faster with reasonably good results. We generate 10 million instruction-response pairs in total and then we do training data decontamination. Specifically, the training instruction-response pairs are decontaminated by removing pairs that contain questions or input prompts from the test and training (if any) sets of benchmarks we evaluate. We exclude the training set of benchmarks we evaluate to verify the generalization capability of our synthetic data.We employ Mistral 7B [16] as our base model. During training, we concatenate each instruction and response pair to a single sequence and only compute loss on response tokens. We train our model for 3 epochs with a learning rate of 3e-6. The batch size is set to approximately 512 instruction-response pairs. We employ a dynamic batch size to ensure a constant total number of tokens per batch. We use a cosine learning rate schedule and we start with a linear warm-up of 1000 steps and the final learning rate is reduced to 0. The training requires approximately 8 days using 32 A100 GPUs.The instruction data GLAN generated spans a wide range of subjects. We evaluate its effectiveness in mathematical reasoning, coding, logical reasoning, and academic exams.Mathematical Reasoning: Mathematics is a common subject in many different disciplines. Hence, it is necessary to test the math reasoning ability of GLAN. We choose the two popular benchmarks for evaluation (i.e., GSM8K [7] and MATH [12] ). GSM8K [7] [20] , we report 0-shot setting results for GLAN. Coding: To evaluate the coding capability of GLAN, we opt for two coding benchmarks HumanEval [4] and MBPP [1] . We employ 0-shot setting for HumanEval and 3-shot setting for MBPP following prior art [4, 21] . BBH: The instruction dataset we generated covers many disciplines, which can potentially enhance the reasoning ability of GLAN. Therefore, we evaluate GLAN on the BIG-Bench Hard dataset (BBH [29] ), which contains 23 challenging tasks from Big-Bench [28] . We employ the standard 3-shot setting with chain-of-thought demonstrations. Academic Exams: We also evaluate GLAN on different academic benchmarks to verify whether GLAN is capable of solving exam questions. We choose two benchmarks (i.e., ARC [6] and MMLU [11] ). Both benchmarks are composed of multi-choice questions. AI2 Reasoning Challenge (ARC [6] ) contains grade-school level, multi-choice science questions. It contains two sub-sets, which are ARC-Challenge (ARC-C) and ARC-Easy (ARC-E). Massive Multitask Language Understanding (MMLU [11] ) consists of a set of multiple-choice questions about 57 subjects ranging in difficulty from elementary levels to professional levels. It covers various of domains of knowledge, including humanities, STEM and social sciences. Note that there is a training set for ARC. However, we have excluded it from our The scaling curve of GLAN on downstream tasks. The x-axis denotes GLAN data size (in log 10 scale following [17] ), and the y-axis denotes the task performance.training set during the decontamination process described in Section 3.1. Previous models mostly leverage probability-based methods on ARC and MMLU, which returns the best option based on the probabilities of the four options conditioned on the corresponding multi-choice question. We observe that after training on 10 million instructions, GLAN is able to generate its predicted options and analysis of multi-choice questions in plain text as GPT-3.5 does. We therefore opt for 0-shot setting for GLAN and extract predictions using rules based on its completions as in [22] .Our main results are shown in Table 1 . We compare GLAN against general domain models (Orca 2 [22] , Mistral Instruct [16] and WizardLM [39] ), math optimized models (MetaMath [40] and WizardMath [20] ) and coding optimized models (CodeAlpaca [3] ). We also report results of base LLMs (i.e., LLaMA2 [31] and Mistral [16] ) as references. GLAN either obtains the best resultsor results close to the best across all benchmarks. We observe that capabilities of math or coding optimized models increase on math or coding benchmarks while usually not others. After instruction tuning, GLAN excels on multiple dimensions from mathematical reasoning, coding, reasoning, and academic exams with a systematical data generation approach. Also note that our method does not use any task-specific training data such as training sets of GSM8K, MATH, or ARC as in Orca 2, MetaMath, and WizardMath, which indicates the general applicability of GLAN.A Closer Look at Academic Exams ARC and MMLU are all multi-choice based benchmarks on academic exams. However, we observe that improvements of GLAN over Mistral on ARC are much larger than these on MMLU (see Table 1 ). By grouping the 57 subjects in MMLU into four categories (i.e., STEM, Humanities, Social Sciences, and Other (business, health, misc.)), we observe GLAN wildly improves on STEM in MMLU while not in other categories (Table 2 ). Also note that ARC is composed of high school science problems, which are also STEM questions. GLAN is good at STEM subjects may be because responses of our dataset are from GPT-3.5-turbo, which by default generates responses with Chain-of-Thoughts (CoT) reasoning. Indeed, we observe that GLAN generates solutions with CoT for multi-choice questions. CoT may help the multi-step reasoning in STEM multi-choice questions [35] , while humanities and social sciences questions involve more memorization and single-step reasoning, where CoT may introduce additional errors. 1 ). This might imply that synthetic data generated by GLAN is diverse and our resulting model avoids convergence to any specific domain or style present in existing benchmarks.IFEval We assess the instruction-following capabilities of GLAN utilizing the Instruction Following Evaluation dataset (IFEval [42] ). IFEval consists of a collection of "verifiable instructions", encompassing 25 distinct types of instructions (around 500 prompts in total). Each prompt comprises one or more verifiable instructions. The evaluation involves four types of metrics at both prompt level and instruction level, evaluating strict and loose accuracies. As shown in Table 4 , GLAN demonstrates superior instruction-following capabilities in both prompt-level and instruction-level evaluations. However, there is still a considerable gap compared to GPT-3.5-turbo and GPT-4. Evol-Instruct Test Evol-Instruct testset [39] contains real-world human instructions from diverse sources, and it consists of 218 instances with 29 distinct skills. Each instruction is associated with a difficulty level from 1 to 10. The responses are often open-ended descriptions, and we believe this benchmark is a necessary supplement to IFEval (answers to their instructions are "verifiable").Following [39] and [5] , we adopt a GPT-4-based automatic evaluation method to conduct a pairwise comparison between GLAN and other models. Specifically, GPT-4 is instructed to assign a score between 1 and 10 overall score w.r.t. the helpfulness, relevance, accuracy, and level of detail of responses generated by two different models for a given input question. A higher score indicates better overall performance. To mitigate potential order bias, we perform bidirectional comparisons for each response pair and determine their average score. The average score difference to GLAN (i.e., avg_score(GLAN) -avg_score(x)) serves as the final metric. Table 5 presents the results of pairwise comparisons across various levels of instruction difficulty. GLAN showcases superior performance compared to LLaMA-2, Orca 2, Mistral Instruct, and even WizardLM-13B (note that GLAN contains only 7B parameters) on most difficulty levels and overall scores. This suggests that GLAN demonstrates improved ability to process diverse instructions, regardless of their difficulty or complexity. Also, note that GLAN falls behind GPT-3.5-turbo as other models in comparison.Additionally, we group Evol-Instruct test according to the 29 skills and observe the same trends.Detailed results are listed in Appendix (Table 9 and 10 ). GLAN demonstrates strong performance on most skills, especially in Math, Coding, and Reasoning. However, it slightly falls short in commonsense related tasks. We also created GLAN-Test, similar to the Evol-Instruct Test but much larger in size, where GLAN outperforms other models as well (see Appendix A.8). 
related work: 
 Recent literature has extensively explored the collection of various human-made resources for instruction tuning. An intuitive direction is to collect existing NLP datasets and corresponding task descriptions [26, 33, 41] , typical LLMs such as BLOOMZ [23] and FLAN [34] are trained on this type of instruction tuning data. However, with only tens to thousands of existing datasets available, the scope and diversity of instruction tuning are inevitably limited. Another common practice is to implement instruction tuning with real-world human user prompts. For instance, InstructGPT [25] was trained on high-quality human prompts submitted by real-world users to OpenAI GPT APIs. Vicuna [5] leverages user-shared prompts along with ChatGPT responses for instruction tuning, and Dolly [8] was trained on simulated human-user interactions written by over 5k employees. Nevertheless, acquiring instructional data from human users typically involves high costs and involves privacy concerns. As LLM capabilities improve, instruction tuning with LLMgenerated data exhibits better scalability and potential in addressing the super-alignment problem [27] .Leveraging the in-context learning ability of LLMs, Unnatural instructions [15] and Self-instruct [32] sampled seed instructions as examples to elicit LLMs to generate new instructions. Taking advantage of the rephrasing ability of LLMs, WizardLM [39] and WizardMath [20] were trained using Evol-Instruct. Evol-Instruct iteratively employs ChatGPT to rewrite seed instructions into increasingly complex instructions. Similar to generation from seed instructions, carefully selected seed topics are used for generating textbook-like synthetic data [18] or self-chat multi-turn dialogues [38, 9] for instruction tuning. However, models trained on these LLM-generated data only work well in specific domains such as math [20, 40] , dialogue [38, 9] or open-ended question answering [30, 39] .These methods encounter challenges in generalization [10] , as the data diversity is restricted by seed instructions or seed topics. 
conclusions: 
 We propose GLAN, a general and scalable method for synthesizing instruction data. Experiments show that GLAN can help large language models improve their capabilities in multiple dimensions, from mathematical reasoning, coding, academic exams, and logical reasoning to general instruction following. Currently, our synthetic data are based on the taxonomy of human knowledge and capabilities, and there are other types of useful data that have not been covered. We are interested in designing methods with border coverage. Our current instruction data are mostly question-answer pairs, and in the next step, we plan to generate synthetic data of multi-turn conversations and long documents. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 3.2, 3.3Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We did not include error bars in the experiments due to the high computational demands.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We included compute resources in Section 3.2.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix A.2Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: To ensure future responsible release, we are still in the process of implementing comprehensive safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing assets used in this paper are properly credited. The license and terms of use are properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: Once authorization is obtained, we will ensure that comprehensive documentation is provided alongside the assets to facilitate their proper use and understanding.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
 
 Appendix: While GLAN presents significant advancements in academic benchmarks. However, there may still have several limitations in real world deployment. The resulting LLMs train on generated data using GLAN may occasionally produce factual incorrect (or even toxic) responses. Further training for refusal, hallucination reduction as well as toxic content reduction should be performed before deployment.Data synthesizing is crucial for the continual scaling of large language models, especially as we exhaust available human data. GLAN demonstrates the potential to generate vast amounts of synthetic data from scratch, paving the way for even larger-scale data synthesis efforts. While GLAN hasshown the effectiveness of synthetic data, we must point out that synthetic data may inherit and even amplify social biases present in the frontier LLMs for generation. Future research should focus on developing techniques to identify and correct biases in the generated datasets and models trained on them.The prompt template for syllabus generation is in Table 6 . The prompt template for instruction generator is in Table 7 .We provide the specific train/test values of different models on different benchmarks in Table 8 .The concrete Evol-Instruct test results on different difficulty levels are shown in Table 9 . The concrete Evol-Instruct test results on different skills are shown in Table 10 .GLAN-Test There are only hundreds of instructions in In IFEval and Evol-Instruct Test and we believe the domains or skills they can cover are rather limited. Therefore, we propose a heldout test set using GLAN data and we call it GLAN-Test. It contains 6,300 instructions on 126 disciplines (50 instructions for each discipline). We further categorize the 126 disciplines to 8 distinct fields (i.e., Academic-Humanities, Academic-Social Science, Academic-Natural Science, Academic-Applied Science, Academic-Formal Science, Industry-Manufacturing, Industry-Services and Industry-Agriculture). We believe that the extensive domain coverage of GLAN-Test renders it an effective test bed for the assessment of generalization capabilities in LLMs. We adopt the same GPT-4 based evaluation protocol as in Evol-Instruct Test (previous paragraph). We prompt GPT-4 to do a pairwise ranking of GLAN and other models in comparison. The overall results and results across the 8 fields are presented in Table 11 , where GLAN obtains higher GPT-4 scores than Orca2-7B, Mistral-7B Instruct and WizardLM-13B, despite using only 7B parameters. GLAN still lag behind GPT-4. Detailed results for the 126 fine-grained disciplines can be found in Appendix A.9 (see Table 12 for more details). GLAN demonstrates its effectiveness on multiple domains (or disciplines) such as Mathematics, Physics, Chemistry, Computer science, Electrical, Mechanical, etc., indicating that smaller models may yield general improvements on various domains through strategic fine-tuning. Furthermore, it is noted that GLAN demonstrates less-than-ideal performance across distinct disciplines such as American history, Divinity, or Radiology. This observation underscores the potential for further refinement and development of our methodology within these domains.Table 11 : Pairwise comparison between GLAN and other models on GLAN-Test (the 126 disciplines are categorized into 8 fields for clarity of the illustration). The scores are the average gap of scores assigned by GPT-4, calculated as avg_score(GLAN) -avg_score(x). Justification: While we are temporarily unable to provide open access to the data and code, we are actively working to gain the necessary authorization to release these resources. Once obtained, we will ensure that all data and code, along with detailed instructions, are made available to faithfully reproduce the main experimental results.Field (Ratio) Orca2-7B Mistral-7B-Instruct WizardLM-13B-V1.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: This study strictly adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a top-down approach to generate high-quality instruction tuning data, which covers taxonomy, subjects, syllabus and instructions. The generated taxonomy is verified by human to guarantee its quality. The performance is strong on many benchmarks such as reasoning, coding and general instruction following. Several reviewers raised concerns about the cost to generate the 10 Million instructions. And there is a mismatch between the motivation to use pre-curated taxonomy of human knowledge and the actual method that relies heavily on LLMs to establish the taxonomy.

Reviews: review: paper_summary: paper_summary: ## Overall summary
- This paper introduces GLAN, a method for enhancing LLMs by generating synthetic instruction data using a taxonomy of human knowledge and capabilities. GLAN constructs this taxonomy by decomposing knowledge into fields and disciplines, leveraging LLMs for generating a comprehensive syllabus for each subject. 
- GLAN’s scalable and customizable framework allows for easy integration of new fields of skills, highlighting its potential for ongoing improvement and adaptation.

## My opinion of the paper
- I think this is a really interesting approach to generate data that can allow LLMs to be potentially smarter. However, I am wondering if there are newer topics, for example (within the medical area, we have the new topic called "Covid-19".) Since GLAN is very dependent on LLMs, the main area of concern would be ensuring that the LLMs that GLAN depends on remains updated.
strength_weakness: strength_weakness: ## Quality
- While the paper claims scalability, there is limited discussion on the computational resources required for generating the synthetic data at scale. Practical constraints related to computational costs and time could be a potential weakness. It was mentioned in the checklist that it is very computationally expensive to repeat experiments.
questions: questions: - Line 269: Why do you say that the reason why errors coming from humanities and social science questions is due to CoT? Could it be due to the lack of knowledge, because it was not trained on that knowledge?
- Figure 2: Any reason why for HumanEval and BBH datasets, the scores dropped even though the GLAN data size increase?
limitations: limitations: Indicated in the appendix (do consider placing it in main paper), but did not mention about computation cost like what was mentioned in the checklist.
paper_summary: paper_summary: This paper proposes a generalized way of creating instruction data. The high-level motivation is to take inspiration from how curriculum is designed for human learning into a taxonomy of subjects and use the same to prompt an off-the-shelf LLM to create data. GLAN does not need seed examples, or pre built-taxonomy like prior work. Human verification is also performed post the building of taxonomy to weed out unimportant or inaccurate divisions. The overall process is High level taxonomy -> subjects -> syllabus -> instructions.
strength_weakness: strength_weakness: 1. No use of actual human curriculum: The paper set the expectation right in the abstract of using/getting strongly inspired from human curriculum. I was disappointed that the method does not utilize existing human curriculum structures, potentially missing out on years of insights in developing the same. Generating synthetic data, and in this case entire taxonomies from pre-existsing models can lead to extremely large amounts of bias. I would have much rather seen the authors delegate only lower level questions to LLMs than high level abstractions, which would lead to a trickle down effect on every single node in the taxonomy. This study, in my opinion, is incomplete without using either human generated taxonomies, and/or a comparison between how different the taxonomies are.
2. Computation cost not compared: The paper does not provide a comparison of computational costs with similar methods, such as WizardLM. For instance, GLAN training required approximately 8 days using 32 A100 GPUs to generate 10 million instructions, but no direct comparisons are made to illustrate the efficiency or cost-effectiveness relative to other approaches.
3. The method is limited by the performance of GPT-3.5/4: The quality of the generated taxonomy and syllabus heavily depends on the capabilities of the underlying LLMs used in the process, namely GPT-3.5 and GPT-4. In general, GLAN does not inform how we can improve capabilities of models beyond GPT4. But also, does not consider the cost of generating 10 million instructions.
4. High variability in results (Table 2): There is significant variability in GLAN's performance across different categories, with particularly weaker results in humanities and social sciences compared to STEM fields. The authors should address this, also discuss the document proportion of each taxonomy, and potentially see if there is a correlation between the data size and performance.
questions: questions: **Did you perform an ablation to verify this hypothesis?**: "For instance, the subject calculus is both in physics and mathematics. We do not de-duplicate those subjects, since it may reflect their importance in human knowledge."

Please see Weaknesses.
limitations: limitations: Please see Weaknesses
paper_summary: paper_summary: This paper introduces GLAN, a general and scalable method for instruction tuning of Large Language Models (LLMs). GLAN employs a top-down approach to generate high-quality instruction tuning datasets. Experiments across various benchmarks demonstrate that GLAN performs comparably to other existing methods.
strength_weakness: strength_weakness: The novelty is limited as similar top-down designs have been utilized in many previous works. Besides, the main experimental results in Table 1 appear mediocre compared to other methods.
questions: questions: Refer to the weakness.
limitations: limitations: Refer to the weakness.
paper_summary: paper_summary: The paper introduces generalized isntruction tuning (GLAN), an approach for synthesizing instruction tuning data using a taxonomy-based approach. GLAN generates synthetic instruction data from pre-curated taxonomy of human knowledge and capabilities and aims to create diverse and broad-ranging instruction dataset.
strength_weakness: strength_weakness: 1. While the paper addresses generalization, there is a risk that the generated synthetic data might overfit to the taxonomy's structure, potentially missing out on more nuanced, real-world instructions.
questions: questions: 1. Are there any measures in place to ensure the generated synthetic data's diversity and prevent redundancy?
2. what is the whole taxonomy of the human knowledge and capabilities? And can each task (e.g., gsm8k, arc) be categorised into any sub category?
3. Are the effects of each category orthogonal to each other? i.e., ablating data from a child category does not effect tasks in another child category. It would be beneficial if the authors could provide some preliminary results.
limitations: limitations: The authors have adequately addressed the limitations

Comments: title: Reply to Author's Comment
Reply to Authors' Rebuttal
comment: Thank you for your thoughtful and detailed feedback. We greatly appreciate the time and effort you have taken to engage with our work and provide valuable insights. 

> a more scientific analysis of "found GPT-4 is good at listing the taxonomy structure" and "our taxonomy overlaps significantly with existing human curriculum"

To quantify the overlap, we calculated the Overlap Coefficient (Szymkiewicz-Simpson coefficient; refer to Equation below) between our taxonomy and a "standard curriculum" (https://en.wikipedia.org/wiki/Outline_of_academic_disciplines)
$$O(A, B) = \frac{|A \cap B|}{\min(|A|, |B|)}$$
the Overlap Coefficient is 90 / 126 = 71.42%

This high overlap coefficient supports our claim that “our taxonomy overlaps significantly with existing human curriculum” and is also a strong indicator that “GPT-4 is good at listing the taxonomy structure of human curriculum”. We also find that recent studies [1][2] have demonstrated strong capabilities of GPT-4’s in generating taxonomies.

We have uploaded our taxonomy as supplementary material (please refer to line 186 in Section 3.1). Besides, if you are aware of a more appropriate standard curriculum for comparison, we would appreciate your suggestions.

> computation cost comparison.

Training cost (measured in FLOPs) for different methods are shown below
| **Methods**       | **FLOPs**         |
|-------------------|-------------------|
| Orca 2 [3]        | $$9.5 \times 10^{20}$$ |
| MetaMath [5]         | $$2.5 \times 10^{19}$$ |
| WizardLM v1.2 [6]    | $$\geq 3.2 \times 10^{19}$$ |
| WizardMath v1.1 [7]  |   –            |
| GLAN              | $$6.3 \times 10^{20}$$ |

According to [6], the training cost for WizardLM v1.0 is $3.2 \times 10^{19}$ FLOPs, while the training cost for WizardLM v1.2 remains unknown, though we expect that at least the same number of examples were used. The exact number of training examples for WizardMath is not disclosed in [7], and it also involves a PPO training stage with limited technical details provided. Please refer to Table 1 for their performance comparisons. Despite utilizing less computational resources, our results surpass those of Orca 2, and we consistently outperform MetaMath and WizardLM across all tasks presented in Table 1.

Also note that GLAN data aims to enhance a model's capabilities across a wide range of tasks (without relying on seed examples), whereas the data generated by previous methods such as MetaMath, WizardLM and WizardMath are focused on improving performance in specific tasks. Therefore, to achieve the same level of performance on a particular task, our method usually needs to generate more data and hence higher training cost.

> Overall, found the answer to "high variability" quite unconvincing, and a reluctance to scientifically examine the phenomenon that is underneath the variability (data density, etc.).

High variability  in humanities and social sciences performance is not due to data density but the inference strategy (CoT vs. non-CoT). 

We observed no correlation between the data size and performance. Specifically, there are 19 disciplines related to MMLU subjects and only 6 of them are STEM disciplines. We generated almost the same number of examples for each of these 19 disciplines. We have more non-STEM data than STEM data with the non-STEM data being 2.16 times greater.

We believe that the Chain-of-Thought (CoT) reasoning contributes to the lower performance on these questions. In our earlier experiments using the MMLU benchmark, we evaluated both CoT and non-CoT settings. While the overall performance was superior with CoT (leading to its adoption), we noticed that CoT was more effective for STEM questions, whereas non-CoT proved advantageous for humanities and social sciences questions. This might be because CoT aids in multi-step reasoning in STEM multiple-choice questions, while humanities and social sciences questions involve more memorization and single-step reasoning, where CoT may introduce additional errors (also see line 264 to 269).



All discussions and results will be added to our revised manuscript.

Thank you again for your valuable feedback.

## References

[1]. Gunn, M. et al. “Creating a Fine Grained Entity Type Taxonomy Using LLMs.” 2024.

[2]. Lee, M. et al. “Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistants.” 2024.

[3]. Mitra, A. et al. “Orca 2: Teaching Small Language Models How to Reason.” 2023.

[4]. Stanford Alpaca GitHub Repository

[5]. Yu, L. et al. “MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models.” ICLR 2024.

[6]. C. Xu et al. “WizardLM: Empowering Large Language Models to Follow Complex Instructions”. 2023

[7]. H. Luo et al. “WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct”. 2023
Thanks for taking time to reassess our work and for your updated score!
> Initially, we intend to automate the whole generation process and found GPT-4 is good at listing the taxonomy structure of human curriculum.

"found GPT-4 is good at listing the taxonomy" : this is based on what metric? was there a scientific study to test this?

> our taxonomy overlaps significantly with existing human curriculum.   


Is there a scientific analysis of this claim?

> computation cost comparison.   


I was looking for a training compute cost comparison since all baselines may have been trained for different durations, which can be a big confounder in the "perceived quality" of the data.

Overall, found the answer to "high variability" quite unconvincing, and a reluctance to scientifically examine the phenomenon that is underneath the variability (data density etc.).
Hi,
Thanks for your reply! Regarding the "similar top-down designs in previous works," I was referring to WizardLM and Ultrachat. However, I noticed that your approach distinguishes itself by incorporating a pre-curated taxonomy, which frees it from the constraints of existing data. In light of this, I will be adjusting my score to 5.
Dear Reviewer koAL,

Since we are approaching the deadline of the discussion period, we were wondering if you have had the chance to review our response to your comments. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have.

Thank you for your time and consideration.


Sincerely, 

All Authors
Dear Reviewer 7Qp1,

Since we are approaching the deadline of the discussion period, we were wondering if you have had the chance to review our response to your comments. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have.

Thank you for your time and consideration.


Sincerely, 

All Authors
Thank you again for your thorough review and for taking the time to carefully read our responses. We are glad that most of your concerns have been addressed, and we appreciate your continued support for our work.
Thanks for the authors responses and most of my concerns are properly addressed. Thus I would like to remain the score as 7 (Accept).
Thank you very much for taking the time to consider our clarifications and for your updated evaluation. We greatly appreciate your thoughtful feedback throughout the review process, which has helped to strengthen our work.
Thank you for your clarifications! I have raised the score rating from 5 to 6.
Thank you for your insightful question and for bringing up this important point.

Regarding new topics that emerged after October 2023, it is still possible to handle these scenarios with some modifications to the underlying LLM or prompts as described in Sections 2.2 and 2.3. Here are three methods to ensure our approach remains up-to-date. In the following, we refer to the prompts in Sections 2.2 and 2.3 as "GLAN prompts":

- **Using Retrieval-Augmented Generation (RAG)**: You can leverage an existing LLM integrated with web search capabilities (e.g., Perplexity AI API https://docs.perplexity.ai/reference/post_chat_completions) or implement RAG using GPT-4 function calls combined with the Google Search API. In this approach, the GLAN prompts remain unchanged. The RAG model first retrieves documents relevant to the new topic and then integrates these documents with the GLAN prompts. This method is very likely to be effective because new topics are typically related to pre-existing knowledge within the LLM. With the retrieved documents, the LLM is still likely to generate a reasonable subject list or syllabus (i.e., breaking down the new topic given docs introducing it).

- **Prepending Documents of the New Topic to Our Prompts**: Conduct a search for the new topic using a search API (e.g., Google Search API) and prepend the top returned documents and "Based on the context above, your task is as follows." to the GLAN prompts. It may also be necessary to prepend these documents to the instruction generation prompt in Section 2.4 to ensure that the LLM comprehends any new terms. This method is similar to the RAG approach but provides more transparency to API users regarding the integration of search results with prompts.

- **Adopting Newer LLMs**: A more straightforward solution is to wait for the release of newer models (in three months or less) by leading LLM companies (e.g., OpenAI, Anthropic, Meta, Mistral) which will likely include training on documents covering the new topic. Once these models are available, they can be adopted.

We appreciate your question as it tends to broaden the scope of our method. Thank you again for your valuable feedback!
Thank you for your clarifications!

Regarding the first point, I apologize for the confusion. Covid-19 is indeed already known to GPT-4 as its training data extends until October 2023. I was referring to scenarios involving new topics that emerged after October 2023, though I can't think of specific examples at the moment. Would such scenarios be handled as outlined in Sections 2.2 and 2.3 as you have explained here?