Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for
  Language Models
Authors: name: Haoran Li; name: Qingxiu Dong; name: Zhengyang Tang; name: Chaojun Wang; name: Xingxing Zhang; name: Haoyang Huang; name: Shaohan Huang; name: Xiaolong Huang; name: Zeqiang Huang; name: Dongdong Zhang; name: Yuxian Gu; name: Xin Cheng; name: Xun Wang; name: Si-Qing Chen; name: Li Dong; name: Wei Lu; name: Zhifang Sui; name: Benyou Wang; name: Wai Lam; name: Furu Wei
Published: 2024-02-20 15:00:35

Abstract: We introduce Generalized Instruction Tuning (called GLAN), a general and
scalable method for instruction tuning of Large Language Models (LLMs). Unlike
prior work that relies on seed examples or existing datasets to construct
instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
human knowledge and capabilities as input and generates large-scale synthetic
instruction data across all disciplines. Specifically, inspired by the
systematic structure in human education system, we build the taxonomy by
decomposing human knowledge and capabilities to various fields, sub-fields and
ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
Subsequently, we generate a comprehensive list of subjects for every discipline
and proceed to design a syllabus tailored to each subject, again utilizing
LLMs. With the fine-grained key concepts detailed in every class session of the
syllabus, we are able to generate diverse instructions with a broad coverage
across the entire spectrum of human knowledge and skills. Extensive experiments
on large language models (e.g., Mistral) demonstrate that GLAN excels in
multiple dimensions from mathematical reasoning, coding, academic exams,
logical reasoning to general instruction following without using task-specific
training data of these tasks. In addition, GLAN allows for easy customization
and new fields or skills can be added by simply incorporating a new node into
our taxonomy.