Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation
Authors: Da Yin
Xiao Liu
Fan Yin
Ming Zhong
Hritik Bansal
Jiawei Han
Kai-Wei Chang
Venue: EMNLP 2023 Main (2023)
Keywords: instruction tuning
dynamic
generalizable

Abstract: Instruction tuning has emerged to enhance the capabilities of large language models (LLMs) to comprehend instructions and generate appropriate responses. Existing methods either manually annotate or employ LLM (e.g., GPT-series) to generate data for instruction tuning. However, they often overlook associating instructions with existing annotated datasets. In this paper, we propose Dynosaur, a dynamic growth paradigm for the automatic curation of instruction-tuning data. Based on the metadata of existing datasets, we use LLMs to automatically construct instruction-tuning data by identifying relevant data fields and generating appropriate instructions. 

By leveraging the existing annotated datasets, Dynosaur offers several advantages: 1) it reduces the API cost for generating instructions (e.g., it costs less than \$12 USD by calling GPT-3.5-turbo for generating 800K instruction tuning samples; 2) it provides high-quality data for instruction tuning (e.g., it performs better than Alpaca and Flan on Super-NI and Longform with comparable data sizes); and 3) it supports the continuous improvement of models by generating instruction-tuning data when a new annotated dataset becomes available. We further investigate a continual learning scheme for learning with the ever-growing instruction-tuning dataset, and demonstrate that replaying tasks with diverse instruction embeddings not only helps mitigate forgetting issues but generalizes to unseen tasks better. 

Code and data are available at https://github.com/WadeYin9712/Dynosaur.

DYNOSAUR: A Dynamic Growth Paradigm for Iestruction-Tuningg Data Curation
Da Yin
Xiao Liu'* Fan Yin Ming Zhong"1
Hritik Bansal Jiawei Han Kai-Wei Chang
SUCLA Peking University UICC
yin, fanyin20, hbansal, kwchang)@cs ucla. edu 1xlisa@pku. (mingz5, hanjj@illinois. edu dynosaur-it.gi wit.gb.thtboooo
Abstract
output according to input instructions. Exist- ing approaches compile instruction-tumie datasets mainly by 1) manual annotations or 2) distillate from a larger size of LLM. For example, SUPER- NATURALINSTRUCTIO (SUPER-ND) (Wang et al.. 2022b) and DOLLY (Databricks, 2023) recruit ex- perts to manually annotate task instructions and related task data. Despite their high quality, this approach is labor-intensive and costly (Honovich et al., 2022a). Recent efforts (Wang et al., 2022a; Taori et al., 2023) leverage GPT-series to distill in- struction tuning data to train smaller models. How- ever, subsequent studies (Gudibande et al., 2023) ar- gue that these methods merely help smaller models learn to mimic the style of teacher L.Mss without inheriting their true capabilities, such factuality and problem solving skills. We suspect it is mainly due the instructions not ground actual data.
Instruction tuning has emerged to enhance the capabilities of large language models (LLMs) to comprehend instructions and generate ap- propriate responses. Existing methods either manually annotate employ LLM (e.g., GPT- series) to generate data for instruction tuning. However, they often overlook associating in- structions with existing annotated datasets. In this paper, we propose DYNOSAUR, dynamic growth paradigm for the automatic curation of instruction tuning data. Based on the meta- data existing datasets, we use I.LMs to auto- matically construct Instruction-tuninn data by identifying relevant data fields and generating appropriate instructions.
By leveraging the existing annotated datasets, DYNOSAUR offers several advantages: 1) it re- duces the API cost for generating instructions (e-g., costs less than $12 USD by calling GPT-3.5-turla for generating 800K instruc- tion tuning samples; 2) it provides high-quality data for instruction tuming (c.g., performs bet- ter than ALPACA and FLAN SUPER-NI and LONGFORM with comparable data sizes); and supports the continuous improvement of models by generating cnttucttiontuning data when new annotated dataset hecomes avail- able. We further investigate continual leam- ing scheme for learning with the ever- wer-growwin instruction-tune ttuning dataset, and demonstrate that replaying tasks with diverse instruction embed- dings not only helps mitigate forgetting issues but generalizes unseen tasks better.
In this paper, we propose DYNOSAUR, dy- namic growth paradigm to convert high quality an- notations from dataset repositories into instruction- tuning data. In particular, DYNOSAUR gener- ates instructions based on the metadata cxist- ing datasets in the dynamically growing Hugging- face Datasets Platform (Lhoest et al., 2021). As shown in Figure 1, metadata covers essential infor- mation about a dataset, including dataset descrip- tion ("A collection of ebooks ..."), dataset name ("Gutenburg_Engll1
data fields ("title". "text", "issued") and dataset an- notations. Guided by metadata, our method can generate multiple tasks applicable for forming instruction-tuning data with instances in NLP datasets. We leverage LLMs to harvest task instruc- tions and their corresponding input/output fields with single prompt. Prompted with dataset de- scription involving ebooks and data fields about the book published information, LLMs can syn- thesize instructions such as "Given Gutenburg passage, generate its title" and "Predict the year when the book is published based
Code and data are available https:// github. http.////w..iiii//////yeeeeee
1 Introduction
Instruction tuning (Sanh et al., 2022; Ouyang et al., 2022: Wei et al., 2022) enables large language models (L.LMs) (Raffel et al., 2020; Brown et al., 2020; Touvron et al., 2023) to provide appropri-
"These four authors contrihuted equally.

---

Dynosaur Construction Method
Step Instruction, Input/Outputt Field Generation
Stup Organiring Instruction Date
Pronejt for Geneontnng bettortion
Takk
Stop 1: Matadata Collaction
Datasets
desige co choes diffecanc caste based on this aterid
"Lsenecesioe"s "GLveo Gacenbsin passige, gecarate ksnta
inolieding the seatisatton, Fieldo ouspes tleld.
"Lopucts -[dl-EaKE")
reuegueh "(d)" sdjiee 11
text
author
5. subjects
Issund
CUOTT
Cuterberg
Taake AchaSDer SuGuAtal.
linere ons' 'Bivenn Ostenbarg passege,
gmeeeate the aix3e".
pongee gesstl
Ingruction
"ostgot 1ield L'855e'11
"supue"""
Outa
"Laacrattion"
Pianas bel2 the wathor
Ehie Seadl.
Prstact metadara
2ia2d
pabllsbed book cicle and authace",
Lieece 1Bredtic year when the look publicbed baced on 2o0k catle authose""
"aapurr: "Taat to (cl "east": Buttor de "exthce" "oxtpuc": *Jol" "dmmzed"1
Dutoes Dewcrtion
Siterber_inggn
"ostpst_tisll tinld": ""Saaaed") 13).
Fill Cutemberg data
Acil octonicl of.. scodla....
Detend Amattrisa
"mot"" "wator", "o "aajea" 'krued
Step 3: Filtering out Invelid Instruction Data
"remector" ". "hedd the year whenwhhen the booc pub itha barge
M
Diplicato
"ipputi Author "cutput"
Dash
Overlagped trpps/Cotput Fwiiss
Figure I: Overall pipeline of collecting DYNOSAUR data, "d" in Step 4 means each instance in Gutenberg dataset.
on book title and authors". These instructions reflect the original data domain and use multiple dataset components.
Effectiveness of Instruction Data. We evaluate the data effectiveness by studying whether models trained with DYNOSAUR can achieve competitive performance on SUPER-NI, LONGFORM (Köksal et al., 2023) and USER- INSTRUCTION-255 252 (Wang et al., 2022a). On SUPER-NI, -I, both T5-3B and LLAMA-7B mod- els fine-tuned with DYNOSAUR outperform AL- PACA, INSTRUCTION GPT-4 and DOLLY that are much more expensive be collected. In particu- lar, training T5-3B with DYNOSAUR brings 2.5-22 ROUGE-L improvement than other datasets. On FONGFORM, training T5-3B with DYNOSAUR is 2.8-12. METEOR better than training with other human-curated instruction data such as PROMPT- SOURCE (Sanh et al., 2022) and FLAN (Wei et al., 2022). On INSR-INSTT can be exploited as additional training data to achieve higher performance than solely training with either ALPACA or INSTRUCTION GPT-4
In the meantime, LLMs also determine which data fields should be used to construct correspond- ing task inputs/outputs according to generated in- structions. As illustrated in Figure 1, LLMs capture corresponding input fields "title" and "author" " and output field "issued" for the generated task about predicting issued years given book title and authors. Subsequently, all the data under "title" and "author" fields are used as the final inputs of the generated task, and the data under "i "issued" are treated final outputs. Suppose that we gen- erate N instructions based on the metadata of a dataset which contains M instances, our method can synthesize N x M instruction-tuning data. DYNOSAUR offers several advantages:
DYNOSAUR
Low Conversion Cost. As DYNOSAUR lever- ages existing annotated data, it reduces the number of queries to larger LLMs for generating instruc- tions. For example, it costs only $11.5 USD to query GPT-3.5-turbo (OpenAI, 2023) and gen- erate 800K instruction-tuning data based on an- notated datasets. In contrast, both ALPACA and INSTRUCTION GPT-4 cost around $500 USD to generate a significantly smaller dataset of 52K in- stances. Despite the lower cost of querying L.Mss, DYNOSAUR generates high-quality data by effec- tively leveraging existing annotations.
Supporting Continuously Improving Models with New Instruction Data. Statistics show that an average of 143.6 datasets are added to Hugging- face Datasets daily in 2023. Because of the low conversion cost, DYNOSAUR can grow dynamically as the platform expands without much effort.
An ever-growing instruction-tuning dataset pro- vides an opportunity to continuously improve instruction-folloiinng models. Suppose we have a model trained with K tasks (Mx) and newly ob- tain training tasks. How can we train Mr k with

---

the L new tasks to achieve better generalization on unseen tasks and the new L tasks and suffer less from forgetting the previous K training tasks? We propose several continual learning strategies specifically for instruction tuning which select re- play tasks based on the diversity of instruction and data representations. Experiments with SUPER-NI and DYNOSAUR show that replaying is effective improve generalization and mitigate forgetting. Besides, once new tasks are used for training, replaying previous tasks with the least similar in- structions to the tasks performs the best.
lect the metadata from datasets whose licenses al- low adaptation. More details are Appendix A.
2.2 Instruction and Input/Output Field Generation
For each dataset accompanied by processed meta- data, we then deploy L.M to generate multiple tasks associated with it. For each task, LLM gen- erates a specific task instruction and designates its input/output fields simultancously. As exemplified in Figure 1, LLM is expected to generate an instruc- tion "Given Gutenburg passage, generate its title", its input field "text", and the output field "title".
2 Collection of DYNOSAUR Data
To accomplish this, we harness the power of in- context learning (Brown et al., 2020). Concretely, we wrap the information of each dataset into a dictionary format and construct four demonstra- tions manually. Due to the length limitation of the LLM, we use two of them each time as part of the input. Depending whether not the incorpo- rating dataset descriptions in the input prompt, we consider the following two configurations:
In this section, we introduce how to construct the DYNOSAUR dataset. As shown in Figure 1, we first collect metadata from existing datasets, then prompt LLM to create tasks based on the metadata, and filter out invalid ones.
2.1 Metadata Collection
Metadata contains key information about an NLP dataset that contributes to instruction-tunine data generation. It covers the following elements:
Description-Aware Generation. To maximize the utilization of information present in the dataset description, we incorporate metadata of the two demonstration datasets as well as the new dataset where we plan to generate new tasks as input. The benefit is that LLM can infer the underlying pur- pose of the dataset creation, thereby generating the most aligned tasks with the original intent, In this setup, LLM generates new tasks, with the in- put prompt being: "Now viv a dictionary as input, please help us to generate new tasks. You may stop when there is no more plausi ble task." and requirements being "Note that the input and output fields should not be duplicated and should both appear in [data fields]. Each task should still be dictio nary, containing no text or explanations outside the dictionary." " The full prompt is shown in Appendix B. This setting, however, still has limitations: firstly, comprehensive metadata may not be available for certain datasets; secondly, LLM exhibits proclivity towards dataset descrip- tions, leading to homogenization of the generated tasks. To mitigate these issues, additionally introduce the following setup.
Dataset Name. Dataset name sometimes pro- vides useful information to help us identify the domain and task category of a dataset. For ex- ample, dataset names with "bio" usually indicate that the dataset is in the biological domain; names with "nli" may suggest that the dataset is originally designed for natural language inference tasks.
Dataset Description. Dataset description offers more detailed information about the motivation for building dataset, the summary of dataset contents, and its supported tasks. facilitates LLM create instructions by supplying extra information about the dataset domain and initial dataset design.
Data Fields and Dataset Annotations. Data fields are the keys included dataset annotations. For example, given an instance ("title": "text": "author":
"subjects" "issued": ...). the data fields are "title", "text", "author", "subjects" and "issued". When LLM generates task instructions, it needs to determine which fields can be used as task inputs/outputs according to the semanties of data field names and contents of the data fields.
Description-Unaware Generation. To fully exploit the annotations and distinct data fields, we exclude the dataset description from the
All the metadata components are collected from the Huggingface Datasets Platform. We only col-

---

input, thereby allowing the LLM to freely gen- erate diverse task instructions and input/output fields. In this scenario, the dataset can be perceived as a description-less database, with the LLM generating diverse potential tasks based on the valid fields within it. For instance, the data fields in the Wikipedia-based QA dataset may encompass "title", "context", "question", and "answers". Possible new tasks could include Wikipedia article gen- Wikipedia
classification task, and add "Answer's must be one of [distinct outputs]." " the end of the instruction. We also discard classification tasks with extremely imbalanced distributions (e.g., only one distinct output value) in this step.
2,4 Statisties and Cases
In total, we collect 2,911 English datasets from the Huggingface Datasets Platform as of Feb 23, 2023, We then feed them to GPT-3 5- 5-turbo (OpenAT, 2023) and generate 13,610 tasks, of which 5,740 are valid and distinct. For each task, we sample up to 200 instances, ending in 801,900 instances that form the DYNOSAUR dataset. The diversity of the instructions shown in Figure 3. Following the approach of Wang et al. (2022a), we plot the top 20 most prevalent root verbs and their top 4 direct nouns, each of which appears at least 5 times. The instructions are quite diverse, especially when con- sidering we only have total of 4 demonstrations. Figure 2 demonstrates examples of datasets and corresponding tasks. The dataset name, dataset de- scription, data fields, and annotations are all used by LLM to design the tasks. LLM infers from the dataset name that it is about anaphor agreement and include this information in the instruction. In Ex- ample 2, LLM creates the task of paraphrase identi- fication by understanding the relationship between the fields "sentencel" and "sentence2" implied in the dataset description. Under the description- unaware setting like Example 3, tasks can be gen- erated based on the names of data fields.
3PT-3.5-turbee
eration ("title"=""ootext")) title generation ("context"==tittl")) and open-domain QA question generation "
By integrating these two settings, we ensure the preservation of the original intent of all datasets, while leveraging the creativity of LLM to delve deeper into the inherent potential in existing data.
2.3 Post-Processing
Filtering Invalid Tasks. Even though we de- scribe the requirements for valid task in the prompt, LLM sometimes neglects the requirements and generate invalid tasks. We filter out tasks with three criteria: I) tasks with non-existent data fields (for instance, a task with the output field "content" is invalid given the data in Figure 1); 2) tasks with more than one output fields; 3) tasks whose in- put/output fields overlap. Moreover, we remove duplicate tasks created during both the description- aware and -unaware generation.
Organizing Instruction Data. We organize the instruction data in the form of "instruction". "input", and "output". Given an instance of a dataset and a generated task containing the in- struction, input fields, and the output field, the "instruction" is the generated instruction and the "output" is the value of the output field. If there is only one input field, the "input" is the value of the input field; otherwise, the "input" de- scribes all the input fields with the format "The [field name] is [value of the field]."
Experiments
We conduct two sets of experiments to evaluate the quality of DYNOSAUR. We first evaluate mod- els trained with DYNOSAUR on SUPER-NI and LONGFORM to examine its ability to solve NLP tasks. Then we run a human evaluation to examine if DYNOSAUR helps in user-oriented situations.
3.1 Automatic Evaluation SUPER-NI and LONGFORM
Adding Label Spaces for Classification Tasks. As we only showcase several dataset instances to LLMs, it does not know the entire label space when generating classification task. As result, the gen- erated instruction may not contain the label space knowledge adequately. To overcome this issue, we automatically add the label space information in the instruction of classification tasks. We simply treat a task with less than 10 distinct outputs as
Experimental Settings. We fine-tune T5-3B and LLAMA-7B with variety of instruction-tuning datasets, including DYNOSAUR, SUPER-NI train- ing set, ALPACA, etc. LLAMA-7B is fine-tuned with LORA (Hu et al., 2022), an efficient fine- tuning approach. We also compare with larger models, including models based on T5-1IB, TO and TO++ (Sanh et al., 2022), Tk-Instruct (Wang

---

Exompe
Exmple 2
Examplc
Daimon Hame
pame indded_fful
Rataset Drcciation
Dateset Devrrption
BUMP major grammotica Engiit
PAWS Pauplinae Acvernzrios fram...
N/A
Data Fiide
Fislds
bad good"
"senteooe"" benoeee
"cnntest" hype "lagef"
Datoott Datastt Acnotacions
Datiost AosestAAznn Arnatalios
I'sentence bad sentence good" (.)
("sonter oel".... "sonteroo2).... 2..... "lasst:... :...
("corousd" " "pin_ype"" "urget:
"Elhispie"
"inetructioon loorl centence with an
"Inetrscrien" "Beteratse whathar twa sentencee aze paraphrmznz of each others" "inpat": "Benbenced as (d["entsncee"") Bentenced ianteeaee "cutgut"s "[d["aabel""
"Inetrucctonn Lon" "GLven context and type, the task to idantityy the target sord asscciated with bisn type."" "inpul"s "Conteat Ls [d]"centent"]] type [di"biax (tpps" type")))"
Encorsect anaphae agreerant, generate semtence with cocrect mnapbor ageementt" "isput"; "ldteeentence bad") I""
"oubput"; "edl""ettnce__ee [antecce_goodd]]
"outsutt" "Ill "targe"".)
Figure 2: Examples the datasets and generated tasks. We only demonstrate one task based on each dataset for simplicity. We highlight the parts in metadata that benefit instruction generation.
Models
Data Size
ROUGE-L
Models TRAINING!
Data Size
ROUGE-L
Larger TO
Larger
50K
33.1
TO
50K
33.1
T0++
12M
40.3
T0+44
12M
40.3
GPT-3 TRAINING
50K
GPT33 CELE-INSTRICCT
82K
39.9
GPT-3 w/ SELF-INSTRUCT'
82K
39.9
InstructGPT!
40.8
InstruciGPT!
40.8
with Generated Inzt Data
LLAMA 7B with Generated Inst. Data
15-3B w/ DOLLY
15K
17.6
LLAMA-TB w/ DOLLY
15K
33.5
T5.3R GPT.4
AMA-77 INST.
T5 3B w/ SELF-INSTRUCT
82K
37.1
LLAMA- 7B w/ SELF-INSTRUS
82K
39.6
T5-38
52K
LLAMA-7B
S2K
39.0
3B w/ DYNOSAUR
67K
40.4
LLAMA 7B w/ DYSOSAUR
67K
41.2
75-38 with Human-curated list. Dota
LLAMA-7B with Haman-curated
TS. 3B w/ PROMPTSOURCE
67K
38.9
LLAMA 7B w/ PROMPTSOURCE
67K
38.2
67K
T3-3B w/ SUPER-NI
68%
43.4
LLAMA- 7B w/ SUPER-NI
68K
42.5
Dynosaur as Angmentution Date
Dynesaur as Augmentaiion Data
T5-3B w/ DYNOSAUR SUPER-NI
135K
44.1
LLAMA-7B w/ DyNOSAUR SUPER-NI
135K
43.2
(a) TS-3B trained with various instruction datascts.
(b) ILAMA-7B trained with various instruction datasets.
Table 1: Evaluation results on SUPER-NI. "Inst." denotes "Instruction". The performance of models with f and are the reported results in Wang al. (2022a) and Honovich et al. (2022a).
et al., 2022b) and GPT-3 fine-tuned on PROMPT- SOURCE (Bach et al., 2022) and SELF-INSTRUCT.
GPT-3. 77-33.5ttumm t-rbbo as task category classifier! to cat- egorize each task into one of 76 task categories in SUPER-NI and avoid selecting tasks with test task categories. Details about fine-tuning hyperparam- eters and training task selection are shown in Ap- pendix C, E.1 and E.2. Following the original eval- uation on SUPER- and LONGFORM, leverage ROUGE-L (Lin, 2004) and METEOR (Banerjee and Lavie, 2005) as the metrics.
To alleviate the effect of data size dispar- ity, instead of training models with the entire DYNOSAUR, we sample a subset that shares a simi- lar data scale with other instruction-tuniin tuniingnnttnnngggnni datasets. Specifically, we select 681 tasks from DYNOSAUR training tasks and sample mostly 100 instances for each selected task, resulting in 66,695 instances in total. For SUPER-NT training set, also lect 681 tasks which are 90% of all SUPER- -NI training tasks and 87,825 instances. The rest 10% tasks are left as the validation set for SUPER-NI evaluation experiments. We also sample 67K data from PROMPTSOURCE and FLAN.
For all the evaluation experiments, we follow the Self-Instruct paper's setting and exclude all the positive and negative examples written in SUPER- NI instructions. It is for fair comparison with the datasets that contain instructions without any exam- ples, such as ALPACA, INST. GPT-4 and DOLLY.
During task selection of SUPER-NT, we ensure that all the selected tasks have distinct categories from SUPER-NT test tasks. Concretely, we use
We evaluate the GPT-3. 5- turbu classifier upon the hu- man evaluation from Amazon MTTurk on 200 classification results. The performance is 96.5%, suggesting the preciseness of removing tasks belonging to test task categories.

---

DYNOSAUR
Tie
SUPER-NI
Helptulness
19.5%
61.5%
19.0%
DYNOSAUR
Tic
ALPACA
Honesty
15.5%
71.8%
12.7%
ALPACA
Helptulness
18.7%
39.1%
22.2%
Harmlessness
13,5%
73.8%
12.7%
Honesty
17.5%
65.4%
DYNOSAUR
Tic
SUPER-NI
Harmlessness
15.5%
70.6%
13.9%
ALPACA
ÁLPACA
Helpfulness
17.1%
65.5%
17.4%
DYNOSAUR
INST. GPT-4
Tic
INST. GPT-4
Honesty
19.5%
59.9%
20.6%
Helpfulness
27.8%
42.9%
29.3%
Harmlessness
15.5%
73.4%
11.1%
Honesty
21.0%
59.9%
19.1%
DYNOSAUR
SUPER-NI NI
Harmlessness
19.8%
62.3%
17.9%
INST. GPT-4
INST. GPT-4
Helpfulness
18.2%
63.9%
17.9%
(a) DYNOSAUR supplement automatically generated instructions ALPACA and INST. GPT-4.
Honesty
17.9%
68.6%
13.5%
Harmlessness
(b) Comparing DYNOSAUR and SUPER-NI -NI
Table 2: Human evaluation LLAMA-7B with user instructions. The percentages in columns with dataset name A indicate how many of the generations produced by models trained with A are better than the ones produced by the other data on USER-INNTTRUCTION22555 "Tie" means that the generations of the two models have similar quality.
Models
METEOR
Models
METEOR
Existing Baselines
Existing Baselines
T0++-118 -1B
5.9
T0+4-118
5.9
Tk-Instruct
6.0
Tk-Instruct-
6.0
TS-1"T
T5-11B
12.5
Alpaca LLaMA-7B*
15.2
Alpaca LLaMA-7B
15.2
I5-3B with Hamun Hamon--craiedd curaied Inst. Data
LLAMA-7B 7B with Hamon curoted lust. Daia
TS 38 w/ SUPER-N NI T5-3B w/ PROMPESOURCE
3.8
LLAMA 7B SUPER-NI NI
6.2
4.8
LLAMA-7B PROMPTSOURCE
8.6
15-38 w/ FLAN
6.7
LLAMA-7B FLAN
11.5
T5:38 w/ DYNOSAUR
9.5
LLAMA-7B DYNOSAUR
19.0
(a) T5-3B trained with various instruction datasets.
(b) I.L.MA-7B trained with various instruction datasets.
Table 3: Evaluation results on LONGFORM. The performance of models with F are the reported results in Köksal
et (2023). Note that the listed existing baselines with suffix "-11B" indicate that their base model size 11B,
DYNOSAUR vs. Other Instruction-Tuiin Datasets on SUPER-NI. As shown in Table I, models trained with DYNOSAUR outperform the same models trained with ALPACA, SELF- INSTRUCT, INSTRUCTION GPT-4 and DOLLY. particular, training T5-3B with DYNOSAUR surpasses the variants trained with other datasets by significant margin around 2.5-22 ROUGE- L score. Also, we notice that fine-tuning smaller models with DYNOSAUR also achieves comparable performance than fine-tuning GPT-3 with SELF- INSTRUCT and PROMPTSOURCE data.
enhance model generalizability.
Other Instruction-Tunine Datasets LONGFORM. To further compare datasets DYNOSAUR and other instruction that are constructed with existing data, we evalu- ate them on LONGFORM, a recently released in- struction tuning benchmark for evaluating mod- els' instruction-folloiing ability on long text generation tasks. LONGFORM is equally un- seen to all these datasets. As shown in Ta- ble 3, DYNOSAUR largely outperforms the other three datasets SUPER-NI, PROMPTSOURCES and FLAN. In particular, with LLAMA-7B as the base model, DYNOSAUR outperforms the other datasets with 7.5-12.8 METEOR score. LLAMA- 7B trained with DYNOSAUR even surpasses other IIB instruction-tun models such as TO++ and Flan-T5 by a large margin.
DYNOSAUR
DYNOSAUR + SUPER-NI Training Set vs. SUPER-NI Training Set. The combination of DYNOSAUR and SUPER-NV NI training set can lead to higher performance than training with SUPER- NI training set. We first find that integrat- ing DYNOSAUR with SUPER-NI performs better than solely training with SUPER-NI around 1.2 ROUGE-L score Table 1. This suggests that DYNOSAUR can be considered as useful supple- ment for existing instruction tunninn data to further
Ablation Studies. We first evaluate how well models perform when only using either description- aware or -unaware instructions as training data. As shown in Table 4, considering both types of in-

---

oriented dataset the generation quality in practical domains such email writing. As there no test category constraint, we resample 67K data from all the task categories in DYNOSAUR. We fine-tune LLAMA-7B with the resampled data, and keep fine-tuning hyperparameters the same as SUPER-NI evaluation. We recruit annotators from Amazon Mechanical Turk, and ask them compare
1400
models' outputs from helpful- ness, honesty, and harmless (three criteria proposed by Askell et al. (2021)). See details about sampling evaluation and
d0
tasks for USER- IOSUR-INSTRUTTION human evaluation interface in Appendix E.3 and F.
DYNOSAUR as Augmentation Data to Auto- matically Generated Instructions. Admittedly, compared to automatically generated instructions whose seed tasks are closer to the ones for daily usage, DYNOSAUR is built upon data from exist- ing NLP tasks and is less involved in user sce- narios. However, DYNOSAUR can be used supplement to the automatically generated instruc- tions. As shown in Table 2a, training together with DYNOSAUR data outperforms solcly trained on AL- PACA or INSTRUCTION GPT-4 in the majority aspects. In particular harmlessness gains a steady boost after incorporating DYNOSAUR.
Figure 3: The top 20 most prevalent verbs and their top 4 direct nouns in the instructions of DYNOSAUR.
Data
ROUGE-L
Perf. of Trainning T5-3B with Following Date
DYNOSAUR
40.4
DYNOSAUR w/o Desp.-Unawar Inst.
40.0
DYNOSAUR w/o Desp-AAare Inst.
38.2
DYNOSAUR w/o Label Space
37.8
Perf. of Traininn LLAMA-7B with Following Deta
DYNOSAUR
41.2
DYNOSAUR w/o Desp.Unaware Inst.
37.6
DYNOSAUR w/e Desp--Aare Inst.
40.3
DYNOSAUR w/o Label Space
37.0
Table Ablation experiment results on SUPER-NI examine whether considering both description-awar and -unaware instructions can improve model perfor- mance. We also study if proc-processing technique like adding label spaces is helpful.
DYNOSAUR vs. SUPER-NI. We also compare DYNOSAUR with SUPER-NI -NI, both of them are constructed from existing task data. Table 2b man- ifests that the model trained with DYNOSAUR ex- ceeds SUPER-NI all the three aspects. More- over, DYNOSAUR an effective addition auto- matically generated instructions like INST. GPT-4 than SUPER-NI.
structions can produce better results than merely relying on description-awarr/unaawww We also study there exists performance drop af- ter we remove the label space descriptions in the instructions. From Table 4, the performance drops 2.6 and 4.2 ROUGE-L for T5-3B and LLAMA-7B.
instructions.
3.3 Unveiling More Benefits of DYNOSAUR
Beyond the evident advantages data qual- ity, which correspondingly enhance model per- formance, we elucidate the additional merits of DYNOSAUR from three perspectives: the validity of data, the cost-efficiency in data construction, and the potential for dynamic data expansion.
DYNOSAUR vs. Larger Models. From Table 1, we observe that T5-3B and LLAMA-7B with DYNOSAUR are comparable with some greater models. For example, our models are com- petitive with T0++ trained with orders of mag- nitude more data and 175B GPT-3 w/ SELF- INSTRUCT. This further shows the effectiveness brought from DYNOSAUR and implies decent qual- ity of DYNOSAUR.
Data Validity. We conduct human evaluation to scrutinize the validity of DYNOSAUR. We ran- domly select 200 task instructions and recruit evalu- ators from Amazon Mechanical Turk confirm the data validity. Each evaluator is instructed to choose from four options for each sample: "completely reasonable". "incorrect input", "incorrect
3.2 Human Evaluation on User Instructions
Experimental Settings. We conduct human evaluation on USER-INNTTUUCTION 2 user-

---

Dutaset
Data Size
Cost
aim to provide an analysis of how to most effec- tively select the tasks replay. We want answer the following questions: 1) Do we need to replay history tasks? 2) Shall we replay tasks based on instructions or data? 3) Which tasks to replay??
ALPACA
52K
5500
GPT-4"
52K
S456
UNKATURAL INST.
68K
$1,370
DYNASAUR sub
67K
S1.36
DYNASAUR-full
800K
$11.48
Table 5: The generation cost of different instruction tuning datasets. indicates that the cost estimation for INSTR. GPT-4 only involves output data generation, as uses the same instruction and input data as ALPACA.
Replay Methods. We compare the following re- play strategies: 1) No Replay: Train models with- out any replay tasks; 2) Instr. Diverse: Replay last stage's s tasks that diverge most from ones in the current stage based on instruction representations; 3) Instr. Similar: Replay last stage's tasks that are most similar to tasks in the current stage; 4) Instr. Support: Replay the most representative tasks in the last stage; 5) Data Diverse: Replay diverse tasks based on similarity of example data.
output", or "incorrect instruction". In situa- tions where a sample contains multiple errors, the evaluators are directed to highlight the most critical one. Remarkably, 84% of generated instances is completely correct. is a substantial improvement over the 54% reported in
Suppose there are tasks in the current stage, and K tasks in the previous stage, we use Sentence Transformer (Reimers and Gurevych, 2019) based on RoBER BRTTRaa-lrrge (Liu et al., 2019) to obtain the instruction representation matrix Ie E R Rl.xd for the current stage and Ip Rxdd for the previ- ous stage, where d is the representation dimension. Then, we compute the cosine similarity between Le and Ip. and 1p itself: S.p cos (LerIp) E RLIK Spp (Tp, Ip) C RKK Then, Instr. Diverse replays the tasks with the least column sum in Sep- Instr. Similar replays the tasks with the largest col- umn sum in Sep. Instr. Support replays the tasks with the largest row sum in Spp.
Data Construction Cost. On average, the cost to formulate a valid task encompassing the gen- eration of the instruction and input/output fields is approximate $0.002. Regarding the subset of our data, DYNOSAUR-Sub, utilized in SUPER-NI experiments, we sample 681 tasks and randomly select around 100 instances per task, resulting in a total cost of $1.36. Notably, the full version of DYNOSAUR achieves a data size of 800K in- stances via generating 5,740 tasks at a total cost of $11.5, This further reveals that our method is cost-efficient, thereby enabling the production of large-scale instruction-tuning datasets.
Metrics. Inspired by CL. literature (Biesialska et al., 2020; Lin ct al., 2022), we design three met- rics to quantify to what extent models generalize to new tasks, how well models perform on the training tasks in current stage, and how much models forget the previously trained tasks Test: ROUGE-L on the test set of SUPER-NI which represents unseen tasks; Holdout: ROUGE-L the holdout data of training tasks in current stage; Previous: ROUGE- L on the holdout data of training tasks in previous stages. As mentioned in $3.3, 16% of DYNOSAUR data are invalid. To avoid evaluating models on invalid holdout data, we do not report Holdout and Previous results for DYNOS experiments.
Dynamic Growth of Data.
The inherent design of DYNOSAUR fosters a capacity for dynamic growth, aligning seamlessly with the ongoing ex- pansion of the Huggingface Datasets Platform. As confirmed by statistics, as of May 20, an average of 143.6 datasets were incorporated into Huggingface daily in 2023, serving continuously as a rich data resource for DYNOSAUR.
4 Continual Learning with Dynamically Growing Datasets
As DYNOSAUR can expand over time as new tasks come in, an important question is how to adapt an instruction-tur tunod model to new tasks without suf- fering from catastrophic forgetting. In this section, we examine continual learning as an approach for learning Eestruction-oollowiin models with dynam- ically growing datasets. We focus on one of the common continual learning techniques (Biesialska et al., 2020), replay methods, which select previ- ously trained tasks for further training stage. We
Experimental Settings. We evaluate replay strategies by training T5-3B with SUPER-NI and
'A concurrent work (Kung et al., 2023) discusses the role of task active learning in effectively improving the general- ization ability on unseen tasks. We highlight here that the difference between the two settings is that we consider not only the generalization performance but the performance on history data as well.

---

Stuge I.
Stage
Stage 3.
Methods
Stage
Methods
Test
Holdout
Test
Holdout
Previous
Test
Holdout
Previous
3
Full
43.4
No Replay
40.6
53.3
40.5
56.3
50.9
43.3
60.1
58.2/ 49.3
No Replay
37.5
38.2
Instr Diverse
37,9
39,9
Data Diverse
43.0
58.9
53.3
42.8
60.3
60.7 52.7
Instr. Diverse
53.3
43,6
59,8
54.2
44,8
63,5
59.5/ 53.3
(b) Continual learning results of
Instr. Similar
42.9
59.2
41.0
/3.9/53.
T5-3B trained with DYNOSAUR on SUPER-NI test set. For simplicity. only compare replay with In- str. Diverse, the best replay strategy based on SUPER-NI.
Instr. Support
43.4
59.6
54.6
44.6
61.3
59.3/53.0
(a) Continual learning results of T5-3B trained with SUPER-NI. We divide the training into three stages. For each stage, report ROUGE-L the test holdout data in current stage, and holdout data previous stages.
Table Contimual learning results T3-3B trained with SUPER-NI and DYNOSAUR. "Full" denotes training with entire SUPER-NI and DYNOSAUR once.
DYNOSAUR. To simulate continual learning scenar- ios, we first randomly split both datasets into three groups. Then we train T5-3B for three stages, each stage one of the groups and 50 replayed tasks from the last stage. For cach task, sample 100 instances for training and another 100 instances for holdout evaluation.
One branch of instruction-tuming data are con- structed with existing human annotations. The instructions in PROMPTSOURCE (Bach et al., 2022) and FLAN (Wei et al., 2022) are cre- ated with human-designed templates for limited task categories. NI (Mishra et al., 2022) and SUPER-NV -N (Wang et al., 2022b) are annotated by NL.P practitioners from GitHub and NL.P courses. Most recent attempts distill instruction-tumi data from LLMs. The methods proposed in Self- Instruct (Wang et al., 2022a) and Unnatural In- struction (Honovich et al., 2022a) generate novel tasks by prompting LLMs with seed instruction- tuning tasks. Other works (Honovich al., 2022b; Zhou al., 2022) study instruction generation upon input/output data. There are another type of works simply using structured metadata as in- structions (Yin et al., 2023). Different from those works, when we generate DYNOSAUR instructions, the inputs/outputs for the generated tasks are un- known to LLMs. LLMs need to generate instrue- tions from metadata and determine which part of the dataset annotations task inputs/outputs si- multaneously.
Results. Displayed in Table 6a, we find that re- playing previous tasks not only mitigates forgetting issues, but also helps better generalize to unseen tasks. For example, in Stage 3, No Replay gets 43.3 on test set and 60.1 on the holdout of Stage 1, while Instr. Diverse achieves 44.8 and 63.5. Further, comparing Instr. Diverse and Data Diverse, we notice that selecting replay tasks based on the diversity of instruction representations bet- ter improves unseen task performance (+0.6/+2.0 at Stage 2/3). Besides, Instr. Diverse can even perform better test set than training with full SUPER-NI data at once.
We see similar trends on DYNOSAUR. We select the best replay strategy, Instr. Diverse, based on re- sults on SUPER-NI and compare it with No Replay. As shown in Table 6b, Instr. Diverse outperforms No Replay by 1.7 at Stage 3. Overall, a proper replay strategy can bridge performance gap or even help surpass training with full dataset.
Conclusions
We propose DYNOSAUR, an automatic paradigm for instruction data construction. We utilize meta- data from existing NLP datasets and generate vari- ous tasks upon them. DYNOSAUR generation costs significantly lower than other methods, while mod- els trained on DYNOSAUR data outperform models trained on existing human-curated and machine- generated instruction datasets on SUPER-NI and LONGFORM. Taking advantage of the dynamic growth nature of DYNOSAUR, we further explore specific replay methods for instruction tuning that are effective in mitigating forgetting.
5 Related Works
LLMs can be empowered to follow instructions via instruction tuning (Sanh et al., 2022; Ouyang et al., 2022; Wei et al., 2022; Mishra et al., 2022; Wang et al., 2022b; Chung et al., 2022; OpenAI, 2023; Wang al., 2022a; Longpre al., 2023; Taori al., 2023; Peng et al., 2023; Wu al., 2023). They fine-tune LLMs with the training data and instructions of diverse upstream training tasks and enable them to do inference on unseen tasks.

---

Limitations
supported in part by CISCO, and Kai- Kai-Wei was sup- ported as Sloan Fellow.
Limited Language Scope. DYNOSAUR is only built upon English datasets in Huggingface Datasets. Whereas, multilingual NLP datasets take up a large proportion in the platform. We plan to further curate a multilingual version of DYNOSAUR and conduct comprehensive experiments for evalu- ating generalization in multilingual settings,
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli. Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for align- ment. arXiv preprint 1
Errors in Generated Instruction Data. Al- though the data validity of DYNOSAUR is high, there are still 16% invalid data present in DYNOSAUR. We conduct error analysis (Appendix D) on the 200 instances used for human evaluation in $3.3 and notice that there are still multiple types of errors that have not been resolved yet. We ex- peet to seek better methods to improve the quality of generated instruction data in future works.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, 7aid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu. Gun- jan Chhablani, Han Wang. Jason Fries, Maged Al- shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. Prompt- Source: An integrated development environment and repository for natural language prompts. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisticx: System Demonstra- tions, pages 93-104, Dublin, Ireland, Association for Computational Linguistics.
Limited Sampled Dataset Instances. Due to the limits of data storage, we only sample at most 200 instances from each dataset for instruction-tunin data generation. We plan to consider more available instances from selected datasets and further scale up DYNOSAUR.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Sammarization, pages 65-72, Ann Arbor, Michigan, Association for Computational Linguis- tics.
Difficulty in Evaluation. is hard comprehen- sively assess the capabilities of instruction-tuna models (Zheng et al., 2023). We make our best ef- forts to evaluate models on a large-scale benchmark SUPER-NI with diverse tasks, along with human evaluation of user instructions.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual lifelong learn- ing in natural language processing: A survey. Proceedings of the 28th International Conference Computational Linguistics. pages 6523-6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Ethics Statement
Our work is based on annotations of existing datasets. As these data may contain selection bias or annotation bias, the bias may be inherited in our paradigm. We recruit annotators for human evalua- tion of data validity and task category classification from Amazon Mechanical Turk. All annotators fairly paid approximately $12 per hour.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual.
Acknowledgments
We thank UCLA-NLP lab members and anony- mous reviewers for their valuable feedback. The research is supported in part by ONR grant N00014- 23-1-2780, DARPA MCS program under contract number N66001 1924032, and an Amazon AWS credit award. Da Yin was supported by an Ama- zon Fellowship, Hritik was supported in part by AFOSR MURI grant FA9550-22- 20550-22-1-03800 1-0380, Fan was
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.

---

2022. Scaling instruction- finetumed language models. arXiv preprint arXiv:2210.11416.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Databricks. 2023.
Databricks' dolly, a large language model trained on the databricks chine learning platform. https:///tt twwb. com/ databriicksaabs/dll1y
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 190711907.11999
Amny Gudibande, Eric Wallace. Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. arXiv preprint 2.3.::2.305.1571...
Shayne Longpre, Le Hou, Tu Vu, Albert Webson. Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le. Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688.
Or Honovich. Thomas Scialom, Omer Levy, and Timo Schick. 2022a. Unnatural instructions: Tuning lan- guage models with (almost) no human labor, arXiv preprint 2rXiv:221..06689.
S Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsoureing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguistics.
Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. 2022b. Instruction induction: From few examples natural language task descriptions. arXiv preprint arxiv:220..107. 10782.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. 2022. Lora: Low-rank adaptation of large lan- guage models. In Interaational Conference on Learn- ing Representations.
OpenAL. 2023, Chatgpt. https: /openai. com/blog/ chatgpt/. Accessed on May 3, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27746
Po-Nien Kung, Fan Yin, Di Wu, Kei-Wei Chang, and Nanyun Peng. 2023. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks, In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. 2023. Longform: Optimizing in- struction tuning for long text generation with corpus extraction.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.0327.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-55 5551.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemiie,, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Seao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Majjee Majorr, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. Proceedings of the 2027 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bers Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Arun Raja, Manan Dey, et al. 2022. Multitask prompted training enables zero-shot task generalization. In International Con- ference on Learning Representations.
Bill Yuchen Lin, Sida Wang, Xi Victoria Lin, Robin Jia, Lin Xiao, Xiang Ren, and Wen-tau Yih. 2022. On continual model refinement in out-of-distribution data streams. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (ACL 2022).
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction following followiin Ilama model. https:// // gi github.co cow wtw/ tatsu-lab tannord_aapacc. _apcca.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 1399777.0...377.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith. Daniel Khashabi, and Han- naneh Hajishirzi. 2022a. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212.1056.0
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuzmia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri. Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. aapper-auurlllnstructooo General- ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference Empirical Methods Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. 2022. Finetuned language mod- els zero-shot learners. In International Confer- ence on Learning Representations.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muham- mad Abdul-Mageed, and Alham Fikri Aji. 2023. Lamini-Im: A diverse herd of distilled mod- els from large-scale instructions. arXiv preprint arXiv:2304. 14402.
Fan Yin. Jesse Vig, Philippe Laban, Shafiq Joty, Caim- ing Xiong, and you read the instructions? rethinking effective- ness of task definitions in instruction learning, arXiv preprint 2rXivv2306.0/155
Jason Wu. 2023. Did
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, 7i Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. 2023, Judging alm-as-a-judde with mt-bench and chatbot arena. arXiv preprint (rXiv:2306.05660 356855
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models human-level prompt engineers. arXiv preprint 211.092211.01910..

---

Appendix
rate 3e 4, lora, 8, and lordaipha 16. All the instruction datasets are finetuned with the same hyperparameters. All the fine-tuning experiments are performed with 48GB NVIDIA A6000 GPUs and 40GB NVIDIA A100 GPUs.
A Details of Metadata Collection
A.1 Extracting Dataset Name and Description
There are many datasets assumed as a subtask un- der parent dataset. For example, SST-2 is in- cluded as part of GLUE dataset. Then we concate- nate the parent dataset name and the dataset's s own name as final dataset name in collected metadata. Dataset description is extracted from dataset card. To shorten the input prompt for generating instructions, only capture the contents in the "Dataset Summary""
D Error Analysis for DYNOSAUR Data
We conduct enror analysis to investigate the error types of generated DYNOSAUR data. Among all 200 instances we evaluate human evaluation, we find that 4% of the human evaluated instances have incorrect instructions, 5% of the evaluated instances have incorrect inputs, and rest 7% have incorrect outputs. Representative wrong shown in Table 9. The cases with incorrect out- puts usually do follow the format requirements mentioned in instructions. The cases with incorrect inputs are unclear and do not meet the requirements noted in instructions. The incorrect instructions are typically irrelevant with the input/output contents.
A.2 Selecting Licensed Datasets
To properly leverage datasets at Huggingface Datasets, our metadata collection process does not apply on the datasets without any licenses, and the ones with cc-by-nc-nd-4.0 cc-by-nd-4. 0, cc-by-nc-nd-3.c 0. ofl, other, or unknown li- censes. The instances of the tasks eventually in- cluded in DYNOSAUR are subject to the licenses under which the original dataset was released.
E
Details of Sampling Strategies
E.1
Sampling Tasks for Evaluation on SUPER-NI
A.3 Removing Index and Nested Fields
Classification tasks take up great proportion of SUPER-NI test tasks. Meanwhile, most tasks of DYNOSAUR belong generation tasks. Therefore, sample classification tasks with higher proba- bility enforce models learn more classification tasks. There are 300 classification tasks among the 681 selected tasks.
Index field frequently exists in datasets, but it should be considered as part of task anno- tations. We remove this field reduce the effect of irrelevant information. Also, for simplicity, we also remove nested fields whose corresponding values are in a hierarchical dictionary structure.
B Example Prompt of Instruction Generation
We also notice that the tasks produced from Big- Science and Flax Stack Exchange datasets are more frequently selected. To promote the diversity of training tasks, our sampling method is constrained to select at most 70 tasks with regard to the two datasets. Besides, guage tasks to mitigate their negative effect on nat- ural language tasks.
We provide an example of the description-aware instruction generation Table 7. For description- unaware generation, the "summary" field is re- moved from the prompt.
discard programming lan-
C Details of Fine-Tuning Hyperparameters
To make a fair comparison, we also sample 67K training data for PROMPTSOURCE and FLAN, and exclude the task categories of SUPER-NI test tasks. Specifically, we exclude the tasks that belong to Structure-to-text Natural Language Inference, Coreference Resolution, and the task COPA from both datasets.
We fine-tune T5-3B with every studied instruction dataset for epoches, with batch size 16 and learn- ing rate le 5. We truncate all the input texts to 1024 tokens and limit the maximum output length 128 tokens. The number of linear warmup steps is set 600. We follow the hyperparameters of AL- PACA in finetuning LLAMA. Models are trained for 3 epoches with batch size 128 and the max length is 512 tokens. Due to memory limit, we apply LORA (low-rank adaptation) finetuning with learning
E.2 Sampling Tasks for Evaluation LONGFORM
Similar to the sampling strategy described in Ap- pendix E. we also sample 681 tasks from the

---

full version of DYNOSAUR, each with at most 100 instances. As the evaluation tasks in LONG- FORM usually have long outputs, we only sample the DYNOSAUR tasks that have the average output length above 50 words.
E.3 Sampling and Processing DYNOSAUR Tasks Tasks for Evaluation on USER-INSTRUCTION-222
Similar sampling tasks for SUPER-NI evalua- tion, also limit the maximum number of se- lected tasks regarding BigScience and Flax Stack Exchange datasets 70.
We observe that the instructions many user- oriented instruction data are not paired with any input data. For example, the instruction "What is the sum of and 5?" does not need any addi- tional input. Thus, after task sampling, we choose part of the instruction data in the sampled tasks and integrate their instructions and corresponding input data emulate the style of user-oriented in- structions. For example, assume that there is an instruction "Please tell me book wri tritten by the given author" and its corresponding input data "Victor Hugo". Through prompting GPT-3.5-turee we can harvest new instruc- tion, c.g., "Please tell me book wri riit by Victor Hugo", by combining its input data. We only perform integration on the instruction data whose corresponding input text is shorter than 50 characters. It aims at preventing task instruc- tions from carrying overwhelmed information. The prompt for integration is shown in Table 8.
F Human Evaluation Interface for USER-INSTRUCTION-22
We show the screenshot of human evaluation inter- face for USER- U INSTRUCTION 252 Figure 4.

---

Given dictionary containing dataset description examples, our goal is design up to three different tasks based this dataset. Each task should still a dictionary, including the instruction, input fields one output field. following are two examples.
Example 1:
Input:
("task_name": 'squad",
"selected_data""
[('title': "University_off Looooeoteeaaaeee 'context': 'Architecturally, the school has Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. ...', "question": 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?'), ('title': 'University_oo oowtrrsttyooooooree_eeeeeeee _ottto Caatre_Dmme,,. 'context': 'Architecturally, the school has Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. ...',
quesstio': "What is in front of the Notre Dane Main Building?'}1,
:sumaary' 'Stanford Question Answering Dataset (SQuAD) is reading comprehension dataset, consisting of questions pased by crowdworkers on set of Wikipedia articles, where the answer to every question is segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.""
Tasks:
(' ('task1'; ('instruction': 'Please answer the question based on the Wikipedia article. The answer to every question is segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.", 'input_fields': ['title', 'context', 'question'], 'output_field": ['answers']),
'task2'; ('instruction': *Create question provided the article.', 'ielut_fields': [ccontet'], 'output_ field'' ['question']]
'task3': ('instruction': 'Can you write title for the passage?', 'input_fields': ['context'], fiell't ['title']])
Example 2:
Now given dictionary input, please help generate tasks. You may stop when there is plausible task.
Input:
(' ('task_name'; revierevi seelected_data"
'review': "Great app! The new version now works on my Bravia Android TV which great as it's right by my rooftop aerial cable. The sean feature would useful... ETA on when this will available? Also the option import list of bookmarks e.g. from simple properties file would be useful.", 'date': 'Octaber 12 2016', 'star": '4'), ('package_name" 'com, wtw.mmattttttaaaeeaa rfanaalyet.....lyyerrrr...... 'review': "Great It's not fully optimised issues with crashing but still a nice app especially considering the price and it's open source.", 'date": "August 23 2016', 'star': '4'}]
[C'package_name
some
"summary': : 'It a large dataset which provides an overview of the types of feedback users report on the apps and documents the evolution of the related code metrics. The dataset contains about 395 applications of the F-Droid repository, including around 600 versions, 200 user reviews (extracted with specific text mining approaches))) )
Android applications belonging to 23 di different apps categories,
Note that the input and output fields should not be duplicated and should both appear in [' ('package_name' 'review', "date', 'star'] Each task should still be dictionary, containing no text or explanations outside the dictionary.
Tasks:
Table 7: An example of the prompt for description-aware instruction generation. Text in gray background varies across datasets.

---

We plan to infuse the text inputs into the user instructions. Here' two examples:
Instruction: Given sentiment label, generate movie review. Input: positive
New Instruction: Generate a positive movie review.
Instruction: Give some exarples of what people usually say in the given social situation. Input: when someone arrives safely New Instruction: Give examples of what people usually say when arrives safely.
Now please do the same thing for new instruction data:
Instruction: Please tell me book written by the given author Input:
Victor Hugo
New Instruction:
Table 8: An example of the prompt for integrating instructions and their corresponding input data to emulate the style of user-oriented instruction data. Text in gray background varies across instances.
Instructions
Inputs
Outputs
Errors
Given question, please gen- erate an answer.
When did Beyonce he- coming popular?
What question that would give the following an- swer><br>Answen the late 1990s:bbr
Incorrect output irrelevant questions
Please generate question and corresponding answers provided the context.
FINAL
Arthur's Magazine
Incorrect output follow- ing the format that the in- struction mentions
XXAM<br><broQuustioon 1. Which magazine started
first Arthur's Magazine or First for Women?
Please generate review based on the app name.
com. mantz jt_, raanayyeerr
Great app! The new version new works my Bravia Android TV which is great it's right by my rooftop acrial cable. The sean fea- ture would be useful...any ETA when this will be availahle? Also the option import list of bookmarks c.g. from simple properties file would be useful.
Incorrect input unclear in- put contents
Can you pick up the correct answer according the in- from COSMOS QA?
In the future, will this per- go other bands play?
This person likes music and likes to the show they will other bands
Incorrect instruction in- put from COSMOS QA is given
Table Error analysis for generated DYNOSAUR data. We provide the errors of each invalid instruction data in the last column.

---

Please thoroughly the Instruction and Response B) and will be asked to select the ome that sounds most Helpful to you (Question 1), the one that sounds honest (Question 2), and the one that Harmless (Question
with possible from different assistants
Question Helptuhess::
eesistant agent will always tyy to do wherl in humans best interests.
Question Homesty:
honest assistant
information the humens and will avoid decerving
Question Harmicssness:
hermlres eesistent will always by evoid doing enything that harms the humens.
Your Task:
$p4)
Instraction and Input
$0not_and__n _inout
Resporss
((elpaca)
Response
Bidynosaur alpaca)
Which the two respansas more helpful to you? more helpful
equally helpful
helpful
Which the two responses more honest you?
more honest
A and B are equally honest
la more honest
respanses
you?
C A more harmless O and B are equally harmless
more harmless
Figure 4: Human Evaluation Interface for USER-INSTRUCTTINN-22


Reviews:
[{"review_id": "sWUFIU4iAz", "reviewer": "Reviewer_FuNK", "summary": "", "questions": "- Will code/data be made available?\n- Why is performance of T5-3b only 43.4 on SNI? If I read the SNI paper right, it reports 54.3 in Figure 3.\n- L922-923: What are the tasks from BigScience and Flax Stack Exchange? Are there references for these efforts available that you could cite?\n- Are the results from a single run? If so, can some of the observed differences in scores be explained through high standard deviations?\n- How many instances were used for the human evaluation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments and nice words for our cost-saving advantage! We address the comments raised by the reviewer below.\n\n### **Part 1: A fairer comparison would have been to compare to SNI, T0 and/or Flan on a dataset that is equally \"unseen\" for all models:** \nOur claim is that by generating data-driven instruction-tuning datasets, the model learns better zero-shot ability. Therefore, we compare Dynosaur with the datasets like Alpaca and Inst. GPT-4, which are not generated based on metadata of existing corpus. \n\nHowever, we acknowledge the reviewer's concern. To demonstrate the effectiveness of our generated instruction-tuning datasets, we add experiments as follows:\n1. We fine-tune T5-3B and LLAMA-7B with two additional instruction-tuning datasets PromptSource and Flan and test the fine-tuned models on SNI. **Results show that Dynosaur can help to achieve better performance, compared with PromptSource and Flan**.\n2. We evaluate the models trained with Dynosaur, SNI, PromptSource and Flan on an instruction-tuning evaluation dataset Longform [1]. Longform is equally unseen to all the aforementioned datasets. **Dynosaur largely outperforms the other three datasets. In particular, with LLAMA-7B as the base model, Dynosaur outperforms the other datasets with 7.5-12.8 METEOR score**.\n3. We evaluate the models trained with Dynosaur, SNI, PromptSource and Flan on User-Instruction-252. **Dynosaur consistently outperforms the others on almost all the dimensions**.\n\nFor a fair comparison, we take the same numbers of training data, 67K, from the aforementioned instruction tuning datasets.\n\n[1] Köksal et al., 2023, LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction, Arxiv.\n\n—---------------------------------------\n\nOur results on SNI test set are shown as follows:\n\n|   **Metric: ROUGE-L**     | **T5-3B** | **LLAMA-7B** |\n|----------|-------|----------|\n| Dynosaur |   40.0 ± 0.2    |   40.4 ± 0.9      |\n| PromptSource       |   38.7 ± 0.2     |     37.4 ± 0.5     |\n| Flan     |   34.1 ± 0.3    |   39.8 ± 0.8       |\n| SNI      |   42.6 ± 0.8   |   42.5 ± 0.2     |\n\nWe find that Dynosaur can bring better performance than PromptSource and Flan for both base models, T5-3B and LLAMA-7B. It shows that our Dynosaur dataset can achieve comparable performance with expensive crowdsourced instruction tuning data built with much expert effort. Although Dynosaur underperforms SNI-train a bit, SNI-train shares a very similar instruction text style with SNI-test, SNI-train is more like in-domain instruction-tuning data and thus brings better results on SNI-test. Later on, we show that SNI has worse transferability on other out-of-domain instruction tuning data. Also, note that SNI-train was annotated by 88 NLP practitioners from GitHub and NLP courses. The annotation cost would be significantly higher than Dynosaur.\n\n—---------------------------------------\n\nWe also evaluate on Longform. Longform is a recently released instruction tuning benchmark for evaluating the model’s instruction-following ability on long text generation tasks. It is not evaluated in our original submission and we will include results on this dataset in the revision. The results are shown below:\n\n|  **Metric: METEOR**    | **T5-3B** | **LLAMA-7B** |\n|----------|-------|----------|\n| Dynosaur |   **9.5**    |    **19.0**      |\n| PromptSource       |  4.8    |    8.6      |\n| Flan     |   6.7    |    11.5      |\n| SNI      |   3.8   |   6.2     |\n\nDue to the time cost of generating long texts, for each of our studied data, we only evaluate on one random seed. We find that Dynosaur outperforms SNI, PromptSource and Flan for T5-3B. The performance gaps become greater when we use LLAMA-7B as our base model, which is 12.8, 7.5, and 10.4.\n\n—---------------------------------------\n\nWe further conduct human evaluation on User-Instruction-252. The comparison results are shown below: \n\n|              | **Dynosaur**  | **Tie**   | **SNI**   |\n|--------------|-----------|-------|-------|\n| Helpfulness  | **19.5%** | 61.5% | 19.0% |\n| Honesty      | **15.5%** | 71.8% | 12.7% |\n| Harmlessness | **13.5%** | 73.8% | 12.7% |\n\n|              | **Dynosaur**  | **Tie**   | **PromptSource**   |\n|--------------|-----------|-------|-------|\n| Helpfulness  | **20.6%** | 63.9% | 15.5% |\n| Honesty      | **19.0%** | 65.5% | 15.5% |\n| Harmlessness | **19.0%** | 62.3% | 18.7% |\n\n|              | **Dynosaur**  | **Tie**   | **Flan**   |\n|--------------|-----------|-------|-------|\n| Helpfulness  | **21.0%** | 58.4% | 20.6% |\n| Honesty      | 16.3% | 71.8% | **16.7%** |\n| Harmlessness | **15.5%** | 71.0% | 13.5% |\n\nThis dataset is also equally unseen to all of our studied data, Dynosaur, SNI, PromptSource and Flan. We find that Dynosaur consistently obtains better performance on almost all the dimensions than SNI, PromptSource and Flan.\n\nOverall, all of these experiments show that Dynosaur is on par with or even better than SNI, PromptSource and Flan on diverse instruction-tuning evaluation sets.\n\n### **Part 2: Helpfulness evaluation for the human evaluation experiments**: \nWe conduct a deeper analysis on the helpfulness evaluation. The possible reason why adding Dynosaur results in lower helpfulness scores is that models trained with Dynosaur and Alpaca/Inst. GPT-4 generate shorter responses, but people prefer longer texts as more helpful texts. This phenomenon is discussed in the AlpacaFarm paper [2]. We show a typical case observed in our experiment to further explain the phenomenon:\n\nInstruction: Enter the words that satisfy the given condition. Input: 5 Countries that Start with S.\n\n**Response of LLAMA-7B w/ Dynosaur + Alpaca**: 1. Sweden 2. Singapore 3. South Africa 4. Spain 5. Switzerland\n\n**Response of LLAMA-7B w/ Alpaca**: Here is a list of 5 countries that start with the letter \"S\": 1. Saudi Arabia 2. Senegal 3. Sierra Leone 4. Singapore 5. South Sudan\n\nBoth models provide five countries that start with “S”. We verify the correctness of all the countries. The main difference between the two model outputs is length - the responses of LLAMA-7B w/ Alpaca are longer than the other because the response attaches another sentence “Here is a list …” that doesn’t provide any additional useful information. However, 3 annotators think that the response of LLAMA-7B w/ Alpaca is more helpful. \n\nFor our experiments, we observe that adding Dynosaur as additional training data to Alpaca and Inst. GPT-4 will decrease the response length by 11.2 and 14.6 words on average, respectively. To mitigate the effect of the length confounder and show the actual effectiveness of our Dynosaur data, we perform a new round of helpfulness evaluation. Different from the evaluation we did in the original paper, for the new evaluation, annotators are asked to score the helpfulness of each individual model response of the two models, instead of doing the comparison between the outputs of two different models. Each response is annotated by three annotators. We find that annotators regard 87.3% of LLAMA-7B w/ Dynosaur + Alpaca responses as helpful, while only 84.5% of LLAMA-7B w/ Alpaca are helpful. Also, annotators treat 89.3% of LLAMA-7B w/ Dynosaur + Inst. GPT-4 responses as helpful, while only 87.3% of responses generated by LLAMA-7B w/ Inst. GPT-4 are helpful. Both comparison results show that adding Dynosaur can still help improve the helpfulness of model responses.\n\nWe conduct annotator inter-agreement experiments for the human evaluation discussed in Table 2 of our submission. Annotators we recruit for all the human evaluation experiments must finish at least 5000 HITs with at least 98% approval rate in Amazon MTurk. Any two annotators agree on 83.7% of all the instances. Cohen’s Kappa value is 0.68, which signifies substantial agreement according to the standards (https://www.statology.org/wp-content/uploads/2021/02/kappa1.png). The statistics further support the reliability of human evaluation studies. We will add the inter-agreement information in the following version.\n\n[2] Dubois et al., 2023, AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, Arxiv.\n\n### **Part 3: Detailed questions for authors**\n#### **Will code and data be released?**\nAs we stated in the abstract, code/data will be released upon acceptance.\n\n#### **Why is the performance of T5-3b only 43.4 on SNI? It is 54.3 in the SNI paper.**\nFor our experiments, we follow the Self-Instruct paper’s setting [3]. We exclude all the positive and negative examples written in the SNI instructions. It is for fair comparison with the datasets that contain instructions without any examples, such as Alpaca, Inst. GPT-4 and Dolly. The original SNI instructions contain multiple positive and negative examples which further help the model better understand the task definition and expected output format. Because of the setting difference, the performance is much higher than what we report. In Table 4 of the original SNI paper, the intersection of row “Def” and column ”Def” shows the performance under the same input formulation as ours. The performance is 45.0, which is close to our reproduced results. Also, as shown in Line 334-337, we only use 90% of the original SNI training data and leave the other 10% as validation. The smaller training data size also contributes to the performance difference.\n\n[3] Wang et al., 2023, Self-Instruct: Aligning Language Models with Self-Generated Instructions, ACL.\n\n#### **Any references for BigScience and Flax Stack Exchange?**\nFor BigScience, we leverage the datasets from the Huggingface spaces  https://huggingface.co/bigscience-data and [4]. For Flax Stack Exchange, we leverage the resource from [5]. (https://huggingface.co/datasets/flax-sentence-embeddings) We will add the references in our submission paper.\n\n[4] BLOOM, 2022, Huggingface BigScience Workshop, https://huggingface.co/bigscience/bloom.\n[5] Flax Sentence Embeddings Team, 2021, Stack Exchange Question Pairs.\n\n#### **How many instances were used for human evaluation?**\nThe number of instances in every human evaluation for the comparison between two model outputs is 252. It is the full size of the User-Instruction-252 dataset. For each of the 252 instances, there are 3 different annotators to score the comparison between two model outputs to mitigate the randomness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The prospect of using LLMs to create template-based instruction-tuning datasets is exciting, because the creation of comparable datasets have been large-scale efforts. For instance, both SNI and Promptsource each have over 40 authors. Dynosaur, on the other hand, could be created by spending less than $12 for the ChatGPT API.\n- Using the proposed technique for data creation could potentially unlock the simple generation of large-scale instruction tuning datasets for languages other than English and for specalized domains.\n​", "reasons_to_reject": "- I am not entirely convinced by the experimental results. That, when tested on SNI, models trained on Dynosaur oupterform those trained on Alpaca, Dolly, Self-instruct and Inst. GPT-4 is not particularly surprising. That is because Dynosaur's data distribution is arguably much more similar to SNI than that of the other datasets, which are more specialized for an interactive setting. A fairer comparison would have been to compare to SNI, T0 and/or Flan on a dataset that is equally \"unseen\" for all models.\n- I have related concerns regarding the human evaluation. The observed differences are rather small and adding Dynosaur to Alpaca and Inst. GPT-4 even leads to a drop in judged helpfulness, which is arguably the most important metric for an interative assistant. As far as I could see, the authors did not report any metrics on the reliability of the human evaluation which makes it hard to judge whether the observed differences are due to random fluctuations.\n- In summary, it's hard to judge from the experiments whether Dynosaur really can lead to consistent improvements over comparable instruction tuning datasets. In my opion, systematically showing that Dynosaur is on par with SNI and Promptsource would make the paper much stronger, because the cost savings in the creation process would be a decisive advantage alone.\n​", "questions_to_authors": "- Will code/data be made available?\n- Why is performance of T5-3b only 43.4 on SNI? If I read the SNI paper right, it reports 54.3 in Figure 3.\n- L922-923: What are the tasks from BigScience and Flax Stack Exchange? Are there references for these efforts available that you could cite?\n- Are the results from a single run? If so, can some of the observed differences in scores be explained through high standard deviations?\n- How many instances were used for the human evaluation?", "paper_topic_and_main_contributions": "The paper \"DYNOSAUR: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation\" proposes a method to automatically generate instruction-tuning datasets by combining LLMs with existing NLP datasets from the huggingface datasets hub. To achieve this, the authors propose to follow works like Promptsource and Super-NaturalInstructions (SNI) and construct prompt templates that transform each instance of a supervised NLP dataset into an instruction-tuning instance. As their main contribution, they replace the crowd sourcing step that Promptsource and SNI used by ChatGPT. For each NLP dataset from the huggingface datasets hub, they prompt ChatGPT with in-context learning to construct a prompt template based on the dataset's metadata. By applying this methodology, the authors obtain the proposed dataset, Dynosaur. They evaluate Dynosaur in an automated evaluation on SNI and in a human-judged evaluation on User-Instruction-252. On SNI the authors find, that T5-3B/Llama fine-tuned on Dynosaur outperforms T5-3B/Llama fine-tuned on other LLM-generated instruction tuning datasets, but performs worse than T5-3b fine-tuned on SNI.  Combining Dynosaur and SNI however, performs better than SNI alone. On User-Instruction-252, the authors find that when combining other instruction-tuning datasets with Dynosaur, the fine-tuned models are rated a bit higher in honesty and harmlessness but lower in helpfulness. Finally, the authors show that replay-based continual learning methods allow T5-3B to learn from new datasets added to Dynosaur without catastrophic forgetting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L336: \"67,825\" -> \"67,825 instances\"\n- L389: \"orders of magnitude data\" -> \"orders of magnitude more data\" [?]"}, "extra_text": {}}, {"review_id": "U2Cx4OodoZ", "reviewer": "Reviewer_Dkdk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments and nice words! \n\n**We will first clarify the definition of metadata.** As shown in Section 2.1, dataset metadata contains 1) dataset name, 2) dataset description, and 3) data fields and dataset annotations. All the datasets we select have metadata since they all have their names, data fields, and annotations. However, some of them don’t have dataset descriptions on their Huggingface dataset page. The instruction-tuning data generation for those datasets without dataset description will be based on description-unaware generation (See Section 2.2). When there’s no dataset description, our method can still generate instruction-tuning data based on the other metadata information (e.g., dataset name, data fields, and annotations).\n\nExample 3 in Figure 2 demonstrates a case for the description-unaware generation. There is no dataset description for the dataset called “stereoset-intersentence”, but it has other metadata information like dataset name, data fields, and annotations. The method can leverage these information to generate a new task instruction “Given a context and a bias type, the task is to identify the target word that is associated with the bias type.” We will make the description of our methods clearer in the future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using the metadata in existing large-scale NLP datasets is novel and effective. Experimental evaluation is comprehensive and strong.", "reasons_to_reject": "It is good that the generation of the instruction tuning dataset is using metadata of existing NLP datasets. However, for domain-specific datasets that often have no such metadata, this kind of instruction-tuning data would be difficult to obtain. The technique described in the paper could fail. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies automatic data curation for instruction tuning. The authors design a system called DYNOSAUR that uses LLM to generate task instructions based on the metadata from existing NLP datasets.  DYNOSAUR has low construction costs, shows better instruction-tuning performance than other baseline datasets of comparable sizes, and supports adding new instruction data to improve models. The authors further adopt continual learning on top of the instruction-tuning datasets and show the diverse instruction embeddings help reduce forgetting and better generalization. \n\nThe main contribution of DYNOSAUR is its data curation advantages in three aspects: low conversion costs, the effectiveness of instruction-tuning data, and support for new instruction data. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "04qzuPGuXW", "reviewer": "Reviewer_eo7N", "summary": "", "questions": "- Do you have any insight into why harmlessness and honesty improves with DYNOSAUR? It seems quite surprising that its main gains are not in the “helpfulness” metric.\n- Is there a difference in quality from the first task produced by ChatGPT to the third one in each example? Why did you predict 3 different tasks at once?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments! We address the reviewer’s questions and concerns as follows:\n\n### **Part 1: Polish writing:**\nWe will address these editorial comments, proofread, and improve the writing of the paper in the revision.\n\n### **Part 2: Discussion about continual learning (CL):**\n- The continual learning section aims to serve as an analysis (instead of proposing new methods) to show that our data curation process can be combined with CL techniques to deal with the ever-expanding new task data without hurting history tasks much. We hope the results will serve as a starting point and bring up an important research question.\n- We are aware that there are other techniques in CL besides replaying. However, replaying works on the data end which is perfectly coupled with our data curation technique. Also, it is worth noticing that the variants investigated are all valid baselines that are partially included in the literature cited by the paper.\n- We thank the reviewer for suggesting additional literature surveys in CL and error analysis in the data augmentation method. With the extra page upon acceptance, we will be able to better coordinate the CL section and include more literature context around CL.\n\n### **Part 3: Questions for the authors:**\n**Do you have any insight into why harmlessness and honesty improves with Dynosaur?**\n\nHonesty is a dimension that requires models to generate faithful responses to the instructions instead of hallucinating. Since Dynosaur data is generated based on existing human annotations, Dynosaur data are likely to suffer less from the hallucination. Adding Dynosaur data may help mitigate the hallucination issue. For harmlessness, we find that annotators usually label repetitive outputs as harmful ones. Since the outputs of Dynosaur data are generally concise, it will not encourage the model to generate too long and repetitive responses. Therefore, adding Dynosaur may also help improve the harmlessness. \n\n**Is there any quality difference between the first generated task to the third task in each example? Why did you predict 3 different tasks at once?**\n\nWe don’t notice significant difference in quality from the first task to the third one in each example. We choose to predict three tasks at once because the task difference can be better guaranteed by asking LLMs to generate three tasks at the same time. Based on our experience, it is hard to control the three tasks to be different if we generate the tasks one by one three times.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Introduces an interesting setup for more accurate automatic instruction-tuning data generation.\n- Presents an important evaluation of the generated data which finds high levels of correctness\n- Thorough automatic and human evaluations over a range of instruction-tuning datasets.\n- A small experiment concerning continual learning based on the fact that more datasets are continually added to the Huggingface platform.\n", "reasons_to_reject": "There are no serious issues that I can see with this work but there are a few issues that could be grounds for rejection until more work is done on this paper:\n\n- Writing needs more polishing\n  - Minor grammar mistakes \n    - Ungrammatical use of LLM throughout the paper (L074, L078, …)\n    - Awkward sentence “while L new 133 training tasks are coming.” L133-134\n    - “with T0++ trained with orders of magnitude data” L389\n    - “implies descent quality of DYNOSAUR.” L392\n    - “Moreover, DYNOSAUR is an effective addition to automatically generated instructions like INST. GPT-4 than SUPER-NI.” L426\n    - Very confusing sentence\n    - Figure 1 is very confusing and caption under-explains it\n  - More serious writing mistakes\n    - Too much detail in introduction, contributions should be clearer and more concise\n    - L050, SuperNatural Instructions definitely uses previous annotated data, confusing sentence\n    - Contentious statement given as fact: “an efficient fine tuning approach on par with normal fine-tuning.” L321\n\n\n- Continual learning section seems rushed and takes space from very important error analysis of the data augmentation itself and more detailed dataset descriptions\n  - Continual learning is a very rich research direction but the experiments presented here seem to lack contextualization (no baselines or discussion of previous work)\n  - Could be removed from the paper or minimized greatly\n", "questions_to_authors": "- Do you have any insight into why harmlessness and honesty improves with DYNOSAUR? It seems quite surprising that its main gains are not in the “helpfulness” metric.\n- Is there a difference in quality from the first task produced by ChatGPT to the third one in each example? Why did you predict 3 different tasks at once?\n", "paper_topic_and_main_contributions": "In this paper, the authors introduce a framework called Dynosaur which extends instruction tuned datasets using ChatGPT prompting as well as data and metadata compiled from the Huggingface Datasets Platform. This framework allows for much faster, cheaper and higher quality generation of large scale instruction-tuned datasets than other automatically created datasets like Alpaca and Self-Instruct. Using Dynosaur in combination with other instruction-tuning datasets such as SuperNI, Alpaca or Instruct-GPT4 leads to improvements. With data parity, improvements are still below carefully curated but less varied datasets like SuperNI. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Describes a method to generate instruction-tuning datasets from NLP datasets (e.g. on HuggingFace) in a more automatic fashion than prior work. In particular, the creation of templates for transforming inputs to instruction examples that are normally done by humans (e.g. PromptSource) is done by an LLM. The resulting large dataset (800k examples) is Dynosaur and is produced inexpensively ($12 querying ChatGPT). Base LLMs are fine-tuned on Dinosaur, and performance on Super-NI (ROUGE-L, automatic) and human evaluation are measured and outperforms competitors such as Alpaca. 

The paper provides a cheap, and effective tool in the instruction-tuning dataset generation tool-box.
Decision: Accept-Main