Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs
Authors: Young-Suk Lee
Md Arafat Sultan
Yousef El-Kurdi
Tahira Naseem
Asim Munawar
Radu Florian
Salim Roukos
Ramón Fernandez Astudillo
Venue: EMNLP 2023 Findings (2023)
Keywords: instruction tuning data generation
prompt learning
ensemble learning
TL;DR: Generation of high quality instruction tuning data from medium-sized language models (40B parameters or less)

Abstract: Using in-context learning (ICL) for data generation, techniques such as Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023) can train strong conversational agents with only a small amount of human supervision. One limitation of these approaches is that they resort to very large language models (around 175B parameters) that are also proprietary and non-public. Here we explore the application of such techniques to language models that are much smaller (around 10B--40B parameters) and have permissive licenses. 
We find the Self-Instruct approach to be less effective at these sizes and propose new ICL methods that draw on two main ideas: (a) categorization and simplification of the ICL templates to make prompt learning easier for the LM, and (b) ensembling over multiple LM outputs to help select high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct seed tasks and employs separate pipelines for instructions that require an input and instructions that do not. Empirical investigations with different LMs show that: (1) Our proposed method yields higher-quality instruction tuning data than Self-Instruct, (2) It improves performances of both vanilla and instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned LMs generate more useful examples than their larger un-tuned counterparts.

Ensemble-Instruet Generating Instruction-Tuning Data with Heterogeneous Mixture of LMs
Young-Suk Lee, Md Arafat Sultan, Yousef El-Kurdi, Tahira Naseem
Asim Munawar, Radu Florian, Salim Roukos, Ramón Fernandez Astudillo {ysuklee,yousefell yousefelk, tnaseem, raduf, roukoss@us. roukss ibm. com {arafat. san, asim, ramon. as astudillo) com
IBM Research Al
Abstract
et al., 2023) showed that small number of expert- annotated seed examples, coupled with in-context learning (ICL) with a base model, can be used to generate an instruction-tuning dataset to effi- ciently instruct that same base model. While this method yielded strong results and multiple follow- up works, most techniques resort to very large LMs (around 175B parameters) (Wang al., 2023; Taori et al., 2023), available only through closed-access APIs, or have restricted model access.
Using in-context learning (ICL) for data gener- ation, techniques such as Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori al., 2023) can train strong conversational agents with only a small amount of human su- pervision. One limitation of these approaches is that they resort to very large language models (around 175B parameters) that are also propri- etary and non- non-public. Here we explore the ap- plication of such techniques to language mod- els that are much smaller (around 10B-40B parameters) and have permissive licenses. We find the Self- Instruct approach to be less effec- tive at these sizes and propose new ICL meth- ods that draw two main ideas: (a) Catego- rization and simplification of the ICL templates make prompt learning easier for the LM, and (b) Ensembling over multiple LM outputs to help select hugh-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct seed tasks and employs separate pipelines for instructions that require an input and instruc- tions that do not. Empirical investigations with different LMs show that: (I) Our proposed method yields qualer-quality instruction tun- ing data than Self-Instruct, (2) It improves performances of both vanilla and instruction- tuned LMs by significant margins, and (3) Smaller instruction-tuned LMs generate more useful outputs than their larger un-tuned coun- terparts. Our codebase is available at https: //gi com/ IBM/ (esBmbeeebb---- bebteembb-iinsruttt...
In this paper, we present Busseble-IInstrcee a novel algorithm enabling high-quality instruction- tuning data generation with smaller LMs (40B pa- rameters or less), that are also fully accessible and have permissive usage licenses. We show that, when using smaller models as generators, Self- Instruct struggles to produce text of adequate qual- ity, adversely affecting the utility of the generated data and downstream model performance. Stay- ing within the ICL framework and using the Self- Instruct seed tasks, Ensemble-Insteee explores two main ideas to solve this problem: (1) Categoriz- ing and simplifying the ICL prompts to ease the few-shot learning process, and (2) Ensembling over multiple LM outputs to improve both accuracy and diversity of the generated data.
A standard instruction-tuning sample exempli- (a) an instruction that de- scribes the action to be performed, (b) an optional input on which the action is performed, and (c) the output of the action. Similar to Self-Instruct, we generate samples in two stages: instruction gener- ation and instance generation, where an instance comprises an input (optional) and an output. Unlike Instructt Mansemble-Instrue seeks to simplify the problem for the generating LM by first cate- gorizing the examples into two types-those with an input and those without-and then employing separate pipelines for the two that leverage their own unique and simplified prompts ($2.1). Further, ensembles over the outputs of different LMs
fies a task comprising:
1 Introduction
Instruction-tuned language models have demon- strated strong zero-shot generalization capabilities to new tasks (Chung et al., 2022a; Wei et al., 2021; Ouyang et al., 2022; Mishra et al., 2022; Wang al., 2022; Longpre al., 2023), creating interest in large-scale automatic synthesis of instruction- tuning data (Honovich al., 2022; Wang et al., 2023; Xu et al., 2032; Sun et al., 2023a; Xu et al., 2023). In this context, Instructt (Wang

---

complementary ways: (1) including examples generated by heterogeneous collection of LMs the final set to increase diversity, and (2) ma- jority voting followed by filtering low-consensus examples to improve accuracy ($2.4).
Algorithm 1 Output Ensembling
Input: LM outputs 01, 02, 03; Threshold t Output: Best output Obest
Obest <- None
To understand the effects of our proposed meth- ods, we run an extensive evaluation of different models for instruction generation. This includes vanilla language models (TS) U1.2-20B (Tay et al., 2022), FALCON-40B (Penedo et al., 2023), the instruction-tune tuned models FL-AN-T5-1 B (Chung et al., 2022b) and FLAN-UL2-20U (Tay et al., 2022) and the chat-tuned' version of GPT-NeoX-20B (Black et al., 2022). As base models to fine-tune with our generated data, we use the vanilla L.M Pythia-1.4B (Biderman et al., 2023) for ablation analysis, MPT-7B², a decoder only LM similar to LLaMA (Touvron al., 2023) as well GPT-JT- 6B², an instructed version of GPT-J (Wang and Komatsuzaki, 2021) trained on Chain of Thought and Natural instruction datasets among others. All chosen models are open-source and have permis- sive licenses (Apache-2).
2: Rs 4-
3: for (i, in {(1, 2), (1,3), (2, 3)) do
4: Rs < Rs Rou
5: end for
if min(Rs) > / then
i.j + argmax(Rs)
Obest oz end if
10: return Obest
A high-level overview of Ensemble-Instrre is given in Figure 1. The algorithm has three main components: Categorization of tasks and their associated prompts, (ii) Generation of instructions followed by instances, where an instance comprises an input (optional) and an output, and (iii) Ensem- ble of outputs from multiple LMs.
We evaluate the models fine-tuned on the data generated by Ensemble-Instrus
the the Super- Natural Instructions (SuperNI) test set (Wang et al., 2022) and 252 user-oriented tasks from Wang et al. (2023). Our contributions can be summarized as follows:
2.1 Categorization of Tasks and Prompts
We divide the tasks, i.e. the instruction-tuning sam- ples, into two categories: those where the instruc- tion needs an input to be meaningful (type A) and those where it does not (type B). Examples of tasks from these two types can be seen in Figures I and 2. Among the seed tasks of Wang et al. (2023), 125 belong to type A and 50 type B. For each cate- gory, we employ dedicated pipeline that (a) uses ICL demonstrations only of that type, and (b) tai- lors the number demonstrations to the difficulty of the type, at different stages of generation.
We propose technique for generating high- quality instruction-tuning data with 40B- parameter or smaller LMs that are openly ac- cessible, with non-restrictive licenses.
We outperform Self-Instruct training of GPT3 (175B) with far smaller base model (MPT- 7B). The technique also improves the perfor- mance of instruction-tuned GPT-JT-6B.
2.2 Instruction Generation
Ablation studies demonstrate the importance of the individual components of our tech- nique.
For type A tasks, we use 24 ICL demonstrations during instruction generation. Out of those, 20 are randomly sampled from the 125 seed tasks the same type, and 4 are sampled from instruc- tions previously generated by the model itself. For type B tasks, we use 10 ICL demonstrations, of which 8 are sampled from the 50 type B seed tasks and 2 from previously generated synthetic instruc- tions. Further, we adopt the approach of Wang al. (2023) of adding a new instruction to the set only if its Rouge-L (Lin, 2004) score with every existing instruction is less than 0.7.
We release the synthetic instruction-tuning dataset of about 45k samples along with our ICL templates and codebase.
2
Ensemblle-Instruet
'ntlps://hugginggaac wttpee/oooooteeeeeoeeeee totehhttccerr/tttee//////
OoT--eoXX-Chht-Basee-
thttps://www www.moosacm...ccm moosalccl.ccmm//lgg/nettt com/blog/mmt-77
hittps://hugging: acee. wttee.c//oooooooee www..togeeeerccmpuueeeeeeee
GPT-JT-68-v1

---

Generate new Instructions (requiring inputs) w/ LM1
Generate instances (input- oulput) for instructions w/
Consensus Filter Exact Match RougeL
LM1
instruction:
input: [10, 92, 2, 5,
Sort the given input ascend- ingly.
-4, 92, 5, [01]
instruction, input and filtered output
output: [-4, 2, 5, 5,
13. 92, 92, 101]
tasks with instruction, input and output
Additional Oulput LM2
output: (-4, 2, 5, 5, 10, 92, 92, 101]
175 Hluann-Annttale Seed Tasks:
Synthetic Training Examples
Additional Output I.M
instr action, ion, Input
output: [-4, 2, 5, 10, 101, 92, 92]
topt ooaa)) , and output
tasks with no inputs
Generate new Instruc-
Generate instances (output only) for instructions w/ LMI
tions (not needing
instruction and filtered output
inputs) w/ LMI
ouput: 85 29.400 44
instruction:
Additional Output output: 24%
Convert to Cal si5.
Consensus Filter Fxact Match
Additional Output LM3
output: 33.1%
or Rougel
Figure I: High-level overview of Ensemble-Instruet for synthetic instruction data generation. The top part generates data the tasks comprising instruction, input, and output while the bottom part generates for tasks without inputs. The instruction generation and instance generation steps done using the same LM with few-shot in-context learning. Additional LMs are used for the additional output generation, for which in-context examples are used only when the LM is not previously instruction tuned. In each box, the bottom gray portion gives an example of what is produced during that step.
2.3 Instance Generation
ated by the different LMs. The algorithm computes the Rouge-L score between all three pairs of out- puts. If the lowest Rouge-L. is above threshold f, returns the first element of the pair with the high- est Rouge-L score. This can be seen as a greedy version of Minimum Bayesian Risk decoding (Goel and Byrne, 2000) with additional thresholding. The minimum threshold & is set to 0.01 across all tasks. is important to note that if the above process does not select any of the three outputs, the example is filtered out.
During instance generation, we use 18 ICL demon- strations for type A tasks and 15 for type B tasks, randomly selected from the seed tasks. Figure shows examples of type A and type B tasks, and the prompts used for instance generation.
2.4 Output Ensembling
The instruction and instance generation steps should in principle complete the process of syn- thesizing an instruction-tuning sample (Wang et al.. 2023). However, samples generated by small LMs can inaccurate, which prompts us to design a final step output ensembling. Instead of simply accepting the already generated example, we use an additional set of LMs to predict new outputs, given either the generated instruction-input pair (type A) the instruction (type B).
Analysis of Instruction Tuning Dataset
We generate multiple instruction-tuning datasets using heterogeneous of LMs. Table I shows the labels of our synthetic datasets according to the LMs used different stages of generation. Table 2 summarizes the set of LMs we use for generation.
The final output is derived by applying the greedy consensus Algorithm to the outputs gener-

---

Label
Instructions
Instances
Additional Outputs for Ensembling
SO-FALCON
FALCON
FALCON
SO- (0-L NBOX)
UL2, CHAT-EEXT-CHAT
UL2, GPT-NEOXT-CHAT
EO-FALCON-LM
FALCON
FALCON
UL2, FALCON FLAN-UL.2, GPT-NEOXT-CHAT FLAN-UL2,
RO-FAI.CON-ILM
FALCCON
FALCON
EO-(U1,, NEOX NEOX|-ILM -ILM
UL.2, GPT-NEOXT-CHAT
UL2. GPT-NEOXT-CHAT
Table Labels of our synthetic tuning datasets according the LMs used for generating instructions, instances and additional outputs for ensembling. Datasets with outputs from single L.M and an ensemble of LMs are prefixed with So- and EO-, respectively. The rest of each label specifies the models that were used at different stages of the process. If additional outputs were generated using instruction-un sauude LMs for ensembling, the dataset is suffixed with -ILM. If vanilla LMs were used for the same purpose, we use the suffix -LM. With instruction-tur LMs, we generate the output zero-shot; for vanilla LMs, we use few-shot ICL.
3.1 Instance vs. Output Generation
Model FALCON
instruction
instance
ensemble
100
72
49 (68%)
As shown in Table I, we use a distinct set of LMs for instruction and instance generation on one hand and output generation for ensembling on the other. The motivations are two-fold: (1) We observed that only relatively large decoder only models with are capable of generat- ing onput- instances (type A). Therefore, we use decoder only models including FALCON, GPT- NEOXI-CHAT for input-output instance generation. (2) Instruction-tunne models are capable of gener- ating high quality zero-shot outputs. Therefore, we use tused models including FL.N--U.2, FLAN-15-XXL, GPT-NEOXI-CHAI for additional output generation for ensembling. We found that vanilla LMs UL2, FALCON lag behind instruction- tuned models for output generation, as shown in BO-FALCON-LM Table 4.
GPT-NEOXT-CHAT
100
40
25 (63%)
FLAN-UL2
100
0 0 (0%)
Table Number valid type A instructions and in- stances generated by different models for 100 samples as well and number (and percentage) of samples filtered by Algorithm All models share the same prompt and examples.
20B parameters
and examples $ Instructed models struggle to gen- erate valid instances and in particular FLAN-UL2 generates no valid instance for the 100 samples. Although not shown in the table, most LMs are capable of generating type B instructions and in- stances, indicating that instructions and instances that do not require an input is an easier task than those requiring an input.
Table 3 reports the number of valid instance gen- erations, well samples accepted by the ensem- ble Algorithm using FLAN-UL2 and FLAN-T5- XXL as additional outputs. We show results for 100 random samples using different models (FALCON, FLAN-UL2, GPT-NEOXT-CHATT to generate instruc- tion and type A instances using the same prompt
3.2 Small LM Dataset Comparsion
We instruction-tun Pythia-1.44 1.4B-deduped with dif- ferent datasets and evaluate them on the 119 tasks of the SUPERNI test set. For validation, we use 10,589 samples from 106 SUPERNI training tasks. Note that the validation and test sets have zero task overlap. We instruction-tune the model for 5 to 7 epochs and select the checkpoint with the highest validation Rouge-L score for evaluation. Perfor- mances of these tuned models the test set are shown in Table 4, where M-SELF-INST denotes the algorithm and ICL templates of Wang al. (2023) applied to [UL.2, NEOX), and F-SEIF-INNTT, the algorithm and ICL templates of Wang et al. (2023) applied to FALCON. We also show the performance of PYTHIAA-- 1...1...----- B-DDUUPED fine-tuned with two ex-
Model
# params
LM type
Rouge-1.
FALCON
40B
causal
12.7
UL2 GPT-NEOXT-CHAT
20B
seq2seq
10.4
20B
causal!
6.6
FLAN-UL2
20B
seq2seq'
77.5
FLAN-T5-XXI XXL
11B
seq2seq'
73.0
Table 2: LMS used for instruction-tuning data genera- tion. seg2seq denotes sequence- to sequence and causal denotes decoder-only. GPT-NEOXT-CHAT tuned on the OIG dataset'. FLAN-UL2 and FLAN-T5-XXL are tuned on FLAN collections. Both OIG and FLAN include SUPERNI data. Instruction-tur tuned models are denoted by 1, Zero-shot performance of each model on the SU- PERNI test set is provided in Rouge-L.
SSee https://gl ttt.bb...ttt..... com/IBN/ensemble- ine-iinsttrc/ wwob/mman//enee commmbl_instruut wwintnuut/saaplleee httpsmpplleinttaneessseeee taacee.ccspp for instance rejection criteria and seripts/ensembleiinttieee instruct.sh for experiment reproduction.

---

Dataset
samples
Rouge-I,
Instance Type
ZERO-SHOT BASELINE
9.8
criteria
output
input-output
total
ALPACA
51,760
33.4
GOOD
77
22
99 (70.7%)
SELF-INST
82,612
34.4
BAD
15
29 (20.7%)
M-SELF-INST
24,984
28.5
MAYBE
9
12 (8.6%)
SO-{ UL2, NEOX] }
25,660
33.6
total
100
40
140
EO- (UL2, NEOX JILM
18,218
38.3
Table 5: Manual evaluation synthetic instruction tun-
F-SELF-INST
38.624
25.6
ing data quality 140 randomly selected samples.
SO-FALCON
30,537
34.4
EO-FALCON-LM
26,503
32.9
EO-FALCON-ILM
26,701
37.1
Table 4: Efficacy synthetic instruction tuning datasets measured by the performance of PYTHIA- 1.BD-DEDUPED tuned models on the SUPERNI test set. Dataset labels are described in Table 1. ALPACA and SELF-INST external synthetic datasets for further comparisons. M-SELF-INST denotes the algorithm and ICL templates of Wang et (2023) applied to { (UL2, NEOX]. F-SELF-INST denotes the algorithm and ICL templates Wang et al. (2023) applied to FALCON. All training sets include the 175 seed tasks and the learning rate is le-5.
ILM and manually assign one of three categories to each: GOOD, BAD and MAYBE. GOOD indicates that there are no errors in the instruction, input (optional) and output, and the sample as a whole is coherent. MAYBE indicates that the input and the output do not contain errors, but the quality is questionable, e.g., the output is not complete. BAD indicates that the input or the output contains errors and incoherent with the instruction.
Manual evaluation results are shown Table 5, which was carried out by one of the authors. We find that examples containing only an instruction and an output (type B) are generally of higher qual- ity (77% GOOD) than those also containing an input (type A) (55% GOOD). This difference in quality is reflective of the relative difficulty of generating them by smaller models, i.c. is easier to generate output-only instances, as suggested in $3.1. Out of the 24,809 M-SELF-INST examples in Table (after excluding the 175 seed tasks), 20,752 (83.6%) are of type B, further demonstrating that it is easier to generate output-only instances. Ensemble-Instre pipeline avoids such unbalanced generation by first categorizing the tasks and then leveraging separate sets of simplified prompts for each. Each of our data sets generated with Ensemble-Instre almost even split between instructions with and without an input.
ternal datasets, ALPACA and SELF-INST? for com- parisons with much larger training data obtained with the SELF-INSTRUCT algorithm.
The performance gap between M- M-SELF-INST and (O-.2,,,, NEOX) shows that our categorization and simplification of ICL prompts for instruction and instance generation already improves perfor- mance over Sclf-Instruct. The same applies to the larger FALCON model, with SO-FAL-CON out- performing F-SELF-INST by a large margin. Out- put ensembling with instruction-tume LMs fur- ther improves performance in both settings. Impor- tantly, find ensembling with vanilla LMs via ICL less effective than ensembling with instruction- tuned LMs that were applied zero-shot. Finally, we produce data that more sample-efficient than Self-Instruct: With only about 30k examples, so- FALCONN yields a Rouge-L score 34.4, which is equal to what Self- Instructt yields with about 82k examples.
an
Figure 3 shows some synthetic examples before and after output ensembling, depicting a few differ- ent ways in which ensembling improves the quality of the generated output. Regarding the effect of en- sembling, observations show that it is particularly effective in selecting accurate output when it is short, e.g. classification tasks, via exact match. For longer outputs from generation tasks, e.g. summa- rization, the algorithm often filters out non-sense non-sensical outputs with hallucinations.
3.3 Qualitative Analysis
We randomly select 140 samples (40 with an input and 100 with no input) from EO- (UL.2, NEOX)
htttps:/hhugiingfacc. http:///gnaaaaeeeoteetttttt///aa alpaca-cleaned
albb/main/data//ptt___a httr/aaniidtttttt..eeeetttttoton
geneeratioon//bbatt..1111..
all_instances_822 82K, jsa.ss

---

Instance Generation with Both an Input and an Output:
Cenerate examples for the following instructions. The instruction requires input and output instances. And have to generate both input and output.
instruction: Extract all the country names in the paragraph, list them separated by commas. input: Dr. is the sixth novel by the English author Ian Fleming to feature his British Secret Service agent James Bond. Written at Fleming's Goldeneye estate in Jamaica, it first published the United Kingdom Jonathan Cape in 1958. In the novel Bond looks into the disappearance in Jamaica of two fellow MI6 operatives who had been investigating Doctor Bond travels No's Car Caribbean island and meets Honeychile Rider, who is there collect shells. They captured and taken luxurious facility carved into mountain. The character of Doctor No, the of German missionary and Chinese woman, was influenced by Sax Rohmer's Fu Manchu stories. Dr. was the first of Fleming's novels face widespread negative reviews Britain, but it was received favourably in the United States.
output: English, British, Jamaica, the Uni United Kingdom, German, Chinese, Britain, United States. |EoS]
instruction: Sort the given input ascendingly. input: [10, 92, 2, 5, -4, 92, 5, 101] output: [-4, 2, 5, 5, 10, 92, 92, 181]
|EaS]
instruction: In this task, you need to compare the meaning of the two sentences and tell they are the same, Output yes or no. input: Sentence 1: The teacher is speaking to the class. Sentence 2: The teacher is speaking to the students. output: yes |EaS]
instruction:
Instance Generation with Only Output:
Generate examples for the instructions. The instruction does require input and generate the output directly.
instruction:
Which exercises best for reducing belly fat home? output: Lying Leg Raises, Leg And Out, Plank, Side Plank, Sit-ups |EoS|
instruction: Is working part-time a good idea for teenagers between 15 and years old? output: There are both pros and cans to having a part-time jab as a teenager. the plus side, part-time job can teach responsibility, time management, and other important life skills. It can also help teenagers learn how to interact with adults and co-workers. On the downside, part-time job can take away from time that could be spent on homework or extracurricular activities. It can also be difficult to find a job that works around a school schedule.
[EoS|
instruction:
Figure 2: Example ICI. templates for instance generation. The top examples are instances with both an input and an output; the bottom are for output-only instances. We append the special token |EoS] the end of each example, clearly demarcating example boundaries, to make sure generation stops after the final output token.
4 Experimental Results
Evaluation Data
tasks
# samples
We carry out automatic evaluations using Rouge-L on the evaluation sets in Table 6. Both 119 Su- PERNI tasks and 252 user-oriented tasks are identi- cal to those evaluated in Wang al. (2023).
SUPERNI User-Oriented 252
119
11,810
252
Table 6: Evaluation datasets for automatic evaluations using Rouge-L. None of the tasks in the evaluation are seen during training.
We refer the reader to $4.4 of Wang et al. (2023) on the characteristics of the 252 user-oriented test set and SA.1 on

---

instruction: Find the maximum number of the number in set of numbers. input: 1, 2, 23, 50, 1, 2, 23, 50, 1, 6, 22
output-before: 1, 2, 23, 50, 1, 2, 23, 23
output-after: 50
instruction: Describe specific feature of the product in sentence. input: The feature of product is that it has rounded body shape, which is convenient carry around. output-before: Our product has round design, making it lightweight and portable. output-after: The product has rounded body shape.
instruction: You given description of item. Your task predict whether it can be found in not. Your output should be yes or rubber ball
input:
output-before:
output-after: yes
instruction:
Please describe situation where you felt really happy.
output-before: When you have a happy life and you are happy, you are happy for yourself. When you' happy for yourself, you are happy for everybody.
output-after: felt really happy when I won the game
instruction: what are some Foods that are high Vitamin C? output-before: 1. Tomatoes 2. Broccoli Strawberries 4. Papaya 5. Oranges output-after: oranges
Figure 3: Instruction tuning dataset examples before and after output ensembling. Ensembling generally improves different aspects of output quality, including correctness and adherence to the specifies of the question. We observe a side effect shorter outputs being preferred over longer ones generation tasks even in some cases that makes the output less accurate, shown in the last example.
We set aside 106 tasks (10, 589 samples) from the SuperNI 756 training tasks as the validation data set. For SuperNI instruction tuming, we ex- clude the validation set from training to simulate evaluation unseen tasks.
ples whereas MPT-7B is fine-tuned only on around 30K samples. MPT-7B fine-tuned on our syn- thetic data generated from instruction-tune tuned mod- els (SD II) outperform the data generated using vanilla LMs (SD I) by up to 3 points. Full fine- tuning outperforms QLoRA fine-tuning by 1.4 on HO- (UL22,EEO)] -ILM (46.8 vs. 45.4). Full fine- tuning again outperforms QLoRA fine-tuning by 2.2 on SuperNI training (50.4 vs. 48.2). MPT-7B fine-tuned on the combination of two synthetic data sets BO- UL2.NEOX U FALCON JILM and the Su- perNI training set improves the Rouge-L score over SuperNI training only by 2.2 points (from 48.2 to 50.4). We see similar pattern in Table 8 for the instruction-tunel base LM GPT-JT-6B, The fact that our synthetically generated data significantly improve the performance of the instruction -unned LM suggests that our technique generates data suf- ficiently different from the instruction tuning data incorporated into the base LM training.
We fine-tune 2 base LMs on the instruction tun- ing data generated by the current technique: (1) vanilla LM, MPT-7B, and (2) an instruction tuned LM, GPT-JT-6B." 9 To fine-tune these models, we adopt QLoRA (Dettmers et al., 2023), which en- ables us to train both LMs with a single A100 GPU (40GB memory) within 24 hours. We also car- ried out full fine-tuning of MPT-7B for 2 data sets, ILM and SUPERNI with A100 GPUs (80GB memory). The results are shown in Tables and for the SUPERNI test set, and in Table 9 for the 252 user-oriented test set.
BO- BO-(UL2 NO-(UL2.NEOX, )
In Table 7, MPT-7B fine-tuned on our syn- thetic data generated from vanilla LMs (SD 1) out- performs both TO and GPT3SELF-NNST despite the fact that the latter are fine-tuned on over 80K sam-
In Table 9, we note that both base models, MPT- 7 and GPT-JT-6B, perform worse on the user- oriented data set than on the SuperNI vs. 16.6 with MPT-7B and 6.2 vs. 10.4 with GPT- 9T--B. Fine-tuning these models on about 45K samples of the synthetic data provides a significant
the analysis the overlap between 175 seed instructions the two evaluation data sets.
"They first train 2.62 billion tokens using the UL2 loss on the Pile. (Gao et al., 2020). followed by 0.92 billion tokens (COT, Longpre et al. (2023)), 20% of Public Pool of Prompts (P3, (Bach et al., 2022)), 20% of SuperNI, and 55% of the Pilc.
set: 10.6
with mixture of 5% of Chain-of-Thoughe

---

Models
Params
Training Set
Samples
Rouge-1.
Vanilla Base LMs
TS-LM. Wang et al. (2023)
11B
None (ZERO-SHOT)
25.7
GPT3. Wang et al. (2023)
175B
None (ZERO-SHOT)
MPT
7B
None (ZERO-SHOT)
16.6
Instruction-tuned w/ SDI
TO, Wang et al. (2023)
11B
Self-Instruct (GPT3)
82,612
33.1
GPT3SELENNST: Wang et al. (2023)
175B
Self-Instruct (GPT3)
82,612
39.9
MPT ours
7B
SO-FALCON
30,537
43.1
MPT ours
7B
RO-FALCON-LMM
26,503
43.2
Instruction-tuned w/ SD
MPT ours
7B
BO-PALCON-ILM
26,701
44.4
MPT ours
7B
FO- FO-[UL.2,NG) (UL,,,NEOXXX J-ILM
18,218
46.8
MPT9ka ours
7B
FO- (O-[UU.2,NEOX/ IL
18,218
45.4
MPT ours
7B
EO- [UL2,NENX FALCON -ILM
44,744
46.4
Instruction-tuned w/ SUPERNI
Te- In-nntruct, Wang et al. (2023)
11B
SUPERNI SUPERNI SUPERNI SUPERNI
50,000
46.0
GPT3, Wang al. (2023)
175B
50,000
49,5
MPT
7B
64,528
50.4
MPT ours
7B
64,528
48.2
Instruction-tuned with SD II & SUPERNI
GPT3SLLFISTT Wang et al. (2023)
175B
Selt-unst & SUPERNI 132,612
51.6
MPT ours
7B
EO- COMBO-ILM & SUPERNI 109,272
50.4
Table Evaluation results on the SuperNI test set. SD denotes synthetic data generated from only vanilla L.Ms, and SD II, synthetic data generated from the combination of vanilla and instruction-tune tund LMs. Superscript" denotes full fine-tuning. Superscriptslun QLoRA fine-tuning. Learning rate set to le-6 for full fine-tuning and 5e-5 for QLoRA tuming. 50-COMBO-ILM denotes EO- (UL,,,,, NEOX FALCON] --MON)-MMMM -ILM. Combination of synthetic data E0-COMBO-ILM and SUPERNI training set improves over SUPERNI training set by 2.2 points, from 48.2 to 50.4. Instruction tuning with SD output-performs instruction tuning with SD I. For instruction tuning with SuperNI, subsample 100 instances from each of the 650 training tasks.
boost to the Rouge-L scores, from 10.6 to 22.1 for MPT-7B, and from 6.2 to 21.5 for GPT-JT-6B. This suggests that the synthetic data we generate capture the characteristics of user-oriented instructions to certain degree. Consistent with the results noted in Table for the SuperNI set, the data gen- erated by our technique is more effective than the data generated using Self-Inst (M-SELF-INST, F-SELF-INST) on the user oriented data set as well.
Trainset
# Samples
Rouge-L
ZERO-SHOT
10.4
FALCON
30,537
41.7
LO-FALCON-LM
26,503
40.5
LO-FALCON-ILM
26,701
41.9
EO- EO-(UL2,NEOXX) -ILM
18,218
42.7
EO-COMBO-ILM 44.744
43.1
SUPERNI
64,528
44.2
Table 8: Results (instraction-tuned base LM) GPT- JT-6B fine-tuned synthetic data. EO-COMBO-ILM denotes EO-/ NEOX U FAL.CON-II.M. All models are fine-tuned with QLoRA with learning rate 5e-5.
In Table 10, we show experimental results with other much larger models to illustrate the scala- bility of the proposed Enssemble-Instrun black-box models. Regardless of the base model sizes, ranging from 6B 40B, fine-tuning the base model with the synthetic data EO- [UL2, NEOX U FALCON -ILM improves the Rouge-L. score signif- icantly. The fine-tuned model performances seem to correlate well with the base model's parameter sizes, i.e. 43.1 for the smallest GPT-JT-6B, 49.9 for the largest FALCON-40B and all other model sizes and scores in between. In particular, the ex- perimental results on FALCON-40B indicates that Ensemble-Instruct is not an instance of model dis- tillation in the sense that the synthetic data gener- ated from FALCON-40B and smaller models signifi-
any
Models
Trainset
Rouge-L
MPT-7B
ZERO-SHOT
10.6
MPT-7B
M-SELF-INST
20.6
MPT-7B
F-SELF-INST
21.6
MPT-7B
RO-COMRO-IIM
22.1
GPT-JT-6R
ZERO-SHOT
6.2
GPT-JT-6B
M-SELF-INST
16.5
GPT-JT-6B
F-SELF-INST
17.4
GPT-JT-6B
LO-COMBO-ILM
21.5
Table 9: Results on the 252 user-oriented test set.

---

Model-Parammize
zero-shot
fine-tuned
et al., 2022), among others. Here use a similar technique in the context of instruction generation. To the best of our knowledge, this the first ap- plication of such an approach to instruction-tuning data generation.
GPT-JT-6B
10.4
43.1
MPT-7B
16.6
46.4
OPEN-LLAAM-
11.9
46.7
MPT-30B
12.2
49.5
FALCON-40B
12.7
49.9
an ensembling framework to improve the generaion qualities by leveraging the diverse strengths of multiple language models. While we utilize the output ensemble in the context of synthetic data generation with Rouge-L as the reference metric, LLM-Blender focuses on improving model output qualities using PairRanker and GenFuser, both ap- proaches capitalize on the efficacy of ensembling as a way of improving output qualities.
Jiang et al. (2023) proposes
Table 10: Fine-tuning results on large models demon- strating the scalability of the Ensemble-Instruct tech- nique to any black-box models. Zero-shot and fine- tuned model scores are Rouge-L. on NUPERNI test set. Performance improvement of FALCON-44 -40 after fine- tuning, compared with its zero-shot performance indi- cates that Ensemble- Instruct not an instance of model distillation. All models are fine-tuned with EO- (UL2, NEOX U F -ILM in Table 7.
Also related to this work are approaches directly distilling from ChatGPT GPT-4 (OpenAI, 2023) without specific instruction strategies, such as Vi- cunal", which distills ChatGPT, Baize (Xu et al., 2032), distilling conversations and Orca (Mukher- jee et al., 2023), which uses a large amount of Chat- GPT and GPT-4 outputs and combines FLAN tasks, system prompts and machine-generater explana- tions sampled from these models. The strength of these approaches seems rely more the amount and quality of teacher samples available than on the inductive biases of the self-instructing technique and still rely on proprictary models with non non-permissive licenses.
cantly improves all model's zero-shot performance including the largest model FALCON-40B.
5 Related Work
This work is directly related to Self-Instruct (Wang et al., 2023), borrowing from it the initial seed tasks and the idea of using ICL for tuning base model into a instruction following model. could also be seen as related to follow-up works such as: Alpaca (Taori et al., 2023) a practical application Self- Instract--Evo-Innstte
(Xu et al., 2023). which iteratively evolves instructions into increasing dif- ficulty levels and Dromedary (Sun et al., 2023b), which combines self-instruct with principle-based correction, similar to Constitutional AT (Bai et al., 2022). One fundamental limitation of these ap- proaches is that they resort to very large language models (around 175B parameters or 65B parame- ters at the minimum) that are also proprietary and non-public. Here we explore techniques for gen- erating instruction tuning data using LMs that are much smaller (around 10B-40D parameters) and have permissive licenses. We crucially draw heterogeneous mixture of smaller LMs to generate diverse outputs and then ensemble over multiple outputs to select high-quality synthetic examples, while also simplifying the instruction creation pro- cess.
Conclusion
We present a novel technique to generate Enstruction-tuing data through ICL., following the recent Self-Instruct work (Wang et al., 2023). Un- like lelf-Instruct, we propose techniques that ex- plicitly avoid the use of proprietary language mod- els like GTP-3, ChatGPT GPT-4. We show that when using smaller models, Self-Instruct becomes less performant. To overcome this, we draw on two main ideas: (a) Categorization and simplification of ICL templates to make prompt learning casier, and (b) Ensembling over multiple LM outputs to select high-quality examples. These ideas allow us to outperform training with Self-Instruct while utilizing the same seed tasks. The resulting syn- thetic data enables base models like MPT-7B to outperform GPT-3, far larger model with 175B parameters. The results of this work also encour- age the departure from closed-access models for advancing instruction generation algorithms.
The use of a reference metric, such as Rouge- L, to ensemble the outputs of multiple language distributions is common technique in Mini- mum Bayesian Risk decoding, with applications to speech-to-text (Goel and Byrne, 2000), machine translation (Kumar and Byrne, 2004), language modeling (Suzgun et al., 2022) and parsing (Lee
"https://mmsss.orr/bllgg -0---000030-vuaa/

---

7 Limitations
Connor Leahy, Kyle McDonell, Jason Phang, al. 2022. Gpt-neox-200 An pren-source autoregressive language model. arXiv preprint
Due to time and resource constraints, some parts of the experimental setup are not ideal. All model out- puts were collected from an internal API serving models from HuggingFace"' Due to limitations of this API, different number of samples were col- lected for each model which may have introduced noise in the performance estimates. We report the exact number of samples used for training along with the results. Note that for cases using ensem- bling one has to take into account that there is an additional filtering process that removes samples. We provide approximate rates for ensembling fil- tering in Table 3. For the small user-oriented test set containing 252 tasks, automatic evaluation is arguably not ideal. Proper human evaluation would provide a clearer signal but this requires of signif- icant time investment and resources. The method employs a set of various LMs, and therefore the generated synthetic data can be susceptible to the limitations of such LMs, particularly the biases in- herent in the training data which may be harmful leading to synthetic data with hate, abuse and social stereotypes.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Le, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022a. Scaling instruction- instraction-finetuned language models. arXiv:2210. 11416.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022b. Scaling instruction- finetuned language mod- els. arXiv preprint arXiv:2210.1144
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14314.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. 2rXiv:2101.0002
References
Vaibhava Goel and William J Bymee 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115-135
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea San- tilli, Zhiqing Sun, Srulik Bev-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani. Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xian- gru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. Promptsource: integrated develop- ment environment and repository for natural language prompts.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tun- ing language models with (almost) no human labor. arXiv:2212.0965
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Lim-blender: Ensembling large language models with pairwise comparison and generative fus ion. In Proceedings of the 61th Annual Meeting the Asso- ciation for Computational Linguisti cs (ACL 2023).
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077
Shankar Kumar and William Byrne. 2004. Minimum bayes-risk decoding for statistical machine transla- tion. Technical report, JOHNS HOPKINS UNIV BALTIMORE MD CENTER FOR LANGUAGE AND SPEECH PROCESSING (CLSP).
Young-Suk Lee, Ramón Fernandez Astudillo, Thanh Lam Hoang, Tahira Naseem, Radu Florian, and Salim Roukos. 2022. Maximum bayes smatch ensemble distillation for amr parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5379-5392.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.0133
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He,
Chin-Yen Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
"https: httpss///ghgggngggeeeec

---

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2rXiv:2302.13393
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In ACL.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A Billion Parameter Autoregressive Lan- guage Model. https: /ttp:://ittti thub..u..... www.shu.....onggg/eeeee com/kingoflolz/ wesh-traanffrmer-ttaa
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4, arXiv preprint arXiv:2306.02707.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hanna- heh Hajishirzi. 2023. Self-instruct: Aligning lan- guage models with self-generated instructions. In ACL 2023.
OpenAl. 2023. Gpt-4 technical report.
Yizhong Wang. Swaroop Mishra, Pegah Alipoor- molabashi. Yeganch Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, 2022. Superr-naurrriisstuuiiinssgegeee generalizatnnn via declara- tive instructions 1600+ tasks. In EMNLP.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll I.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arzivoorg/abs/2203.2222
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoe V Le. 2021. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, EEbtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306001166
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large lan- guage models follow complex instructions. arXiv preprint ar arXiv:2304. 12244.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023a. Principle-driven self- alignment of language models from scratch with min- imal human supervision. arXiv:2305.03047.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2032. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, arXiv: 2304.01196.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. 2023b. Principle-driven self- alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03047.
Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022. Follow wisdom the crowd: Effective text generation via minimum bayes risk decoding. arXiv preprint arXiv:2211.07634
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-followiii llama model.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unify- ing language learning paradigms. arXiv preprint arXiv.2205.05133.


Reviews:
[{"review_id": "ESN4z28PjZ", "reviewer": "Reviewer_NX6N", "summary": "", "questions": "1. The current method is not really selecting \"high-quality\" samples, but more like selecting \"popular\" samples, which may further magnifies the latent biases in the training data due to the training set exposure to different LLMs. \n\n2. Why separating the two types of tasks help? \n ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful and helpful review.\n\n**Rebuttals to Reasons To Reject:**\n\n**Critic**: The main performance metric used is the Rouge-L score, which is far from being satisfactory. If human evaluation is too costly, at least some AI evaluation could be done (e.g., using GPT-4).\n\n**Response**: We originally planned human evaluation of the 252 user-oriented test set, which we did not get to due to various constraints. Human evaluations of generative model outputs take a high priority in our next things to do.\n\n**Critic**: The key step is the high-quality sample selection from multiple LLMs. Although these LLMs were built by different groups, there might still be some hidden correlation between their outputs (e.g., due to training on the same subset of data). Hence, the current method is not really selecting \"high-quality\" samples, but more like selecting \"popular\" samples, which may further magnifies the latent biases in the training data.\n\n**Response**: We acknowledge that the current ensemble algorithm selects “popular samples” in the sense that  Algorithm I may be viewed as “majority voting”.\n\n**Critic**: There is no detailed case study, e.g., comparing the samples generate by the proposed method and those by the Self-Instruction method. Additional, such case studies can be carried out among different LLMs used in this study. It is desirable to see insightful analysis or trends from these studies.\n\n**Response**: In Section 3.2 (lines 239--247), we analyze that the examples generated by SELF-INST mostly consists of output-only instances (83.6%) whereas our proposed Ensemble-Instruct pipeline avoids such unbalanced generation. We will add more detailed analyses on the capabilities and limitations of different LLMs  in the final version of the paper.\n\n**Responses to \"Questions For The Authors\":**\n\n**Q**: The current method is not really selecting \"high-quality\" samples, but more like selecting \"popular\" samples, which may further magnifies the latent biases in the training data due to the training set exposure to different LLMs.\n\n**Response**: We acknowledge that the current ensemble algorithm selects “popular samples” in the sense that Algorithm I may be viewed as “majority voting”. \n\n**Q**: Why separating the two types of tasks help?\n\n**Response**: We initially observed that none of the LMs with 20b parameters or less was capable of generating decent input-output instances when ICL templates included the mix of output-only and input-output instances with heterogeneous labels for instruction, input and output. We hypothesized that separating out the two types of instances as well as unifying the labels would make the in-context learning easier for the LMs weaker than GPT3. And these hypothesis seems to be borne out. In Table 3 of Section 3.1, M-SELF-INST denotes the algorithm and ICL templates of SELF-INST applied to {UL2, NEOX} whereas SO-{UL2,NEOX} denotes our  algorithm incorporating categorized and simplified ICL templates. While RougeL of M-SELF-INST is 28.5, SO-{UL2,NEOX} achieves 33.6, i.e. 5.1 point improvement. F-SELF-INT denotes the algorithm and ICL templates of SELF-INST applied to FALCON whereas SO-FALCON denotes our algorithm incorporating categorized and simplified ICL templates. RougeL of F-SELF-INST is 25.6 compared with RougeL of SO-FALCON 34.4, i.e. 8.8 point improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposes a feasible method to create high-quality ICL samples for small (and hence less-capable) LLMs. This topic has practical importance in many application areas. \n\n* The evaluation is quite comprehensive, covering many settings and using several popular permissive LLMs. ", "reasons_to_reject": "* The main performance metric used is the Rouge-L score, which is far from being satisfactory. If human evaluation is too costly, at least some AI evaluation could be done (e.g., using GPT-4). \n\n* The key step is the high-quality sample selection from multiple LLMs. Although these LLMs were built by different groups, there might still be some hidden correlation between their outputs (e.g., due to training on the same subset of data). Hence, the current method is not really selecting \"high-quality\" samples, but more like selecting \"popular\" samples, which may further magnifies the latent biases in the training data. \n\n* There is no detailed case study, e.g., comparing the samples generate by the proposed method and those by the Self-Instruction method. Additional, such case studies can be carried out among different LLMs used in this study. It is desirable to see insightful analysis or trends from these studies. ", "questions_to_authors": "1. The current method is not really selecting \"high-quality\" samples, but more like selecting \"popular\" samples, which may further magnifies the latent biases in the training data due to the training set exposure to different LLMs. \n\n2. Why separating the two types of tasks help? \n ", "paper_topic_and_main_contributions": "The paper proposes an improved ICL sample generation method based on several permissive LLMs. The main idea is rather simple: it treat ICL templates (i.e., tasks) with and without input separately; for each task, it uses a greedy consensus algorithm to select high-quality samples based on the output of multiple LLMs. \n\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "1. It is desirable to briefly describe the Self-instruction method to make the paper self-contained. \n\n2. The floating of various tables & figures makes reading rather hard. "}, "extra_text": {}}, {"review_id": "7vSjcBqptH", "reviewer": "Reviewer_CwUv", "summary": "", "questions": "Can you please add a Figure to visually show the process of generating instruction-tuning samples?\n\nDid you consider ensembling based on model confidence scores, rather than RougeL?\n\nIs there a graph that shows how much downstream results improve as a you add more synthetic data?  I'd be curious to know if there is some tipping point where performance improves dramatically.  Also, include the point at which your ensemble-instruct data surpasses the self-instruct data.  ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for insightful and helpful review.\n\n**Rebuttals to \"Reasons To Reject\":**\n\n**Critic**: Ensembling is not a particularly novel way to denoise generated samples. More sophisticated methods exists and it would have been nice to compare against them.\n\n**Response**: We will try to explore techniques other than ensembling to denoise generated samples.\n\n**Critic**: Could be improved with more experiments on how many candidates to ensemble, or how many examples to include during ICL.\n\n**Response**:  We will try to incorporate the analysis in our next round of revisions.\n\n**Critic**: Missing larger scale human evaluation for qualitative review.\n\n**Response**: We originally planned human evaluation of the 252 user-oriented test set, which we did not get to due to various constraints. And human evaluations of generative model outputs take a high priority in our next things to do.\n\n**Responses to \"Questions For The Authors\":**\n\n**Q**: Can you please add a Figure to visually show the process of generating instruction-tuning samples?\n\n**Response**: Figure 1 is intended to be a visual illustration of our instruction tuning sample generation process. \n\n**Q**: Did you consider ensembling based on model confidence scores, rather than RougeL?\n\n**Response**: Although we thought of this possibility, it was not feasible due to our black box approaches to the language models for data generation. We are currently integrating the synthetic data generation pipeline into our model fine-tuning and inferencing framework, which will enable us to utilize model confidence scores in addition to external evaluation metrics for emsembling model outputs.\n\n**Q**: Is there a graph that shows how much downstream results improve as a you add more synthetic data? I'd be curious to know if there is some tipping point where performance improves dramatically. Also, include the point at which your ensemble-instruct data surpasses the self-instruct data.\n\n**Response**: We applied our ensemble-based approach to content-grounded synthetic data generation for RAG-style QA tasks and observed that adding more synthetic data to human annotated data improved response generation system performances. We will try to plot graphs, illustrating the impact of synthetic data on downstream tasks both in terms of synthetic data size and quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Paper is well written and easy to follow.  Ensemble-Instruct technique shows clear gains over Self-Instruct across a wide variety of settings. \n\nAblation studies show that both aspects (simplified prompts, ensemble) are both useful in leading to superior performance.\n\nSynthetic dataset of instruction-tuning examples can be used by others to build on.  The dataset only includes 45k examples, but I can see this being expanded to 100K+ examples if the codebase is opened up to others.\n\nAnything that works with open-source, permissive licenses is great for the community, especially when doing so outperforms the black-box counterparts.", "reasons_to_reject": "Ensembling is not a particularly novel way to denoise generated samples.  More sophisticated methods exists and it would have been nice to compare against them.\n\nCould be improved with more experiments on how many candidates to ensemble, or how many examples to include during ICL.\n\nMissing larger scale human evaluation for qualitative review.", "questions_to_authors": "Can you please add a Figure to visually show the process of generating instruction-tuning samples?\n\nDid you consider ensembling based on model confidence scores, rather than RougeL?\n\nIs there a graph that shows how much downstream results improve as a you add more synthetic data?  I'd be curious to know if there is some tipping point where performance improves dramatically.  Also, include the point at which your ensemble-instruct data surpasses the self-instruct data.  ", "paper_topic_and_main_contributions": "Authors propose a novel method of generating instruction tuning data, which can then be used to modify LLMs be become better aligned to human preferences.  They offer two key techniques (1) Categorizing and simplifying the ICL prompts and (2) ensembling over multiple LM outputs to improve accuracy and diversity. Ensemble-Instruct leads to a \"small\" model (MPT-7B) outperforming a much larger one (GPT 3 at 175B) trained using the typical Self-Instruct method, as tested on two benchmarks (SuperNI and User-oriented tasks)\n\nCategorization is broken down into two types: (a) those with inputs and (b) those without inputs.  (The required parts of an instruction sample are the instruction and the output.)   By separating these types, the prompts can be simplified without losing any impact.  Specifically, Pipeline A (for inputs) includes adding the input during data generation.  In contrast, Pipeline B (without inputs) includes a different seed set of data.  \n\nThe ensemble of multiple LMs (a) improves diversity by using many different types of LMs and (b) using majority vote of LMs to improve quality of data.  The voting is conducted by comparing the Rouge-L score of candidate generations, keeping only those above a certain threshold. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cAWjeiTWft", "reviewer": "Reviewer_gP8F", "summary": "", "questions": "- A: Why do UL2 and NEOX appear together? Specifically, isn't UL2 vanilla model and NEOX ILM?\n- B: Is the generation via sampling? In that case, what about generating multiple SOs and self-ensembling them?\n- C: Why do you think the vanilla model ensembling is not very effective?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive review.\n\n**Rebuttals to \"Reasons to Reject\"**\n\n**Critic**: The proposed methodology appears disconnected from the primary motivation of the paper, which revolves around the model's size and openness. Although it was demonstrated using small models, there is potential for its applicability to all black-box models. To better showcase the effectiveness of the methodology, conducting experiments with larger models would be beneficial.\n\n**Response**: We provide new  experimental results on three larger models, including falcon-40b, mpt-30b and open_llama-13b. All models are qlora fine-tuned with 44,744 synthetic samples generated by our ensemble-instruct technique, i.e. EO-{UL2, NEOX U FALCON}-ILM in Table 6. Experimental results on the SNI evaluation data set are shown below, where the numbers in parentheses are obtained after post-processing the model outputs.\n\n| Models                                    |   zeroshot baseline RougeL |  qlora fine-tuned RougeL |\n| :--                                            |     --:                      | --:                       |\n| GPT-JT-6b (Table 6)                 |    10.4 (36.4)          |  43.1                   |\n| mosaicml/mpt-7b (Table 6)      |    16.6 (31.3)          |   46.4                  |\n| **open_llama-13b  (new)**       |    11.9 (32.6)          |  46.7                   |\n| **mosaicml/mpt-30b  (new)**   |    12.2 (44.8)         |  49.5                   |\n| **tiiuae/falcon-40b (new)**       |     12.7 (41.4)        |     49.9             |\n\n**Critic**: The analysis of the effectiveness of categorization and simplification lacks in-depth exploration, and it would be advantageous to delve further into this aspect. While section 3.2 briefly mentions the balance between type A and B, a more comprehensive analysis would add valuable insights.\n\n**Response**: Table 3 of Section 3.1 provides a quantitative analysis.  M-SELF-INST denotes the algorithm and ICL templates of SELF-INST applied to {UL2, NEOX} whereas SO-{UL2,NEOX} denotes our  algorithm incorporating categorized and simplified ICL templates. While RougeL of M-SELF-INST is 28.5, SO-{UL2,NEOX} achieves 33.6, i.e. 5.1 point improvement. F-SELF-INST denotes the algorithm and ICL templates of SELF-INST applied to FALCON whereas SO-FALCON denotes our algorithm incorporating categorized and simplified ICL templates. RougeL of F-SELF-INST is 25.6 compared with RougeL of SO-FALCON 34.4, i.e. 8.8 point improvement.\n\n**Critic**: The absence of a comparison with other datasets, such as Alpaca mentioned in the paper, is somewhat regrettable. Including such comparisons would provide a more comprehensive evaluation of the proposed approach. Additionally, a qualitative comparison with other methods would be valuable in further validating the proposed methodology.\n\n**Response**:  To  address this concern, we fine-tuned PYTHIA-1.4B-DEDUPED LM on https://huggingface.co/datasets/yahma/alpaca-cleaned under the same experimental condition as all other models in Table 3. Shown below is the summary, where Alpaca dataset performs similarly to SELF-INSTRUCT but worse than our proposed Ensemble-Instruct.\n\n| Dataset                                                |    Sample size    |  RougeL |\n| :--                                                         | :--                      | :--           |\n| **Alpaca (new)**                                  |   51,760            |    33.4    |\n| SLEF-INSTRUCT (Table  3)                 |   82,612             |    34.4     |\n| EO-{UL2, NEOX}-ILM (Table 3)           |   18,218             |   38.3      |\n| EO-FALCON-ILM (Table 3)                 |   26,701             |   37.1      |\n\n**Response to \"Questions For The Authors\":**\n\n**A**: Why do UL2 and NEOX appear together? Specifically, isn't UL2 vanilla model and NEOX ILM?\n\n**Response**: Before the release of falcon-40b (06/09/2023, 1 week before abstract submission deadline) and mpt-30b (06/28/2023) models, UL2 and NEOX (each 20b parameters) were the only models capable of generating both the input and the output given an instruction. Since we were looking for ways of diversifying the instructions (and consequently the input and output instances), we resorted to these models for initial instance generations.\n\n**B**: Is the generation via sampling? In that case, what about generating multiple SOs and self-ensembling them?\n\n**Response**:\n\n* For input-output generation given instruction, generation is done via sampling. For zero-shot output generation given instruction and  input, generation is done via greedy decoding. \n\n* We tried ensembling multiple outputs generated via sampling for falcon-40b, which turned out to be similar to or slightly lower than the performance of a single output, cf. EO-FALCON-LM in Table 3. \n\n**C**: Why do you think the vanilla model ensembling is not very effective?\n\n**Response**: For instruction tuned models such as flan-t5-xxl and flan-ul2, high quality outputs are generated zero-shot given the instruction and input.   On the other hand, output generation requires few shot learning with ICL templates for vanilla auto-regressive LMs. For paper submission, we used the same ICL templates for output generation as instance generation, which was not effective. However, our ongoing investigation indicates that vanilla model ensembling can also be effective with re-design of ICL templates and diverse outputs from multiple LMs, e.g. Falcon and LLaMA-2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The simplicity of the methodology is a key advantage, as it facilitates easy application, which is commendable.\n- Despite utilizing smaller models, the approach achieves notable performance, particularly demonstrating improved results even with a limited number of samples.\n- The insightful analysis of the differences between ILM and LM, following experimentation with both models, adds valuable understanding to the study.", "reasons_to_reject": "- The proposed methodology appears disconnected from the primary motivation of the paper, which revolves around the model's size and openness. Although it was demonstrated using small models, there is potential for its applicability to all black-box models. To better showcase the effectiveness of the methodology, conducting experiments with larger models would be beneficial.\n\n- The analysis of the effectiveness of categorization and simplification lacks in-depth exploration, and it would be advantageous to delve further into this aspect. While section 3.2 briefly mentions the balance between type A and B, a more comprehensive analysis would add valuable insights.\n\n- The absence of a comparison with other datasets, such as Alpaca mentioned in the paper, is somewhat regrettable. Including such comparisons would provide a more comprehensive evaluation of the proposed approach. Additionally, a qualitative comparison with other methods would be valuable in further validating the proposed methodology.", "questions_to_authors": "- A: Why do UL2 and NEOX appear together? Specifically, isn't UL2 vanilla model and NEOX ILM?\n- B: Is the generation via sampling? In that case, what about generating multiple SOs and self-ensembling them?\n- C: Why do you think the vanilla model ensembling is not very effective?", "paper_topic_and_main_contributions": "The paper proposes a new methodology for creating instruction-tuning data using LLM.\nParticularly, instead of using excessively large and less accessible 175B-level models that were commonly used in the past, it suggests utilizing relatively smaller models below 40B.\nThe proposed method consists of two key methodologies: the first one involves the categorization and simplification of ICL templates, and the second one is about ensembling multiple LM outputs\nThrough experiments, the proposed approach demonstrates superior data generation compared to self-instruct, shows good performance on both vanilla and ILM, and indicates that smaller ILMs generate data better than larger LMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- While there are citations, it would greatly enhance the paper's readability and understanding if it directly compares itself to Self-instruct, highlighting the differences.\n- In T7, there seems to be a reference to SO-FALCON, but the \"SO\" part is missing, which causes some confusion.\n- It would be beneficial to move the examples to the appendix and use the main content to provide additional analysis and supplementary information. This adjustment would make the paper even better, as the current amount of space dedicated to examples feels excessive relative to the information they provide."}}]

Author rebuttal:

Meta-review: Proposes a method to generate instruction-tuning data using smaller LLMs in contrast to most earlier work such as Self-Instruct. The main idea is to ensemble multiple LLMs via a greedy consensus algorithm (high inter-model ROUGE-L) to select high-quality samples; the result is similar performance to using much larger closed-LLMs, as measured by ROUGE-L. One issue with closed-LLMs is they often do not have permissive licenses and so a way to generate IT data without them is potentially valuable to the open-source community.

There were concerns about of using solely ROUGE-L as eval with no human evaluation. In particular, ROUGE-L is also used in the consensus algorithm. There is a risk of over optimizing on that metric.
Decision: Accept-Findings