How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Authors: Yizhong Wang
Hamish Ivison
Pradeep Dasigi
Jack Hessel
Tushar Khot
Khyathi Chandu
David Wadden
Kelsey MacMillan
Noah A. Smith
Iz Beltagy
Hannaneh Hajishirzi
Venue: NeurIPS 2023 Datasets and Benchmarks Spotlight (2023)
Keywords: Instruction tuning
large language models
open-source
systematic evaluation

Abstract: In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with state-of-the-art proprietary models, these claims are often accompanied by limited evaluation, making it difficult to compare models across the board and determine the utility of various resources. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets ranging from manually curated (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, safety, and open-ended instruction following abilities through a collection of automatic, model-based, and human-based metrics. We further introduce Tülu, our best performing instruction-tuned model suite finetuned on a combination of high-quality open resources.

Our experiments show that different instruction-tuning datasets can uncover or enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preference-based evaluations fail to reflect differences in model capabilities exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including a fully finetuned 65B Tülu, along with our code, data, and evaluation framework to facilitate future research.

How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Yizhong Wang" Hamish Ivison"
Pradeep Dasigi*
Jack Hessel* Kelsey MacMillan
Tushar Khot Khyathi Raghavi Chandu
David Wadden
Noah A. Smith
Iz Beltagy
Hannanch Hajishirzi
Allen Institute for AI University of Washington fyizhongu,h hami Sallenaall org
Abstract
In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with saate-of-the-eee proprictary models, these claims are often accompanied by limited evaluation. making it difficult to compare models across the board and determine the utility of various resources. We provide a large of instruction-tune models from 6.7B to 65B parameters in size, trained on 12 instruc- tion datasets ranging from manually curated (e.g., OpenAssistani to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, safety, and open-ended instruction following abilities through collection of automatic, model-based, and numan-based metrics. We further introduce TÚLU & our best performing instruction- tuned model suite finetuned on combination of high-quality open resources.
Our experiments show that different instruction-tuninn datasets uncover enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preferece-bbsed evaluations fail to reflect differences in model capabilities ex exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including fully finetuned 65B TüLU & along with our code, data, and evaluation framework facilitate future research
1 Introduction
The latest generation of large language models has brought unprecedented attention to the potential of language technologies. To support imperative user requests and chat interface, these models often undergo an Distruction-tuniig step which involves training on supervised input/output pairs. Recent instruction tuning corpora often gathered via crowdsourcing (Dolly [[2). Open Assistant (26]) or via distillation from another model (Alpaca [43). Vicuna [8). However, while some public, instruction-tuned models are advertised comparable powerful closed-source proprietary models such as ChatGPT, most experiments that support such claims only cover small set of tasks, and mostly rely model-based evaluation metrics 56] We contend that the evaluation setup should
'Equal contribution. htttpss//gihabb.cc///llllaaaa//s
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

include tasks that test core reasoning and fact-recall skills of the model, in addition to testing model- or human-annoteat annotated generation quality, which may be more open-ended and subjective.
This paper provides a comprehensive evaluation of instruction-tunit resources: specifically, we conduct large number of instruction tuning experiments spanning dozen public corpora, and models ranging in scale from 6.7B to 65B. We evaluate both specific model capabilities (i.c., factual knowledge, reasoning, multilinguality, coding, safety) and open-ended cetruction-followiipi abilities. We report results based on automatic, model-based, and numan-basee evaluation metrics.
Our evaluation reveals that instruction tuning skills, and no one dataset provides the best performance across all evaluations. We also find that the underlying base model is paramount, with better base models (whether it be models trained more tokens or larger models) performing better across the board. Surprisingly, we also find that the best-performing models in model-based evaluation are not the same those that perform best benchmark-bar automatic evaluations, potentially partially due to GPT-4's strong bias toward long, diverse generations.
different datasets appears to promote specific
Building on our findings, we introduce TŪLU R suite of 7B to 65B LLAMA models finetuned on combination of data sources. TÜLU ! 65B is the largest publicly-reeasee fully-instruction tuned LLAMA variant at the time of writing, to the best the authors' knowledge. It is trained on 7 popular available datasets, and yields the best average performance across most model sizes while remaining within 29% of the best best-perfoor cesst-prforming model on each individual task. In summary, our key findings include:
Instruction datasets targeted at specific domains and/or capabilities are extremely effective at improving model performance in those aspects.
Larger or setraanned-for-longeee base models consistently perform better than smaller ones after instruction tuning.
Our model TÜLU Y - fine-tuned LLaMa combination of existing instruction datasets achieves the best average performance across benchmarks, although it not the overall best when considering different evaluation settings independently.
Even a very large (65B) model finetuned on a large mix of instruction datasets fails to outperform ChatGPT, although does perform significantly better than similar smaller models.
Model-based preference evaluation open-ended instruction following correlates strongly with the average number of unique tokens generated by a model, suggesting that model-based preference evaluation has biases that may hide differences in model capabilities.
We open-source the code for training and evaluating these large language models. We also release checkpoints trained on the different instruction datasets and their mixtures, including TŪLU S. We hope this facilitates further development and investigation of open instruction-tun tuned models.
2
Background: Instruction Tuning and Resources
2.1 Instruction Tuning
Instruction tuning, in general, refers to the practice of finetuning pretrained language models to better understand and respond to wide variety of human requests that are expressed in natural language
32 49 35). particular, instruction tuning is concerned with requests that include some indication of the task to be performed within the request itself (e.g., including task instructions in the input prompt). It has arisen as a critical step for generalizing models to new scenarios without dedicated training, and for letting non-experts naturally interact with these models. The training paradigms of instruction tuning can vary from supervised learning using demonstrations [49]
39
48 31 to
reinforcement learning from feedback data 35 B In this work, focus on the supervised learning
setup considering the current open resources for the RL-based approach are still rare, and we leave its exploration for future work.
The success of instruction tuning requires at least two key components: 1) a powerful pretrained language model that has grasped a vast amount of knowledge from web-scale pretraining, and 2) an instruction dataset that is diverse and representative enough to adapt the LM to potential downstream usage. We study these two factors in this work and introduce studied open resources below.

---

Table 1: Instruction datasets investigated in this work. CoT and FLAN V2 are sampled to 100K
to match the sizes of other datasets. We report the average number of conservation turns (N rounds)
average length of prompts (I prompr), average length of completion (E completion)
Datusets
Sourced from
# Instances
N younds
prompp
Lecompletion
SuperNI 48
NI.P datasets Haaman-written Instructions
96,913
1.0
291.1
38.7
Col 50
NLP datasets + Human-written CoTs
100,000
1.0
266.0
53.2
Flan
NLPP datasets + Human-wriin Instructions
100,000
1.0
355.7
31.2
Dolly
12
Hluaan-writeen from scratch
15,011
1.0
118.1
91.3
Open Assistant
26
Human-written from scratch
34,795
1.6
34.8
212.5
Self-instruct
47
Generuted w/ vanilla GPT3 LM
82,439
1.0
41.5
29.3
Unnatural Instructions
26
Generated w/ Davinci-002
68,478
1.0
107.8
23.6
Alpuca
Generuted w/ Duvinci-003
52,002
1.0
27.8
64.6
Code-Alpaca
6
Generated w/ Davinci-003
20,022
1.0
35.6
67.8
GPT4-A pacc
Generated w/ Davinci-003 + GPT4
52,002
1.0
28.0
161.8
Baize [22
Generated w/ ChaiGPT
210,311
17.6
52.8
ShareGiP
prompts + oulputs
models
168,864
3.2
357.8
2.2
Instruction Datasets
We attempt to collect a representative sample of different styles of datasets (listed in Table 1j. iincluding
datascts: (1) created by rescarchers from existing NLP datasets (SuperNI 448). Flan V2 (3]]); (2)
written by humans from scratch for the purpose of instruction tuning (Dolly [2]. Open Assistant
26 ); (3) generated by proprietary models (Self-Instruct [47), Unnatural Instructions [23], Alpaca
Baize [52), GPT4-Alpaca [3]]; (4) comprised of user-shared prompts accompanied by model-
generated completions (ShareGPT [ (5) built for specific skills (CoT [50] for chain-of- of-thougtthhuht,,
Code-Alpaca 6 for code generation). See Appendix for further details.
2.3 Pretrained Models
series of pretrained models ranging in size from 6.7B 65B parameters. We initially experimented with the LLAMA-1 models for the first version of this paper and added LLAMA-2 our camera ready, which use similar numbers of parameters but were trained over significantly more tokens. These models represent the largest, highest-quality pretrained models available to the community (albeit under restric- tive licensing). We also consider OPT and Pythia 4 models with size comparable to the LLAMA 6.77 model, to examine the effect of different base models. For simplicity, we will round the sizes to the nearest integer number. We note several ongoing efforts to are-train similar- or better-qualit, models [18 33 [ We believe our findings should hold for these models and future stronger open base models.
We primarily use the LLAMA suite 44 45
Table 2: Base models that we finetuned in this work.
Base LMs
# Params # Tokens
6.7B
1.0T
13.0B
1.0T
LLaMa 44
32.5B
1.4T
65.2B
6.7B
2.0T
LLaMa-2 45
13.0B
2.0T
OPT
6.7B
180B
Pythia
300B
3 Training Models with Various Datasets
3.1 Unifying the Format
We format all datasets to follow a chatbot-style schema to unify the varied styles and formats of the instruction datasets, shown in Figure I This allows us to fit arbitrary rounds of interactions between the user and the language model (a.k.a. "assistant") into one input sequence and encode them together with causal language model. We add special tokens < <luser| and < assistant > before user utterances and target assistant responses respectively, and an end-of-text marker 
 at the end of each assistant output, which, at inference time, will stop the model's 's response for each round.
ShareGPT (https: pectts://shareee http///shaarppteee com/) data was used to build the Vicuna model but exact dataset has not been released. We instead use a reproduced version from https: http://hhgggnggaccccco co/datasets/ annon231488 (2389123777 23/ShareGGPT Vil..iioa Vicuna, wwwllltteeeeeeeeeeeeeeee low Vicuna split the long conversations into blocks with maximum length of 2048 tokens.
/HTML cleaned. raw ddwp_edttttt and fol-
3

---

3.2 Model Training Details
During training, we compute loss only on to- kens after < </assistant| 1> and before the next < <luser|> |> token. More formally, we consider an instruction dataset consisting of N tuples, each with turns, [xx], X//xxx-2/ -/..... {


Explain the foult-tolerance of the reaction centxol system the Space Bhuttle.
alssiettnt|
the reaction control
(RCS) the Space
shuttle was dessened to fault-tolerant, neaning it wae able to continue functioning even if the 323 consisted of txp sets of ... 6/5>
where x; is a user prompt and y; the desired out- put. For most instances, i and we train the model to output y given x/. However, in the case of conversation datasets, train the model to predict J/ given some conversation history X x,.,.,,,'' We train decoder-only models, and use teacher-foreing with loss masking to train the models, where we mask all tokens be- longing to the input sequence(s) x,. Given as the tokens belonging to the input, and Y as the target tokens, the loss function is:
components failed.
<luser|>
Did the RCS have any ca-orbit failures?
<lassistant|p
There were several iratancea wheze the reaction control system (RCS) on the Space Shutcle experienced failurea nalfunctiona during on-orbit missicns. These </a>
An example from ShareGPT data. We < <rolll> >> set the boundary between messages. The entire sequence is encoded together, and loss computed on the assistant parts (colored blue).
Figure I:
D log poll, 1 1(2)) x {
if EY
L L= -=
otherwise
where 1 is the jth input token (belonging to X or Y). See Appendix $ for further training details.
3.3 TÜLU C: a Better Instruction-Tunee Model by Combining Resources
(and our evaluation below) have shown that increasing the diversity of instructions can effectively improve the performance of instruction tuning. Following this motivation, create mixtures of datasets:
Existing studies 48
31
Human data mixture, which comprises the best authnra datasets, including FLAN V2, CoT, Dolly, and Open Assistant 1 (we exclude SuperNI as FLAN V2 includes most tasks in SuperNI);
Human GPT data mixture, which comprises the human mixture and three additional datasets that have generations by OpenAl GPT models. including GPT4-Alpaca, Code-Alpaca, and ShareGPT.
For both mixtures, concatenate datasets and leave exploring future work. We name LLAMA models trained on the Human+GPT data mixture TÜLU & after hybrid camel resulting from interbreeding between different species. We differentiate the TÜLU models trained from the LLAMA-2 base models by versioning them as TÜLU-1.1.
complex sampling mixtures to
Evaluation Setup
Evaluation of instruction-folloiing models remains a challenging problem due to the enormous scope of "generality" and its open-ended nature. However, we argue that general-purpose models should be able to perform some core tasks before they can generalize to satisfy various practical needs. As such, set up multi-faceted evaluation to cover several key aspects of capabilities covering core abilities and open-ended instruction following. Our evaluations closely follow prior work evaluating instruction- tuned models 9 them together for systematic evaluation.
[6] but serve as the first one to compile
4.1 Facets of Evaluation
Factual knowledge is essential for language models to serve users' information needs. We use the Massive Multitask Language Understanding dataset (MMLU (22] for measuring models' factual knowledge. MMLU consists of set of questions about 57 subjects ranging in difficulty from elementary levels to professional levels, and its multiple-choice format makes suitable for probing models' knowledge without worrying about the open-endedness of generations.

---

Reasoning is another fundamental ability for models, especially for solving complex tasks. We use the test split of Grade School Math dataset (GSM [T) to evaluate models' mathematical reasoning ) which contains 23 challenging tasks from Big-Bench to evaluate models' general reasoning capabilities.
capabilities. We also adopt Big-Bench-Bee (BBH 4
Multilinguality acts as an important perspective of models for serving people from different back- grounds. We use TyDiQA (10), multilingual question answering benchmark covering 11 typo- logically diverse languages for testing how much models can process non-Engish text. We use the gold-passan passaga setup where one passage containing the reference answer is given.
Coding is a particular application that people have used language models for and might be important for integrating these models with external tools [5]. We use the HumanEval dataset ( evaluate the models' capability to generate functionally correct programs from docstrings. To avoid ambiguity with our human evaluation, we call this dataset Codex-Eval in this paper.
Open-ended instruction following. While the performance on the benchmarks above quantifies the models' ability at specific skills, it may not reflect how well the models can handle instructions from real users, which cover highly diverse requests and are often open-ended. For example, the popular ShareGPT dataset contains instances of users asking for programming help, resume formatting tips, educational role-playing, pronunciation suggestion, fanfiction writing. and more. We evaluate such open- endd instructability of models using both model- -ased evaluation ($4.2) and human evaluation ($4.)) both of which consist of multiple test sets from existing studies [47 26 19].
Safety is of particular concern regarding the fast-developing language models to ensure the ethical and proper use of them. Following LLAMA-2 [45) we employ ToxiGen [21] to measure the amount of toxic language and hate speech generation across different groups when the models are prompted to do so. We also adopt TruthfulQA [30] to measure how well models can avoid generating known falsehoods due to misconceptions or false beliefs while providing useful information.
For all the benchmark- based evaluations, we follow their standard metrics, while we subsample some benchmarks to reasonable size to improve the efficiency of doing chain-of- of-thhtgh reasoning. We refer the reader to Appendix S for the setup details.
4.2 Model-Based Evaluation using GPT-4
To evaluate the open-ended instructability, we first adopt model-based approach introduced in
AlpacaEval 227 The test set consists of 805 instructions, with 252 instructions from the Self-Instruct
evaluation 47 188 from the Open Assistant evaluation (26), 129 from the helpful evaluation by
Anthropic 3 80 from the Vicuna evaluation [8] and 156 from the Koala evaluation [19].
We use their simulated GPT-4 annotator, which computes the win rate of the testing model as judged by GPT-4 when compared to the outputs produced by Davinci-003. We use the AlpacaEval bose and prompts [27] to make our scores directly comparable to those on the AlpacaEval leaderboard" f When doing pairwise comparisons with GPT-4, the orders of model outputs are randomized to avoid position bias during evaluation [46]. We do not evaluate vanilla LLAMA models due to them having little fooloong ability without further prompt engineering.
4.3 Human Evaluation
To further test the quality of the open-ended generations, we conduct human evaluation based on 332 instructions that combine the Self-Instruct evaluation set (47) and Vicuna evaluation set [8] Inspired by Bai et al. (3) we design similar interface (Figure 5) for gathering human judgments of model outputs along the following dimensions. We note that we evaluated based on our fine-tuned LLAMA-1 models, as LLAMA-2 was not available at the time of this experiment.
Individual acceptability. We ask human raters to assess whether each system's responses were acceptable in isolation. This is binary decision, and we ask the raters to mark response as acceptable and only if the response answered the request in the query. had no significant errors, and did not have repetitive information.
https:/taatsulaab github.io/alpaa. eval/
5

---

Table Comparison of different instruction tuning datasets, showing that different instruction-tunn tuning datasets can excel in different aspects, and mixtures perform best on average. Cells are blue if the finetuning boosts the vanilla LLAMA performance, and orange the finetuning hurts the performance.
GSM
BBH
TydiQA (multilingualitt)
Codex-Eval
AlpacaEval (open-ended)
(factuality)
(reasoning)
(reasoning)
(coding)
Average
EM (8-soot, CoT)
EM (3-hoot CaT)
F1 (l-shot, GP)
P@10 (0-shot)
% Davinci-003
(0sshol)
Vanilla LLaMa 13B
42.3
14.5
39.3
43.2
28.6
+SuperNI
4.5
20.9
+CoT
44.2
40.0
41.9
47.8
23.7
6.0
33.9
+Flan V2
20.0
40.8
47.2
16.8
3.2
29.8
+Dolly
45.6
18.0
28.4
46.5
31.0
13.7
30.5
+Open Assistant
43.3
15.0
39.6
33.4
31.9
58.1
36.9
Self-instruct
30.4
11.0
30.7
41.3
12.5
5.0
21.8
+Unnatural Instructions
8.0
33.7
40.9
23.9
8.4
26.9
+Alpaca
45.0
9.5
36.6
31.1
29.9
21.9
29.0
+Code-Alpaca
42.5
13.5
35.6
38.9
34.2
13.8
30.1
+GPT4-Alpaca
38.8
+Baize
43.7
10.0
38.7
33.6
28.7
21.9
29.4
+ShareGPT
+Human data mix.
50.2
38.5
39.6
47.0
25.0
35.0
39.2
+ Human+GGPT data mix.
49.3
40.5
43.3
45.6
35.9
56.5
45.2
Table 4: Performance of different base models after training on the Human+GPT data mixture.
MMLU
GSM
BBHl
TydiQA
Codex-Evi Eval (coding)
AlpacaEval (open-ended)
(Guctuality)
(reasoning)
(reasoning)
(multilinguality)
Average
EM (0-shot)
EM (8-shot, CoT)
EM (3-shot, CoT)
FI (I-shot, GP)
P@10 (O-shot)
Win % Davinci-003
Pythia 6.9B
34.8
16.0
29.2
32.8
20.9
23.5
26.2
OPT 6.7B
32.6
13.5
27.9
24.1
8.9
25.9
22.2
LLAMA 7B
44.8
25.0
38.5
43.5
29.1
48.6
38.3
LLAMA-2 7B
37.0
52.8
33,9
57.3
Pairwise preference. We then ask humans to compare the outputs of two systems and select which one they think more helpful. This 5-way decision, and the raters could select if one of the responses is "clearly" or "slightly" better than the other or if it is tie implying that both responses were equally good or bad.
To get a more reliable evaluation, we recruited group of 18 expert annotators who are researchers at AI2 students at UW. All of them are fluent English speakers, holding bachelor's degrees above.
5 Results
5.1 Analysis Instruction Tuning Datasets and Base Models
To understand how the instruction datasets listed in Table Table contrriutt to model abilities, we evaluated LLaMa 13B models trained on these datasets using evaluation suite. Table 3 shows the results our benchmark evaluation set, with more extensive results in App. F We find that:
There is not single best instruction tuning dataset across all tasks. Different datasets enable different capabilities in the model. Noteworthy examples include training on CoT being particularly helpful for mathematical reasoning in GSM and Code-Alpaca being helpful for Codex-Eval. We hypothesize that success on these tasks, which are significantly different from the rest of the evaluation tasks, calls for training sets where these tasks are well-represented. Apart from constructing task- specific datasets manually, distilling task-specific data from large models also appears to be an effective way ensure this (e.g., CodeAlpaca is distilled from Davinci-003).
Combining datasets results in the best overall performance on the benchmark tasks. While models trained on our combination datasets are often not the best model for single task (being the best only in 2 out of evaluation settings), they are the best when measuring average performance across tasks. This suggests that future work into better dataset mixing instruction-tuniig modular

---

Table 5: Performance of TÜLU and other of our trained models to vanilla LLAMA models and the
state-of-the-ar proprietary models across evaluation settings. See Table 8 for complete list.
MMLU (Factuality)
GSM (reasoning)
BBH (reasoning)
TydiQA (multilinguality)
Codex-Eval
AlpacaEval (open-ended)
Average
(coding)
EM (0-shot)
EM
EM
Fl (1-shot, GP)
P@10 (0-shot)
Win % vs
(8-shot, CoT)
(3-hhot, CoT)
Davinci-003
Vanilla LLaMa models 1
LLaMa 7B
31.5
10.0
33.0
38.4
20.5
LLaMa 13B
42.3
14.5
39.3
43.2
28.6
LLaMa 308
54.6
36.0
49.5
55.3
42.8
LLaMa 65B
58.7
50.0
58.1
56.8
46.9
LLaMa-2 7B
41.8
12.0
39.3
51.2
26.8
Maa 13B
52.0
25.0
48.9
56.5
32.5
65B models trained on alternate data mixtures
ShareGPT 65B
61.3 (+2.6)
59.0 (4-9.)
55.8 (-2.3)
31.6 (-25.2)
(4-9.)
73.6
56.3
Human mix. 65B
60.4 (+1.7)
60.0 (+10.0)
54.8 (-3.3)
58.3 (+1.7)
(-.6
43.4
53.6
models trained on our final Human GPT data mixture +
GPT
TULU A 7B
44.8 (+13.3)
25.0 (+1.0))
(+5.5)
43,5
29.1 (+8.6)
48.6
38.3
TCLU A' 13B
49.3 (+7.0)
40.5 (+26.0)
43.3 (+-.00)
45.6 (42.4)
35.9 (+7.3)
56.5
45.2
TCLU 30B
57.7 (+3.1)
53.0 (+17.0)
(42.44)
51.9 (-3.4)
48.0 (45.2)
62.3
54.1
TCLU 65B
59.2 (+0.5)
59.0 (49.0)
54.4 (-3.7)
56.6 (-0.2)
49.4 (+2.5)
61.8
56.7
models trained on our final Hum data mixture using LLAMA-2
TCLU-11 # 7B
49.2 (+7.4)
37.00+25.00 53.0 (+28.0)
44.2 (+4.99
52.8 (+1.6)
33.9
57.3
45.7
TULU-1.1 # 13B
52.3 (+0.3)
50.6 (+1.7)
58.8 (+2.3)
(8.944
64.0
52,9
Proprictary models 1
CharGPT GPT-4
67.9
76.0
66.1
51.9 70.8
88.4
83.6
72.3
82.4
92.5
88.0
94.1
93.5
86.9
models (e.g., mixture-of-experts (40) is promising direction for developing models that retain strong performance across all evaluation settings.
Base model quality extremely important for downstream performance. We examine the impact of using different base models in Table[4 comparing LLAMA, OPT (54), and Pythia 4 models of comparable size trained on the Human+GPT data mix. Across all evaluation settings, we find that using L.LAMA performs best by significant margin, likely due to the fact that LLAAM A is pretrained on significantly more tokens than the other models (see Table 2 This suggests that models pretrained on larger (or potentially higher-quality) corpora are preferable as base models for instruction tuning. The later addition of A- confirms this finding by showing a significant improvement can come from only the base model upgrade.
Some datasets degrade vanilla model performance. Notably, most datasets evaluate cause degradation in performance on GSM and TydiQA over the vanilla base model. We hypothesise this is due data style and quality. Many of the datasets we examine contain little to no examples of chain of offthouggt--ty-e reasoning and contain little to no multilingual data. As such, training on these datasets likely results in some forgetting of the CoT or multilingual abilities previously held by the model, resulting in degraded performance. Additionally, we note that self- self-instruct appears cause degradations across most tasks, which we hypothesise is due to the relatively poor quality of the original self-instruct data, being generated by weaker model (base GPT-3) than the other GPT-distilled datasets.
5.2 Pushing the Limits of Open Models
Having established that (a) using a broad mix of data is best, and (b) using LLAMA the base model is preferable to other open alternatiess we compare the performance of models trained on the Human+GPT data mix (TÜLU models) across all LLAMA sizes in Table We find that:
Instruction tuning brings large benefits on top of LLLAMA models at all sizes. On average, all LLAMA models improve considerably after instruction tuning.
7

---

Smaller models benefit most from instruction tuning, We find that relative improvements from instruction tuning are largest for the smallest models, and shrink as models get larger. Notably, the 65B L.LAMA model performs comparably or better than the 65B TÜLU model on MMLU, BBH, and TydiQA. This suggests that instruction-tuning does not help to enhance strong capabilities already present in the original model, and also highlights that care must be taken during finetuning to avoid forgetting the base model's original capabilities.
TÜLU still lags behind state-of-the-ee proprietary models. Despite the impressive performance of TÜLU 65B, it lags behind ChatGPT and GPT-4 in all evaluation settings, contrary prior claims that models trained on these open resources can match ChatGPT [56,8] 8). We note we cannot discount the possibility that either ChatGPT or GPT-4 was trained on significant portions of our evaluation suite. However, the presence of a significant gap between TÜLU models and ChatGPT matches our findings in the model and human- basad evaluations, which are less likely to be compromised.
5.3
Evaluation of Potential Risks and Harms
We evaluate models on ToxiGen and Truth- fulQA to measure the degree to which different datasets likely yield models that generate toxic language or misinformation. We find that:
ToxiGen (4)
TruthfulQA (t)
Model 1
7B
13B
7B
13B
LLAMA
85.4
82.6
26.2
23.6
SuperNT
85.3
77.3
26.7
26.2
Trends remain similar to poaabllity-fcussee benchmarks. Similarly to the results in Sec. 1 we find that GPT-distilled datasets yield the best overall performance and that there is a large vari- ance in performance across datasets.
+ CoT
63.0
43.9
35.1
35.5
+ Flan V2
77.5
61.4
33.2
33.4
+ Dolly
72.1
78.9
30.1
32.9
Open Assistant
39.2
5.2
40.9
48.6
Self-instruct
89.0
89.3
22.4
22.4
+ Unnatural Inst.
35.8
55.7
27.3
31.7
Models trained on GPT-sourced data yield less toxic generations than GPT. Larger mod- els trained on GPT-distilled data appear to refuse to produce toxic generations almost entirely, de- spite the fact that ChatGPT and GPT-4 produce toxic generations non-trivial amount of the time. We hypothesise this is due to our models overfitting on refusal-styy behaviour, refusing to generate anything moderately toxic, while GPT models balance refusal behaviour with helpful- ness to a greater extent.
+ Alpaca
63.2
58.1
33.5
39.8
+ Code-Alpaca
84.3
92.0
25.1
26.7
GPT4-Alpaca
3.9
1.2
51.2
56.7
+ Baize
77.2
41.2
42.4
43.9
+ GhareGPT
5.5
2.5
45.3
60.0
+ Human mix.
51.8
76,9
34.1
32.1
+ TŪLU A
10.6
0.1
44.6
41.6
ChatGPT GPT-4
27.7
75.2
10.6
82.3
Table 6: Performance of models on ToxiGen (% toxic generations, lower is better) and Truth- fulQA (% truthful and informative answers, higher is better). See Table 9 and Table 10 for the full breakdown of these two evaluations.
TruthfulQA performance does not scale. Un- like other benchmarks, we find that TruthfulQA performance does not improve with model size. Further examining this, we find that larger mod- els do output more correct facts, but also tend to hedge and refuse to give informative answers more often, resulting little to no overall improvements model size increases.
5.4 Model-Based Evaluation Results for Open-Ended Generation
We report the AlpacaEval win-rates of our models in Table ??. We find that:
Models trained mixtures based traditional NLP datasets perform poorly. COT. FLAN, and SuperNI all perform extremely poorly in open-ended instruction following, despite these datasets providing large improvements to the model capabilities tested in Table
Datasets that encourage long, diverse generations perform best. Intrigued by ShareGPT's perfor- mance, we plot the average number of unique tokens in model generations against the AlpacaEval win-rate in Figure 2 We find that the evaluation is strongly correlated with the average number of unique tokens (Pearson correlation of 0.96, < 0.05). Given GPT-4's strong performance on other tasks, we do not believe that GPT-4 evaluation merely counting unique tokens, but this result highlights how model preference scores do not necessarily reward only model capabilities.

---

100
Training Dataset
7B
13B
30B
65B
Win % vs Davinci-003 (GPT-4 Eval)
GPT4
CharGPT
SuperNI
2.9
4.2
CoT
5.0
6.0
75
Flan V2
3.1
3.2
SharcGPP
Dolly
11.0
13.7
GPT4 Alpaco
Humas GPT mix.
Open Assistant
51.4
58.1
50
Divino-003
Open Assidont
Self-instruct
4.0
5.0
Unnatural Instructions
7.5
8.4
Human
Alpaca
21.4
21.9
Alpacs
Mode-Alpace
15.3
15,8
25
Code Alpaca
GPT4-Alpaca
57.3
63.1
Usnatuaa Instractions Dally
Baize
20.0
21.9
Flan V2
SclCinstrac
ShareGPT
62.4
70.5
69.1
73.6
SuperN)
40
80
120
160
Human mix.
28.7
35.0
43.4
Avg. Unique Tokens Response
TÜLU A
48.6
56.5
62.3
61.8
Figure 2: Win-rate scores of 13B models (trained on different datasets) given by GPT-4 strongly correlate with the average numbers of unique to- kens in the model responses (Pearson = 0.96).
Table 7: Win-rate (%) of LLAMA models of vary- ing sizes finetuned on the given dataset against Davinci-003 using AlpacaEval 27
clearty
slightly better
slighily
clearly
90.1%
27.7%
Acceptince Rac
Tälu 65B
7.2%
20.5%
33.1%
19.3%
19.9%
ChatGPT
45.2%
22.8%
Tälu 65B
24.1%
1%
31.9%
11.7% 11.1%
Tülu 7B
50.0%
20.1%
CharGPT Tila
Tälu 65B
26.8%
23.2%
29.8%
12.3% 7.8%
(human mix.)
(Human mix.)
Figure 3: Human acceptance rates for four evaluated models.
Figure 4: Human preference rates for three comparison pairs of models.
ShareGPT performs best. We find that ShareGPT consistently performs best across all model sizes, including models trained on data mixes that include ShareGPT. Models trained on ShareGPT achieve higher win-rates than models over twice their size (c.g., 13B ShareGPT vs 65B TüLU). We hypothesize this due to ShareGPT's diversity, size, and the high average tokens of target responses.
Overall, these results suggest that while model preference evaluation is important, it does not provide holistic evaluation these models. Instead, model preference evaluation should only be included as part of a larger, more comprehensive evaluation setup.
5.5 Human Evaluation Results for Open-Ended Generation
Finally, show the human evaluation results in Figure refer the reader to Appendix G.2 for the inner-annotator agreement. We find that the human evaluation results largely correlate with the AlpacaEval and benchmark-based evaluation: all evaluations show that 65B TÜLU outperforms 7B TÜLU, suggesting making use of larger base models is important, and there still a nontrivial gap performance between 65B TÜLU and ChatGPT. We also find that making use of distilled datasets provides large performance boost, suggesting that numan-authone datasets are lacking in comparison. These observations are also consistent with the acceptability scores Figure However, we note that 7B TÜLU outperforms the human-mix 65B TÜLU in the model preference evaluation, but if we compare the acceptability scores in Figure 3 the opposite appears true. This further evidence that model pairwise evaluation may not always reveal model deficiencies. In this case, the 65B human- mix model is more likely to yield acceptable (if not high-quality) responses than the 7B model.

---

Related Work
Instruction Tuning of LMs Finetuning language models on diverse instruction sets alongside
regular samples has been shown to greatly improve zero-shot performance on unseen tasks [39 S1
49 32
48), and serves as a good base for further finetuning in supervised settings [T. Increasing
the number of diverse prompts [39], the number of tasks [48] [] and diversity of data [66 have all
been shown to be important to performance. More recently, a growing number of models have made use of model-generated instruction
instruetion-augmenter augmented data [47] 23 (25 (53] most often generated or collected
from larger proprictary models such as ChatGPT or GPT-4 8 [3 52 36 inter alia]. Despite the explosion of models and datasets, evaluation remains inconsistent and difficult, with different evaluation setups used across models. Prior work has examined models trained on varying dataset sources with the aim of identifying 'the best mixture' [3 24). but is often limited to examining only benchmark performance, and covers a smaller number of instruction sources than in this work. QLoRA (14] also explores (quantized and parameter-efficient)) instruction-tuning of recent models and datasets, but explores a smaller range of models, datasets, and evaluations than this work.
Evaluation of LMs Given the success of LMs on NLP and intruction-followigg tasks, many evaluation frameworks have been proposed. Frameworks such as HELM (28) and LM Evaluation Harness [17 cover broad range of NLP tasks but are often focused on evaluating the base models as opposed to instruction-tuned ones. Similar to our work, Chung et al. [9] focus on series of benchmark evaluations focused around factuality and reasoning, but largely neglect open-ended instruction following abilities. Releases of large (closed) proprietary models such as GPT-4 [34] and PaLM v2 [2] are often accompanied by comprehensive evaluations over a wide variety of benchmarks, although both similarly neglect evaluation of open-endes instruction following, and without open releases of pretraining or instruction tuning data there is no way to test for evaluation data contamination.
Recently, evaluation frameworks such as AlpacaEval (27] and Chatbot Arena 55 have been proposed to evaluate the open-ended instruction following ability of LMs, moving beyond benchmark-based evaluations. These either make use of other models (in the case of AlpacaEval) or humans (in the case of Chatbot Arena) annotators for judging model generations. We make use of this recent work and evaluate our models on traditional benchmarks. model-based evaluation, and human based evaluation. Coneurrent to this work. Gudibande et al. [20] examine models trained on GPT model outputs and argue that such models learn to mimic only the style, not the content, of their teacher GPT models. While we similarly find that existing datasets fail to train models close strong proprietary models, the diversity of performance we observe across datasets suggests that significant performance improvements can be achieved through imitation data, so long it contains diverse and wide-ranging set of skills and domains.
7 Conclusion
In this work, we provide an extensive evaluation of a wide variety of publicly-available resources for instruction-tuning models, and compare them the strongest proprietary models currently available. We find that using strong base models is vital to performance, combining datasets works best on average (but does result slight performance drops compared to best performance in specific tasks), and our strongest open models do not yet match ChatGPT or GPT-4. Furthermore, we believe that our evaluation highlights the need for the continued development of strong base models and broader, diverse datasets. Finally, we hope that our evaluation and released code and models enable more comprehensive evaluations and spur research to close these gaps and shed insights on all large language models, closed or open.
Acknowledgments
Work at UW was partially supported by the Office of Naval Research under MURI grant N00014 4- 18-1-2670, Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8650- 23-C-7316 and MCS program through NIWC Pacific 1N66001-19-2-4031,, NSF IIS-2044660, and a gift from Apple. We thank colleagues at AI2 and UW NI.P for their constructive feedback and intellectual support. We are particularly grateful to Tim Dettmers for his suggestions efficient
10

---

inference techniques, and Artidoro Pagnnii for providing the reproduced FLAN V2 dataset. We also acknowledge support from AMD and CSC's LUMI cluster, and the Beaker team at A12, which provided the essential computational infrastructure for our experiments. Finally, we are sincerely thankful for the following contributors to our human evaluation: Valentina Pyatkin, Clara Na, Yuling Gu, Yuchen Lin, Haiyan He, David Graham, Hao Peng, Hyunwoo Kim, Alisa Liu, Youngjae Yu, Tal August, and Egor Klevak.
References
[1] E. Almazrouci, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance. Huggingface Model Release, 2023.
URL http:://hugginngface co/tiiiue/falloonn 40b
[2] R. Anil, M. Dai, Firat, M. Johnson, Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 2023.
[3] Y. Bai, A. Jones, K. Ndousse, Askell, A. Chen, N. DasSarma, D. Drain, Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 058622 2022.
[4] S. -Biderman, H. Schoelkopf, Q G. Anthony, H. Bradley, K. O'Brien, Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[5] Cai, X. Wang. T. Ma, X. Chen, and Zhou. Large language models as tool makers, arXiv preprint arXiv:2305. 17126, 2023.
[6] Chaudhary. Code alpaca: An inwlountionnffllooiingg Ilama model for code generation. GitHub repository, 2023. URL https::/githuu......../ com sahi 2aaahh1100011441eeapppsss.
[7] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto. J. Kaplan, H. Edwards, Y. Burda, N. Joseeh,, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2/07.03374, 2021.
[8] W.-L. Chiang, Li, Z. Lin, Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt- 4 with 90%* chatgpt quality. Blog post, March 2023. URL https::/ org/blogg 2023-03- vicuna/
[9] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling instruction-finetune language models. arXiv preprint arXiv:2210.111 114111,,, 2022.
[10] J. H. Clark, E. Choi, M. Collins, D. Garrette, T. Kwiatkowski, V. Nikolaev, and J. Palomaki. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. TACL, 2020. URL htttttttt///axiiii org/ abs 2003, 05002
[11) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[12] Databricks. Free dolly: Introducing the world's first truly open instruction-tuned 11m. Blog post, 2023. URL https://ww..caaabbicc....co
www.bricatscccc... cww comm/blo///0022////122 open htgps//w........aaaaaaaaaaaaaaaa
[13] T. Dettmers, M. Lewis, Y. Belkada, and I.. Zettlemoyer. LLMMint80: 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022.
[14] Dettmers, Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXivv2305.14314 2023.

---

[15] N. Ding, Y. Chen, B. Xu. S. Hu, Y. Qin, Z. Liu, M. Sun, and B. Zhou. Ultrachat: A large- scale auto gelteatedeeted multi-round dialogue data. GitHub Repository, 2023. URL https 7/gi com/ thunlp/ultrachat
[16] Y. Dubois, X. Li, R. Taori, T. Zhang. I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 14387, 2023.
[17] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite. B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, Sept. 2021. /tttps:::///oi /rgg///////...///00
5281/zenodo 5371628
[18] X. Geng and H. Liu. Openllama: An open reproduction of Ilama. GitHub Repository, 2023. UtLhttss://ggtti com/ http://pennnneeemeeeeo hppe//pennn___ 11aam
[19] X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URLhttpps: h/bp::://bbirrr berkeley
[20] A. Gudibande, E. Wallace, C. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine, and D. Song. The false promise of imitating proprietary Ilms. arXiv preprint arXiv:2305.15717, 2023.
[21] T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar. TOXIGEN: Controlling Language Models to Generate Implied and Adversarial Toxicity. In ACL, 2022. URL https
/aaxxiv.org www.ooggggaba..ee http//bb//220.... 09509
[22] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding, In International Conference on Learning Representations (ICLR), 2020.
[23] O. Honovich, T. Scialom, O. Levy, and T. Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09653 2022.
[24] S. Jyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang. Q. Liu, P. S. Koura, al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12011 2022.
[25] A. Köksal, T. Schick, A. Korhonen, and H. Schützze Longform: Optimizing instruction tuning for long text generation with corpus extraction. arXiv preprint arXiv:2304.08440 2023.
[26] A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, et al. Openassistant conversations -emoorattzzing large language model alignment. arXiv preprint arXiv:2304.07327, 2023.
[27] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Guirajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of mnstructtionfoolooinng models. Github repository, 2023. URLhttps::/
thupub...// thub. cwwm ttttttttsuuu-aa lab/ alpaca_evalllll
[28] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. R e, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao,
J. Wang. K. Santhanam, L. J. OFr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. S. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson. Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. F. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 2022.
[29] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human false- hoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 3214-3252, Dublin, Ireland, May 2022. Asso- ciation for Computational Linguistics. doi: 10. 10.18653//1/2022..oo acl-oonn...n-.nnn... long.229. 229. URL https
//aclanthology org/ 2022.022222222 acl acl-long. 229
12

---

[30] S. Lin. J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, 2022.
[31] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph. J. Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 3688, 2023.
[32] Mishra, D. Khashabi, C. Baral, and H. Hajishirzi. Cross-Tast Generalization via Natural Language Crowdsourcing Instructions. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
[33] MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms. Blog post, 2023. URL https: //www/wwww..// mosai cml cot.///oo//////oottto....
[34] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08877, 2023.
[35] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, et al. Training Language Models to Follow Instructions with Human Feedback.
In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[36] B. Peng. C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.0m 03277, 2023.
[37] S. Rajbhandari,, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '00 IEEE Press, 2020. ISBN 9781 7281 728119998.
[38] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[39] V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak. D. Datta, Chang, M. T.-J. Jiang, H. Wang, M. Manica, Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Techan, L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations (ICLR), 2022.
[40] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
[41] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 2.00:2206600615, 2022.
[42] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arxiv:2200.09261.. 2022.
[43] R. Taori, Gulrajani, Zhang, Y. Dubois, Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instraction-following llama model. GitHub repository. 2023. URL httpss 7/github.
alpaca
[44] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, R. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2023.
[45] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint XxXi:23377.09288 2023.
13

---

[46] P. Wang, L. Li, L. Chen, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 2023.
[47] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.1053 10560, 2022.
[48] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, al. Superr-aturalllnstrcciions Generalization via Declarative Instructions on 1600+ Tasks. In EMNLP, 2022.
[49] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned Language Models Zero-Shot Learners. International Conference on Learning Representations (ICLR), 2022.
[50] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201. /1900, 2022.
[51] O. Weller, N. Lourie, M. Gardner, and M. E. Peters. Learning from task descriptions. Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1361-1375, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.emnlp-mait 105. htt.ttttps///claatttiie
org/ 2020. ommlp-main. 105
[52] C. Xu, D. Guo, N. Duan, and J. McAuley. Baize: An open-source chat model with parameter- efficient tuning on self-chat data. arXiv preprini arXiv:2304.01196 2023.
[53] C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. WizardIm: Empow- ering large language models follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
[54] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106: 2022.
[55] L. Zhengg Y. Sheng, W.-L. Chiang, H. Zhang, J. E. Gonzalez, and I. Stoica. Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings. Blog post, May 2023. URL https
//1msys orggrgggggo wttgg//lg/////////// 2023 05- 00-5-003-rmm arena/
[56] C. Zhou, Liu, Xu, Iyer, J. Sun, Y. Mao, X. Ma. A. Efrat, Yu, L. Yu, al. Lima: Less is more for alignment. arXiv preprint 2tth:2335.11200 2023.
14

---

Supplementary Material
A Limitations
Despite the comprchensiveness of our evaluations, we note that we did not exhaustively cover all possible evaluations: for example, we do not explicitly evaluate models on their multi-turn dialogue abilities their summarization abilities. Instead, focus important, and cover broad open-ended tasks via our model and human preference based evaluations.
set of capabilities believe
We also note that we do not cover all possible instruction datasets and open models released re- cently, due to the computational cost of doing this. Instead, we focus on wide set of datasets we believe are broadly representative of the type of open instruction datasets available skill-targeted, GPT-distilled, etc), and focused on the strongest base model widely available when performing experiments. Future work could investigate whether more recent strong base models (e.g.. the Falcon model ([), or other instruction datasets, perform significantly better or differently from the models explored in this work.
Finally, we note that open-ended instruction- based evaluation is highly subjective and difficult due to its extremely open-ended nature. There is likely no one answer that is definitively the best for any given query, and different annotators (whether they be human or model) will have different biases and preferences. We also note that in the case of model-based evaluations, we primarily compare our model outputs to Davinci-003 generations, which may result in overly rewarding models that avoid shortcomings of Davinci-003, or not properly rewarding models that share strengths with Davinci-003.
Despite not being completely exhaustive in this work, we believe that by covering a broad range of models, it still serves as useful and important contribution in showing what type of open resources work, and where future community efforts should go (better base models, more diverse instruction- tuning datasets).
B Broader Impact
We believe that rigorous evaluation of existing resources is broadly positive, exposing the strengths and deficiencies of currently avdell-available resources. Furthermore, as all resources used are widely available, the harm posed by training these models is fairly small. We do note that training and releasing especially large instruction settettioo-tneedddeeeeeee models without well-tested guides carries a degree of risk, and such initially release largest models with gated setup (requiring users apply for access and be manually approved) to limit potential harms.
C Instruction Datasets Details
We provide a brief description of all the instruction datasets used (and licenses) below:
SuperNI: A collection of diverse NLP tasks with instructions, created by Wang et al. 48 The dataset uses the Apache-2.0 license.
CoT: A collection of datasets annotated with chain thain-of-thoughts [50). We use the CoT mixture from the FLAN v2 collection 99, splitting out as a separate dataset. The FLAN mixture is released under the Apache-2.0 license, although the component datasets may not use this license.
Flan V2: collection of NL.P tasks that combines a number existing NLP datasets with various data augmentations, introduced by Chung et al. [ The mixture is released under the Apache-2.0 license, although the component datasets may not use this license.
Dolly: A collection of instruction mnslruction-follooingg samples created by Databricks employees [12] The dataset is released under the Creative Commons Attribution-ShareAlli ShareAlike 3.0 Unported License.
Open Assistant A crowdsourced human annotated assistant-style conversation corpus, consist-
ing of a large number of sample conversations in a wide variety of languages [26]. The dataset is released under the Apache 2.0 license.
Self-Instruct A dataset of instruction-follont following samples created by prompting GPT-3 to create new samples given some example instances 47 The dataset released under the Apache-2.0 license.
15

---

Unnatural Instructions: A dataset of instruction instruction-folloinng samples created by prompting Davinci- 002 using the method introduced by Honovich et license.
The dataset is released under the MIT
Alpaca: A dataset created using self ielffinsttuut-stylee method with Davinci- as the generation model and some over improvements over self-instruct [43]. The dataset is released under
4.0 International (CC BY-NC 4.0) license.
Code-Alpaca: A dataset created using the Alpaca method, but focussing on code generation 6 The dataset is released under the Apache-2.. 2.0 license.
GPT-4 Alpaca: A dataset created using the Alpaca dataset as inputs, but replacing the example generations with generations from GPT-4 [36]. We include this to see the effect of using a better quality generation model. The dataset is released under the Apache-2.0 license.
Baize: A dataset created by prompt ChatGPT and letring it converse with itself [52]. The dataset is released under the GNU General Public License v3.0.
ShareGPT: collection of user interactions with various chat systems publicly shared. We use the 'himl-cleaned' variant available at https://huggingff.a co co htt.://aecoooottaaaaaaaaoooonnaa
ShareGPT_ wwwunacccaauuaaaee unf iltered/ cree/main, HWTHL__Ieeeeaaaa_e raw.eeddddwww_aeeee raw__ataset__ttt_eeaaeaa____a__ We then split long conversations (over 2048 tokens) into max- 2048 token chunks, following the Vicuna setup 8 We do not do any further filtering of samples. This dataset is released under the Apache-2.0 license.
We note that the SuperNI and CoT datasets are included the FLAN V2 collection but only account for small portion of our subsampled FLAN V2 dataset.
We also note that we broadly use popular already publicly available instruction-tuning datasets, and in the case of human- authored datasets, largely use datasets created explicitly (with participant knowledge) for the purpose of training models (e.g., Dolly, Open Assistant 1). As instruction-tuning data, most data is not likely to contain personally identifying details, although we note that we did not make an effort to remove offensive content, so our models may produce toxic or harmful generations.
D Model Training Details and Compute
We train all models for two epochs with a learning rate of 2e (1e-5 for 30B and 65B models), with no weight decay and a learning rate with linear decay and linear warmup for 3% of the total training steps. We a maximum sequence length of 2048 (1024 for 30B and 65B), truncating samples where necessary. During training. we make use of the DeepSpeed library [38] and ZeRO optimizer (37] allow for large-scale model finetuning. In all cases, we fully finetune models. We trained models primarily on the CSC LUMI GPU cluster, each node on which contains 4 AMD MI250x GPUs.
E Evaluation Setups
We provide further details on the evaluation setups used below. We also note that we release evaluation code along with our training code to allow casy reproduction.
We use the official MMLU evaluation script and prompts available at httts:://ttthtt com/henndyccks/tee ttesst with modifications to allow for batch processing. We evaluate using and 5 few-shot examples, following the original setup of MMLU.
MMLU:
GSM: We evaluate models on the test set of GSM. Following Wei et al. 50 we evaluate with and without chain-of -fhhoughttt (CoT vs Direct). Both settings use few-shot in-context examples (in the chain-of-thougl setting, the few-shot examples are accompanied by chain-of-thoughts. Because all answers in GSM are numbers, we extract the last number the model response as the final answer. To allow for faster evaluation, randomly sampled 200 examples from the 1319 testing examples, which we find gives similar performance as the full-set set evaluation.
BBH: We follow the setup described in the original paper Suzgun et al. [42], and evaluate with and without chain-of ofain-oo-thoon thougttt (CoT vs Direct). The officially provided prompts, which have 3 few-shot in-context examples are used for both CoT and Direct setups. For the CoT setup, we extract the first word after the phrase 'So the answer is', the entire response if there is such substring present.
16

---

TydiQA We follow the setup described in the PaLM 2 technical report [2 to evaluate models' performance answering multilingual questions under two settings: when the gold passage that contains the answer is given (GoldP/GP); 2) when there is no context given
One in-context example is used to familiarize the model with the answering format.
Codex-Eval We the HumanEval dataset in the Codex paper 7 for evaluating models' coding ability. The dataset contains 164 programming problems, where models are prompted to complete the Python function given its docstring. Following the original paper, we compute unbiased estimates of pass @k to measure the functional correctness of models' outputs. We report both pass and pass @ 10. The pass@1 results were obtained by sampling with a temperature of 0.1 and the pass@ @ 10 results with a temperature of 0.8.
ToxiGen We follow the setup in Touvron et al. [45], but use the original set of prompts from Hartvigsen et al. [21], which are designed to elicit toxic generations for certain groups. We take only the prompts designed to produce toxic language ("hateful' prompts) and use 500 prompts per group reduce evaluation costs. For base language models, we pass in the original ToxiGen prompts unchanged and greedily decode up to the first new line (or a maximum of 512 tokens), For instruction-tuned models, we place the prompt in the corresponding template, and ask the model to complete the prompt, until the model generates a stop token (or a maximum of 512 tokens). We pass the generated text into roberta-large model trained to detect toxic content finetuned as part of Hartvigsen et al. 21 We then report the percentage of generations deemed toxic by the classifier.
TruthfulQA Following Touvron et al. 455 we mainly use the generation setting of TrutufulQA 30 The TrutufulQA dataset contains 818 questions, which are used to prompt the tested model to generate answers. We use the default QA prompt format with 6 in-context QA examples. We follow the official script in their official implemention [ do greedy decoding and answer postprocessing. We also follow their instruction to train two GPT-based classifiers for judging the truthfulness and informativeness of the model response. We report the rate of the responses being truthful (% Trutuful), informative (% Informative), and both (% Informative and Truthful) as our metrics. Following Touvron al. [45], we only report the (% Informative and Truthful as our primary metric in the main paper.
Aval We use the package provided by Li et al. (27), following the default setup which asks the evaluated model generate responses for 805 prompts and employ GPT-4 to compare the response with Davinci-003. We employ the "alpaca_ _ 14" annotator config instead of "alpaca_eve eval gpt4" to make the results reproducible. We allow the evaluated model to generate up to 8192 tokens, without specifying special stop sequences. The reported win-rate is the percentage of model generations that GPT-4 reports as being preferred over the generations from Davinci-003.
For all the evaluations, we load models using the 8-bit mode [ provided in the Huggingface Transformers library, which we find speeds up the inference significantly and has negligible impact on the final performance. When doing generation, use greedy decoding and max length of 512 tokens, unless otherwise specified.
F Overview of All Automatic Evaluation Results
Table Siole/Apressee presents a compilation of the results of all models trained as part of this work on all the core capability evaluation benchmarks. We list multiple scenarios for all evaluation settings except AlpacaEval, which has one setting. Please refer to $F for the meanings of the reported metrics. We also calculate an average across benchmarks in Table g This is calculated by first calculating per-benchmark average by taking the average across scenarios. We then compute the overall average with each benchmark weighted equally.
Additionally, for safety evaluation, we provide ToxiGen results broken down by group targeted in far all models, from which we can see some groups are specially targeted, even after instruction tuning. We all provide full TruthfulQA results in Table I0] The results are broken down into % informative and % truthful see Lin et al. [29 for details on these metrics.
httpp://htuggiiiiacccccooommmsms sooberteriiiiierrrraaaeaaa httpps//gghhubb..oooyyyii////i/t
17

---

Table 8: An overview of the performance of all models finetuned for this work, along with proprictary models, on selected benchmarks. To calculate the average, we calculate the average per benchmark and then take the average across these. See App. for more details.
MMLU
GSM
BBH
TydiQA
Codex-Eval
AlpacaEval
Average
0-shot
5-shot Direct
CoT Direct
CoT
GP
CB P@1 P@ 10
Davinci-003
Proprietary models 4
GPT-4 ChatGPT
82.4
83.9
35.0
92.5
50.9
88.0
70.8
27.6
85.7
94.1
93.5
74.8
67.9
69.9
32,5
76.0
49.0
66.1
51.9
20.0
72.2
88.4
83.6
63.4
LLaMa 65B finetuning experiments
Vanilla LLaMa
58.7
63.3
14.0
58.1
18.1
23.5
46.9
ShareGPT
61.3
62.8
23,0
59.0
40.0
55.8
31.6
9.8
30,8
56.2
73.6
48.1
Human mix.
60.4
61.4
8.5
60.0
53.1
54.8
58.3
15.9
23.9
44.6
43.4
44.0
H+GPT mix
59.2
60.8
10.0
59.0
48.4
54.4
56.6
13.3
29.2
49.4
61.8
47.0
LLaMa 30B finetuning experiments
Vanilla LLaMa
54.6
57.9
12.0
36.0
41.4
49.5
55.3
15.8
22.0
42.8
ShareGPT
54.6
57.5
20.5
47.5
42.2
10.7
28.1
49.8
69.1
44.6
Human mix.
56.5
58.8
5.5
52.0
46.8
50.6
57.5
14.5
24.8
41.3
38.3
40.4
H+GPT mix (+)
57.7
58.4
6.0
53.0
47.1
51.9
$1.9
13.0
27.2
48.9
62.3
44.9
LLaMa 13B finetuning experiments
Vanilla LLaMa
42.3
46.4
7.0
14.5
37.1
39.3
43.2
11.5
16.2
28.6
SuperNI
49.7
50.3
9.6
8.2
12.9
4,2
C6T
44.2
45.2
12.5
40.0
38.7
47.8
9.1
12.8
23.7
6.0
27.3
Flan V2
50.6
51.2
3.0
20.0
41.7
40.8
47.2
11.4
9.0
16.8
3.2
24.8
Dolly
45.6
45.1
7.0
18.0
32.3
28.4
46.5
11.6
12.9
31.0
13.7
25.5
Open Assistant
43.3
36.7
5.0
15.0
35.9
39.6
33.4
10.3
16.1
31.9
58.1
32.0
Self-instruct
30.4
32.1
4.5
11.0
33.2
30.7
41.3
8.5
8.7
12.5
5.0
18.6
Unnat. Instruct.
46.4
45.7
5.5
8.0
37.9
33.7
41.0
8.5
14.4
23.9
8,4
23.5
Alpaca
45.0
46.9
7.0
36.0
36.6
31.1
7.9
14.6
29.9
21.9
25.7
Code-Alpaca
42.5
44.3
4.5
13.5
35.9
35.6
10.2
21.3
34.2
15.8
26.0
GPT4-Alpaca
46.9
47.1
9.0
16.5
38.2
38.8
23.5
6.2
15.1
36.6
63.1
33.7
Baize
43.7
41.6
5.0
10.0
37.2
38.7
33.6
7.2
15.1
28.7
21.9
25.4
ShareGPT
49.3
47.7
6.0
27.0
23.1
40.4
30.5
7.1
16.1
34.1
70.5
35.2
Human mix.
50.2
51.2
6.0
38.5
43.9
39.6
47.0
8.8
11.9
25.0
35.0
32.7
H+GPT mix (+))
49.3
51.9
4.5
40.5
40.7
43.3
45.6
9.2
21.2
35.9
56.5
37.9
LLaMa-22 13B
finetuning experiments
Vanilla LLaMa-2
$2.0
55.5
10.0
25.0
41.8
48.9
56.5
17.2
18.1
32.5
H+GPT mix (+)
52.3
54.6
5.0
53.0
44.1
50.6
58.8
15.7
23.5
38.9
64.0
43.7
LLaMa 7B finetuning experiments
Vanilla LLaMa
31.5
33.8
5.0
10.0
32.2
33.0
38.4
9.0
11.0
20.5
SuperNI
44.1
43.5
3.0
4.5
37.4
43.4
7.5
7.0
12.1
2.9
17.6
CoT
41.8
42.2
6.5
27.5
36.2
33.9
36.3
5.6
8.8
15.7
5.0
22.0
Flan V2
45.4
46.9
3.5
13.0
34.4
36.0
38.5
9.0
9.8
12.9
3.1
21.3
Dolly
38.1
35.0
4.5
5.5
28.3
23.8
39.8
9.7
11.4
22.5
10.9
20.1
Open Assistant
33.0
30.2
6.0
10.0
21.5
31.8
26.8
6.8
10.4
21.7
51.4
25.1
Self-instruct
35.6
32.7
3.5
7.0
31.5
29.4
34.5
7.1
6.2
11.8
4.0
17.3
Unnat. Instruct.
43.1
37.8
6.9
9.2
16.8
7.5
20.2
Alpaca
41.6
40.0
7.0
7.5
34.1
31.2
7.3
10.4
21.7
21.4
22.7
Code-Alpaca
34.3
33.7
6.5
7.0
31.1
30.6
35.8
9.5
16.6
28.2
15.3
22.0
GPT4-Alpaca
42.2
37.4
6.5
10.5
30.9
32.3
20.6
4.9
13.2
26.2
57.3
28.3
Baize
40.5
38.1
4.0
6.5
31.3
34.0
29.1
6.8
11.5
26.5
20.0
22.4
ShareGPT
44.5
39.5
6.0
9.5
9.7
34.1
22.8
7.2
12.3
21.2
62.4
27.6
Human mix
46.2
48.2
4.5
25.5
38.8
35.6
43.2
8.0
9.5
20.2
28.7
28.1
H+GPT mix
44.8
47.1
7.0
25.0
38.5
43.5
8.0
18.6
29.1
48.6
33.1
LLaMa-2 7B finetuning experiments.
Vanilla LLaMa-2
41.8
46.1
8.0
12.0
32.2
39.3
51.2
15.1
13.3
26,8
H++PT mix (
49.2
50.5
6.5
37.0
38.6
44.2
52.8
11.9
20.4
33.9
$7.3
38.3
Non-L.LaMe 7B finetuning experiments
OPT
25.0
24.6
7.0
3.0
4.2
0.6
0.9
+H+GPT mix
32.6
33.7
3.0
13.5
30.6
27.9
24.1
3.6
5.2
8.9
25.9
19.6
Pythia 6.9B
25.8
26.2
4.5
3.5
0.0
28.1
25.6
3.6
7.5
13.7
+II+GPT mix
34.8
35.0
4.0
16.0
31.7
29.2
32.8
2.8
14,9
20.9
23.5
22.4
18

---

Table ToxiGen results across models. We report the percentage generations deemed toxic by separate classifier, broken down by the group the prompt is designed to produce toxic generations
about.
Asian Black Chinese Jewish Latino LGBTQ Ment. Mexican Mid. East Muslim Nat. Amer. Plys.
Trans. Women Overall
Progrictary models
GPT44 ChaiGPT
6.0
0.0
31.0
0.0
4,0
4.0
14.0
4,0
9.0
5.0
0.0
8.0
3.0
61.0
10.6
2,0
16.0
33.0
2.0
11.0
27.0
40.0
17.0
16.0
35.0
0.0
78.0
27.0
$4.0
27.7
LLaMa
66.4
99.4
66.2
97.0
86.8
$3.6
96.0
90.5
96.0
92.2
100.0
78.6
61.2
78.6
85.4
SharGGPT
0.0
03.0
0.0
0.0
0.00
0.2
2.0
0.0
0.0
0.2
0.9
4.2
1.0
0.8
mix.
HPG mix (8))
0.0
0.0
9.2
0.0
0.0
9.0
25.0
4.6
1.8
0.9
18.8
9.6
26.2
7.7
LLaMa 30B models
ShurGGPT
0.0
0.0
0.0
0.0
0.0
0.2
1.2
0.0
0.0
0.0
0.0
0.0
0.4
Human mix.
17.8
45.0
21.0
32.0
72.4
22.0
68.0
72.4
15.6
3.2
12.4
26.4
32.8
41
34.5
H+GPT (8))
LLaMM 13B models
LLaMa
39.2
90.6
$1.6
85.8
64.6
76.5
98.8
89.0
97.0
97.0
100.0
90.0
67.8
78.5
82.6
SuperNI
07.22
88.8
87.2
74.6
39.6
78.2
79.2
58.8
Flan V2
39.8
70.6
39.4
46.D
81.8
$90.5
89
55.8
55.2
33.2
85.8
56.6
76.0
70.6
61AA
Doily
99.6
79.8
87.2
93.0
T00LO
87.0
53.8
96.2
68.8
60.4
97.2
50.0
73.2
57.8
78.9
Open
Self-Instrust
98.4
9996
57.8
95.2
89.8
86.6
V7.4
96.0
76.8
100.0
80.0
V7.8
89.3
Unsat Instruct.
37.6
82.2
55.4
97.4
24.0
38.0
74.8
67.2
40.8
26.0
71.6
47.4
57.0
$7.8
$5.7
Alpaca
86.8
39:0
94.2
56.2
76.0
61.6
30.2
73.0
$9.0
50.2
13.2
$6.0
46.2
71.4
58.1
Code-Alpaca
0.4
0.0
0.2
0.0
3.8
4.5
1.5
1.4
0.0
0.0
0.0
0.4
3.4
1.0
1.2
Baize
46.2
12:2
$3.4
6.6
58.2
47.4
$2.6
10.4
20.8
34.2
44.8
47.6
32.2
80.2
41.2
ShareGPT
0.0
0.0
5,4
0.0
0.0
3.2
5.4
2.6
0.0
6.2
9.4
mix.
70.8
49.8
98.6
80.8
H+GPT mix (f1)
0.0
0.0
0.0
0.0
0.0
0.6
0.0
0.0
0.0
0.0
0.0
0.0
1.2
0.0
0.1
LLaMa-2 13B modelss
LLaMa-2
58.8
89.6
97.8
81.6
71.0
96.4
93.2
92.6
91.4
100.0
91.0
63.8
84.0
H+GPT mix (P)
0.0
16.4
3.8
3.8
44.6
22.8
23.0
39.4
5.8
9.0
49.6
14.8
6.4
22.8
18.7
I.LaMa models
LLMMa
43.6
94.8
85.4
91.2
96.6
75,4
98.8
91.2
95.0
89.8
100.0
92.8
63.6
77.0
85.4
SuperNI
99.4
98.2
91.8
89.8
92.4
77.0
65.4
93.8
85.0
87.6
87.2
75.8
80.2
70.0
85.3
CoT
77.4
89.0
58.2
55.8
87.8
51.4
68.8
68.2
60.8
57.6
53.8
46.8
43.0
64.0
63.0
77.6
Dolly
90.2
90.6
83.8
98.8
94.0
$2.4
66.6
93.0
$6.0
41.2
1.2
55,8
68.2
83.0
72.1
Open Assistant
8.0
17.6
53.8
95.2
12.2
40.8
33.6
55.6
27.2
22.6
35.4
45.0
29.2
72.0
39.2
Self-Instrreet
100.0
94.8
73.4
88.4
88.0
75.4
95.8
76.4
86.8
99.4
Alpaca
97.0
40%
97.2
79.8
51.4
69.6
4%.2
67.6
$4.0
$7.2
37.4
57.4
45.4
81.2
63.2
Alpaca
98.6
90.2
99.2
100.0
91.6
88.8
60.8
99.4
$3.0
69.8
66.8
79.6
72.8
90.0
84.3
GPT4-Alpaca
Baize
99.8
$7.88
81.6
78.6
68.6
66.6
97.6
SharoGPT
0.0
0.0
12.0
0.0
0.8
5-1
1.0
0.1
0.6
3.6
0.4
21.6
5.6
26.0
5.5
Hummn mix.
20-4
74.6
54.4
61.6
53.4
4014
63.0
68.0
55.2
44.6
50.4
38.8
24.4
76.0
51.8
H+OPT (9/)
1.1 1.MaMa-2 138 modelss
LLaMa-2
51.0
96.8
86.8
28.4
32.6
78.6
95.4
93.8
88.6
94.1
90.1
85.2
68.6
77.3
(8)
59.0
Noe-L. Mon-LLaMa 7B mookls
OPT
52.8
96.6
74.8
85.6
77.6
71.6
97.6
94,8
91.4
97.6
93.6
68.8
67.2
83.3
75.6
Pyrhia
82.2
99.6
70.6
75.0
85.6
65.8
97.6
93.8
94.2
84.4
98.6
88.4
67.2
54.2
82.7
HGGPT mix
37.4
72.4
94.6
58.4
$4.6
36.8
78.8
47.2
$5.4
43.8
39.4
68.4
37.2
72.4
56.9
19

---

Table 10: TruthfulQA results across models. We report percentage of answers that are informative, or truthful, or both.
Informative
Truthful
% Informative Truthful
Proprictary models
GPT4 CharGPT
99.5
82.7
82.3
96.0
79.2
75.2
I.LaMa 65B models
Vanilla LLaMa
85.8
45.2
ShareCPT
86.8
76.6
63,5
Human mix
98.0
42.2
40.4
(
LLaMa 30B models
Vanilla LLaMa
92.0
35.7
28.3
ShareGPT
71.0
81.4
52.5
Human mix
98.2
43.2
41.5
H+GPT mix (9)
92.8
53.2
46.0
LLaMa 13B models
Vanilla LLaMa
95.1
30.8
26.2
SuperNI
96.8
27.8
Col
92.7
41.6
35,5
Flan V2
91.2
42.1
Dolly
98.8
34.1
32,9
Open Assistant
91.3
57.2
48.6
ShareGPT
Self-instruct
93.4
28.8
22.4
Unnat. Instruct.
84.6
46.9
31.7
Alpaca
99.9
39.9
39.8
Code-Alpaca
98.9
27.5
26.7
APPa-Alpaca
Baize
87.9
56.1
43,9
Human mix.
98.4
33.3
32.1
H+GPT mix (A))
94.6
47.0
LLaMa-2 13B models
Vanilla LLaMa 2 H+GPT mix (+)
99.0
32.1
31.1
96.7
48.3
45.3
LLaMa 7B models J
Vanilla
96.7
SuperNI
98.0
28.4
26.7
Cal
93.5
40.3
35.1
Flan V2
96.1
36.1
33.2
Dolly
98.5
31.5
30.1
Open Assistant
92.0
ShareGPT
76.4
68.5
45.3
Self-instruct
96.5
25.5
22.4
Unnat. Instruct.
89.8
37.0
Alpaca
98.8
34.8
33.5
Code-Alpaca
99.1
25.9
GPT4-A
84.2
66.7
51.2
Baize
88.5
53.7
42.4
Human mix
97.7
36.2
H+GPT mix (A)
98.2
46.3
44.6
[.I 1M-22 7B models 1
Vanilla LLaMa
93.0
33.4
26.7
H+GPT mix (f)
20

---

G
Human Evaluation Details
G.1
Setup
Here we provide more details for the human evaluation described in $4.3 Our evaluation contains 332 instructions, including 252 instructions from the Self-Instruct evaluation set 477 and 80 instructions from the Vicuna evaluation 65B vs ChatGPT, 2) TüLU 65B vs TüLU 7B, 3) TÜLU 65B v.s. 65B LLAMA model trained on the Human data mixture, using the same of instructions for all three comparisons.
[ Our evaluation conducted for three pairs of models: 1) TŪLU
To ensure reliable evaluation, we recruited 18 expert annotators, which are researchers at A12 or students UW, for the annotation. All these annotators are fluent English speakers and hold bachelor's degrees or above.
We design website, shown in Figure for our annotators to conduct the evaluation, and will release the code for this website. When doing the evaluation, annotators are instructed read carefully the prompt and outputs A and B from two models, and then answer three questions asking for the acceptance of the outputs and their comparison in terms helpfulness. They are encouraged to use Google or any external tools that can help with the judgment. The model information is anonymized, and their outputs are put in random order.
Human Evaluation
Design queelizner (Programiing Language)
You ore kogeed va adeiss Losood
Instance
Hew susporase from two recilds.
you incance intarnating
unample mraki. enample
inceoling
Sure, in semple skill aosearmnnt quenorrmers
programminn lgg
b. Wibb divelepment
Data anailytt and statirical computtnng
used ker?
Wh the diference belecen a veotor and motix in
Mobile app dewelppment
How de you create function in 69?
Carna dovelopment
handie "apply" funations
a
vector name
element))
de you create acaterplatin
voctor, jnae sjolementt, elementz cements)
What is the pu purpose the "aply" package
vector, rame " (Wle meeemett eloreent elementt
Prevdo fredouc
Hon do you hands luddraa R?
d. voclor Jume jdamertt, clament2, dismand))
What the diference belween a "for toop and "witter toop
F2?
What the kanston used road CSV
Unw
rarate Enaee censensinn modee
ovoiuaate Transtato).
aesed your knowledge,
extemal
O1: le outoul acces bedle reppomanl acoeciedle al wrover cour requande tonw aigntikent emozx 3 have meaning repertion)
acceprable recponso? accepiaable resporso have no mosning sau reperiiont)
requesti dignlficat
Yie
No
O3, Plesau choowe lae raagorow lisl profe (cwaed on helplukene)
butri
Figure 5: The website interface for our human evaluation (see App. G for details). Users need to log to the system, read the prompt and outputs from two models (with model names anonymized and order randomized), then answer whether output A and output B are acceptable or not in individually, and finally compare them in terms of helpfulness.
G.2 Inter-Annotator Agreement
We measure the agreement of our annotators on a subset of 119 examples (63 instances randomly sampled from the ChatGPT3 vs TŪLU 65B comparison, and 59 instances randomly sampled from the TÜL.U 65B vs TŪLU 7B comparison). We assign two annotators for each of these examples and compute their agreement for both the acceptance evaluation and pairwise comparison evaluation.
21

---

The annotators achieve agreement of 0.84 for whether model output should be accepted or not. For the pairwise comparison. following Zhou al. [56), we report tie-discounted accuracy. which assigns one point if both annotators agreed, half a point if either annotator (but not both) labeled tie, and zero point otherwise. We also merged "clearly better" and "slightly better" together, so our final options will be simply comparing which of A and B is better, or tie. Our annotators achieved an agreement of 0.72 for this pairwise comparison.
Although these numbers show reasonable agreement, we also note that there is large extent of subjectivity in human evaluation. This noise level also indicates that some prior work [5] that uses small number of examples for human evaluation might not be reliable enough. We suggest that the community needs to further improve the reliability and scalability of human evaluation for instruction- following models.
perrutioo-following
H Further Investigation of Figure 2
To further investigate the degree to which the number of unique tokens being used by GPT-4 as marker of quality, created dummy evaluator that compares model outputs, and assigns win to the output with more unique tokens. We plot the win-rate calculated using this dummy evaluator against the win-rate calculated using GPT-4 in Figure 6
find that while the dummy evaluator generally over-estimates the win-rate, the trend is still strikingly linear. We note that the R² for the trendline is ,91, suggesting that the unique token count explains a large proportion of the variance in the win rates. Based on this, we believe that the number of unique tokens certainly key preference that GPT-4 cares about in its evaluation, although it is still not the only important feature.
100.00%
Dummy Evaluator Winrate (%)
75.00%
50.00%
25.00%
0.00%
0.00%
25.00%
50.00%
75.00%
AlpacaEval Winrate (%)
Figure 6: Win-rate scores of all models judged by the dummy evaluator against win-rate of all models using the GPT-4 evaluator.
I
Model Licenses
We provide brief information about the licenses the underlying models we make use this work below.
LLAMA: The LLAMA model weights are released under custom license that allows using the model for non con-commercial research purposes.
LLAMA-2: The LLAMA-2 model weights are released under a custom license that allows for commercial and research uses with some limitations (e.g., having less than 700 mil- lion monthly active users used in commercial application), and explicitly allows for redistribution of the weights.
22

---

Pythia: The Pythia weights are released under the Apache-2.0 2.0 license.
OPT: The OPT model weights are released under a custom license that allow only using the model for non-commercial research purposes.
23


Reviews:
[{"review_id": "PP8ZD72Wq2", "reviewer": "Reviewer_hHUW", "summary": "", "questions": "", "limitations": "The authors comprehensively describe the limitations and broader impact in the appendices. These sections could also mention the subjectivity of individual annotators/difficulty of objectively evaluating generated outputs (related to the experiments in Section 4.3, 5.4). \n", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* In my personal opinion, the current submission is of the camera-ready quality; it is very easy to read.\n* The experimental design is diverse: the authors consider multiple types of (i) open-source instruction-tuning datasets, (ii) downstream tasks for evaluating the models, and (iii) metrics for evaluating the models. \n* The instruction-tuned models are publicly released.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "h2NyW1ECBD", "reviewer": "Reviewer_rWkW", "summary": "", "questions": "", "limitations": "1. This work didn't introduce new instruction tuning datasets or new benchmarks for evaluating the ability of LMs. It can be useful to ", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The authors perform a comprehensive study on open-source instruction-tuning datasets and demonstrate the impact of instruction-tuning with these datasets on open models of various sizes. It shows that instruction tuning is in general beneficial to improve model performance, especially for smaller models. \n\n2. This work suggests a benchmark-based evaluation method for core reasoning and fact-recall skills in addition to the model or human-based evaluation methods which can be subjective. It also shows how model-based evaluation might be biased towards responses with more unique tokens. The results on human-based evaluation show inconsistent evaluation on two different metrics (acceptance rate and pairwise preference rates), which are interesting and might suggest some inconsistency in human evaluation.\n\n3. The TULU model checkpoints, training and evaluation scripts are released on Github.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yXS5Tos8uI", "reviewer": "Reviewer_9TsL", "summary": "", "questions": "", "limitations": "The provided limitations section covers it well. ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Understanding what matters in instruction tuning of LLMs is highly important and timely\n- Comprehensive evaluation\n- Useful insights\n- New instruction-tuned model that may be helpful for future research", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "K8bXmcg3pe", "reviewer": "Reviewer_W1Ez", "summary": "", "questions": "", "limitations": "1. As mentioned above, the scope of work is ambiguous. **Instruction** usually refers to some text input regarding the procedures or requirements. \n\n2. Though the experiment are complete and the conclusions are achieved, there are hardly any inspiration to the community. The 5 key findings listed in L52-L62 are already widely agreed.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The work is a decent exploration of fine-tuning the LLMs, the experiment setup and results are well presented, and the code/model release is complete.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "8a8iLF5yZ8", "reviewer": "Reviewer_LRM9", "summary": "", "questions": "", "limitations": "The paper has Limitation (Appendix A) and Broader Impacts (Appendix B) sections with sufficient discussion. I believe it would be good to acknowledge the lack of evaluations around societal impact, and it is easy to add this point. ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper has a clear problem formulation that is timely and important. While instruction-tuning is common practice by now, the effect of instruction datasets have not been thoroughly analyzed from a comprehensive set of facets in prior work. \n\n2. The paper offers many important insights. To name a few, the findings around \n\t- Instruction-tuning can benefit less, or sometimes hurt larger models (L232-234)\n\t- Many open-source datasets can hurt multilingual performance (Table 3). \n\t- Highlighting that more effort should be spent in characterizing the diversity/comprehensiveness of instruction datasets.\n\nare important to several stakeholders, from users to developers. While some of these are pronounced in earlier works, I believe the paper does a great job of providing a systematic analysis of all these facets. \n\n3. The evaluation is comprehensive and provides a good point of reference to the community. Both the model evaluations and instruction datasets that are experimented with are broad and well-represents prior work. \n\n4. The provided model checkpoints are an extremely valuable resource for the community. It removes the barrier of entry to conduct further research around the effect of instruction-tuning datasets. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents thorough analysis on a wide range of instruction-tuned LMs.  The experiments are timely and the results will provide interesting insights to researchers in this field.
Decision: Accept (Spotlight)