SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yizhong Wang*♠♦ Hamish Ivison*♠ Pradeep Dasigi♠ Jack Hessel♠  
Tushar Khot♠ Khyathi Raghavi Chandu♠ David Wadden♠ Kelsey MacMillan♠  
Noah A. Smith♠♦ Iz Beltagy♠ Hannaneh Hajishirzi♠
$\spadesuit$ Allen Institute for AI  $\spadesuit$ University of Washington {yizhongw,hamishi}@allenai.org
In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with state-of-the-art proprietary models, these claims are often accompanied by limited evaluation, making it difficult to compare models across the board and determine the utility of various resources. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets ranging from manually curated (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, safety, and open-ended instruction following abilities through a collection of automatic, model-based, and human-based metrics. We further introduce TULU, our best performing instruction-tuned model suite finetuned on a combination of high-quality open resources.
Our experiments show that different instruction-tuning datasets can uncover or enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preference-based evaluations fail to reflect differences in model capabilities exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average  $87\%$  of ChatGPT performance, and  $73\%$  of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including a fully finetuned 65B TÜLU, along with our code, data, and evaluation framework to facilitate future research.2
The latest generation of large language models has brought unprecedented attention to the potential of language technologies. To support imperative user requests and a chat interface, these models often undergo an instruction-tuning step which involves training on supervised input/output pairs. Recent instruction tuning corpora are often gathered via crowdsourcing (Dolly [12], Open Assistant [26]) or via distillation from another model (Alpaca [43], Vicuna [8]). However, while some public, instruction-tuned models are advertised as comparable to powerful closed-source proprietary models such as ChatGPT, most experiments that support such claims only cover a small set of tasks, and mostly rely on model-based evaluation metrics [8, 56]. We contend that the evaluation setup should
2https://github.com/allenai/open-instruct
include tasks that test core reasoning and fact-recall skills of the model, in addition to testing model- or human-annotated generation quality, which may be more open-ended and subjective.
This paper provides a comprehensive evaluation of instruction-tuning resources: specifically, we conduct a large number of instruction tuning experiments spanning a dozen public corpora, and models ranging in scale from 6.7B to 65B. We evaluate both specific model capabilities (i.e., factual knowledge, reasoning, multilinguality, coding, safety) and open-ended instruction-following abilities. We report results based on automatic, model-based, and human-based evaluation metrics.
Our evaluation reveals that instruction tuning over different datasets appears to promote specific skills, and no one dataset provides the best performance across all evaluations. We also find that the underlying base model is paramount, with better base models (whether it be models trained on more tokens or larger models) performing better across the board. Surprisingly, we also find that the best-performing models in model-based evaluation are not the same as those that perform best on benchmark-based automatic evaluations, potentially partially due to GPT-4's strong bias toward long, diverse generations.
Building on our findings, we introduce TULU, a suite of 7B to 65B LLAMA models finetuned on a combination of data sources. TULU 65B is the largest publicly-released fully-instruction tuned LLAMA variant at the time of writing, to the best of the authors' knowledge. It is trained on 7 popular available datasets, and yields the best average performance across most model sizes while remaining within  $29\%$  of the best-performing model on each individual task. In summary, our key findings include:
- Instruction datasets targeted at specific domains and/or capabilities are extremely effective at improving model performance in those aspects.
- Larger or pretrained-for-longer base models consistently perform better than smaller ones after instruction tuning.
- Our model TÜLU - fine-tuned LLaMa on a combination of existing instruction datasets - achieves the best average performance across benchmarks, although it is not the overall best when considering different evaluation settings independently.
- Even a very large (65B) model finetuned on a large mix of instruction datasets fails to outperform ChatGPT, although it does perform significantly better than similar smaller models.
- Model-based preference evaluation on open-ended instruction following correlates strongly with the average number of unique tokens generated by a model, suggesting that model-based preference evaluation has biases that may hide differences in model capabilities.
We open-source the code for training and evaluating these large language models. We also release checkpoints trained on the different instruction datasets and their mixtures, including TÜLU. We hope this facilitates further development and investigation of open instruction-tuned models.
2 Background: Instruction Tuning and Resources
2.1 Instruction Tuning
Instruction tuning, in general, refers to the practice of finetuning pretrained language models to better understand and respond to a wide variety of human requests that are expressed in natural language [32, 49, 35]. In particular, instruction tuning is concerned with requests that include some indication of the task to be performed within the request itself (e.g., including task instructions in the input prompt). It has arisen as a critical step for generalizing models to new scenarios without dedicated training, and for letting non-experts naturally interact with these models. The training paradigms of instruction tuning can vary from supervised learning using demonstrations [49, 39, 48, 31] to reinforcement learning from feedback data [35, 3]. In this work, we focus on the supervised learning setup considering the current open resources for the RL-based approach are still rare, and we leave its exploration for future work.
The success of instruction tuning requires at least two key components: 1) a powerful pretrained language model that has grasped a vast amount of knowledge from web-scale pretraining, and 2) an instruction dataset that is diverse and representative enough to adapt the LM to potential downstream usage. We study these two factors in this work and introduce our studied open resources below.
Table 1: Instruction datasets investigated in this work. CoT and FLAN V2 are sampled to 100K to match the sizes of other datasets. We report the average number of conservation turns ( $\bar{N}_{rounds}$ ), average length of prompts ( $\bar{L}_{prompt}$ ), average length of completion ( $\bar{L}_{completion}$ ).
Datasets	Sourced from	#Instances	\( \bar{N}_{rounds} \)	\( \bar{L}_{prompt} \)	\( \bar{L}_{completion} \)
SuperNI [48]	NLP datasets + Human-written Instructions	96,913	1.0	291.1	38.7
CoT [50]	NLP datasets + Human-written CoTs	100,000	1.0	266.0	53.2
Flan V2 [31]	NLP datasets + Human-written Instructions	100,000	1.0	355.7	31.2
Dolly [12]	Human-written from scratch	15,011	1.0	118.1	91.3
Open Assistant 1 [26]	Human-written from scratch	34,795	1.6	34.8	212.5
Self-instruct [47]	Generated w/ vanilla GPT3 LM	82,439	1.0	41.5	29.3
Unnatural Instructions [23]	Generated w/ Davinci-002	68,478	1.0	107.8	23.6
Alpaca [43]	Generated w/ Davinci-003	52,002	1.0	27.8	64.6
Code-Alpaca [6]	Generated w/ Davinci-003	20,022	1.0	35.6	67.8
GPT4-Alpaca [36]	Generated w/ Davinci-003 + GPT4	52,002	1.0	28.0	161.8
Baize [52]	Generated w/ ChatGPT	210,311	3.1	17.6	52.8
\( ShareGPT^3 \)	User prompts + outputs from various models	168,864	3.2	71.0	357.8
2.2 Instruction Datasets
We attempt to collect a representative sample of different styles of datasets (listed in Table 1), including datasets: (1) created by researchers from existing NLP datasets (SuperNI [48], Flan V2 [31]); (2) written by humans from scratch for the purpose of instruction tuning (Dolly [12], Open Assistant 1 [26]); (3) generated by proprietary models (Self-Instruct [47], Unnatural Instructions [23], Alpaca [43], Baize [52], GPT4-Alpaca [36]); (4) comprised of user-shared prompts accompanied by model-generated completions (ShareGPT $^3$  [8]); (5) built for specific skills (CoT [50] for chain-of-thought, Code-Alpaca [6] for code generation). See Appendix C for further details.
2.3 Pretrained Models
We primarily use the LLAMA suite [44, 45], a series of pretrained models ranging in size from 6.7B to 65B parameters. We initially experimented with the LLAMA-1 models for the first version of this paper and added LLAMA-2 in our camera ready, which use similar numbers of parameters but were trained over significantly more tokens. These models represent the largest, highest-quality pretrained models available to the community (albeit under restrictive licensing). We also consider OPT [54] and Pythia [4] models with a size comparable to the LLAMA 6.7B model, to examine the effect of different base models. For simplicity, we will round all the sizes to the nearest integer number. We note several ongoing efforts to pre-train similar- or better-quality models [18, 33, 1]. We believe our findings should hold for these models and future stronger open base models.
Table 2: Base models that we finetuned in this work.
Base LMs	# Params	# Tokens
LLaMa [44]	6.7B	1.0T
13.0B	1.0T
32.5B	1.4T
65.2B	1.4T
LLaMa-2 [45]	6.7B	2.0T
13.0B	2.0T
OPT [54]	6.7B	180B
Pythia [4]	6.9B	300B
3 Training Models with Various Datasets
3.1 Unifying the Format
We format all datasets to follow a chatbot-style schema to unify the varied styles and formats of the instruction datasets, shown in Figure 1. This allows us to fit arbitrary rounds of interactions between the user and the language model (a.k.a. "assistant") into one input sequence and encode them together with a causal language model. We add special tokens  $< |$  user  $|>$  and  $< |$  assistant  $|>$  before user utterances and target assistant responses respectively, and an end-of-text marker  $</ s>$  at the end of each assistant output, which, at inference time, will stop the model's response for each round.
<sup>3</sup> ShareGPT (https://sharegpt.com/) data was used to build the Vicuna model [8], but the exact dataset has not been released. We instead use a reproduced version from https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/tree/main/HTML cleansed_raw_dataset, and follow Vicuna to split the long conversations into blocks with a maximum length of 2048 tokens.
3.2 Model Training Details
During training, we compute loss only on tokens after  $<$  assistant  $|>$  and before the next  $<$  user  $|>$  token. More formally, we consider an instruction dataset as consisting of  $N$  tuples, each with  $i$  turns,  $\{(x_1^j, y_1^j, x_2^j, y_2^j, \ldots, x_i^j, y_i^j)\}_{j=1}^N$ , where  $x_i$  is a user prompt and  $y_i$  the desired output. For most instances,  $i = 1$ , and we train the model to output  $y^j$  given  $x^j$ . However, in the case of conversation datasets, we train the model to predict  $y_i^j$  given some conversation history  $x_1^j, y_1^j, x_2^j, \ldots, x_i^j$ . We train decoder-only models, and use teacher-forcing with loss masking to train the models, where we mask all tokens belonging to the input sequence(s)  $x_i$ . Given  $X$  as the tokens belonging to the input, and  $Y$  as the target tokens, the loss function is:
$$
L = - \sum_ {j} \log p _ {\theta} (t _ {j} \mid t _ {<   j}) \times \left\{ \begin{array}{l l} 1 & \text {i f} t _ {j} \in Y \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
Explain the fault-tolerance of the reaction control system on the Space Shuttle.
The reaction control system (RCS) on the Space Shuttle was designed to be fault-tolerant, meaning it was able to continue functioning even if one or more of its components failed. The RCS consisted of two sets of ...
Figure 1: An example from ShareGPT data. We use  $< | \text{role} | >$  to set the boundary between messages. The entire sequence is encoded together, and loss is computed on the assistant parts (colored in blue).
Did the RCS have any on-orbit failures?
There were several instances where the reaction control system (RCS) on the Space Shuttle experienced failures or malfunctions during on-orbit missions. These ..  $\langle /s\rangle$
where  $t_j$  is the  $j$ th input token (belonging to  $X$  or  $Y$ ). See Appendix §D for further training details.
3.3 TÜLU: a Better Instruction-Tuned Model by Combining Resources
Existing studies [48, 31] (and our own evaluation below) have shown that increasing the diversity of instructions can effectively improve the performance of instruction tuning. Following this motivation, we create two mixtures of datasets:
Human data mixture, which comprises the best human-authored datasets, including FLAN V2, CoT, Dolly, and Open Assistant 1 (we exclude SuperNI as FLAN V2 includes most tasks in SuperNI);
Human+GPT data mixture, which comprises the human mixture and three additional datasets that have generations by OpenAI GPT models, including GPT4-Alpaca, Code-Alpaca, and ShareGPT.
For both mixtures, we concatenate datasets and leave exploring more complex sampling mixtures to future work. We name LLAMA models trained on the Human+GPT data mixture TÜLU, after a hybrid camel resulting from interbreeding between different species. We differentiate the TÜLU models trained from the LLAMA-2 base models by versioning them as TÜLU-1.1.
4 Evaluation Setup
Evaluation of instruction-following models remains a challenging problem due to the enormous scope of "generality" and its open-ended nature. However, we argue that general-purpose models should be able to perform some core tasks before they can generalize to satisfy various practical needs. As such, we set up a multi-faceted evaluation to cover several key aspects of capabilities covering core abilities and open-ended instruction following. Our evaluations closely follow prior work on evaluating instruction-tuned models [9, 2, 47, 8, 16], but serve as the first one to compile them together for systematic evaluation.
4.1 Facets of Evaluation
Factual knowledge is essential for language models to serve users' information needs. We use the Massive Multitask Language Understanding dataset (MMLU [22]) for measuring models' factual knowledge. MMLU consists of a set of questions about 57 subjects ranging in difficulty from elementary levels to professional levels, and its multiple-choice format makes it suitable for probing models' knowledge without worrying about the open-endedness of generations.
Reasoning is another fundamental ability for models, especially for solving complex tasks. We use the test split of Grade School Math dataset (GSM [11]) to evaluate models' mathematical reasoning capabilities. We also adopt Big-Bench-Hard (BBH [42]), which contains 23 challenging tasks from Big-Bench [41], to evaluate models' general reasoning capabilities.
Multilinguality acts as an important perspective of models for serving people from different backgrounds. We use TyDiQA [10], a multilingual question answering benchmark covering 11 typologically diverse languages for testing how much models can process non-English text. We use the gold-passage setup where one passage containing the reference answer is given.
Coding is a particular application that people have used language models for and might be important for integrating these models with external tools [5]. We use the HumanEval dataset [7] to evaluate the models' capability to generate functionally correct programs from docstrings. To avoid ambiguity with our human evaluation, we call this dataset Codex-Eval in this paper.
Open-ended instruction following. While the performance on the benchmarks above quantifies the models' ability at specific skills, it may not reflect how well the models can handle instructions from real users, which cover highly diverse requests and are often open-ended. For example, the popular ShareGPT dataset contains instances of users asking for programming help, resume formatting tips, educational role-playing, pronunciation suggestion, fanfiction writing, and more. We evaluate such open-ended instructability of models using both model-based evaluation (§4.2) and human evaluation (§4.3), both of which consist of multiple test sets from existing studies [47, 8, 26, 3, 19].
Safety is of particular concern regarding the fast-developing language models to ensure the ethical and proper use of them. Following LLAMA-2 [45], we employ ToxiGen [21] to measure the amount of toxic language and hate speech generation across different groups when the models are prompted to do so. We also adopt TruthfulQA [30] to measure how well models can avoid generating known falsehoods due to misconceptions or false beliefs while providing useful information.
For all the benchmark-based evaluations, we follow their standard metrics, while we subsample some benchmarks to a reasonable size to improve the efficiency of doing chain-of-thought reasoning. We refer the reader to Appendix §E for the setup details.
4.2 Model-Based Evaluation using GPT-4
To evaluate the open-ended instructability, we first adopt a model-based approach introduced in AlpacaEval [27]. The test set consists of 805 instructions, with 252 instructions from the Self-Instruct evaluation [47], 188 from the Open Assistant evaluation [26], 129 from the helpful evaluation by Anthropic [3], 80 from the Vicuna evaluation [8], and 156 from the Koala evaluation [19].
We use their simulated GPT-4 annotator, which computes the win rate of the testing model as judged by GPT-4 when compared to the outputs produced by Davinci-003. We use the AlpacaEval codebase and prompts [27] to make our scores directly comparable to those on the AlpacaEval leaderboard<sup>4</sup>. When doing pairwise comparisons with GPT-4, the orders of model outputs are randomized to avoid position bias during evaluation [46]. We do not evaluate vanilla LLAMA models due to them having little instruction-following ability without further prompt engineering.
4.3 Human Evaluation
To further test the quality of the open-ended generations, we conduct a human evaluation based on 332 instructions that combine the Self-Instruct evaluation set [47] and Vicuna evaluation set [8]. Inspired by Bai et al. [3], we design a similar interface (Figure 5) for gathering human judgments of model outputs along the following dimensions. We note that we evaluated based on our fine-tuned LLAMA-1 models, as LLAMA-2 was not available at the time of this experiment.
Individual acceptability. We ask human raters to assess whether each system's responses were acceptable in isolation. This is a binary decision, and we ask the raters to mark a response as acceptable if and only if the response answered the request in the query, had no significant errors, and did not have repetitive information.
4https://tatsu-lab.github.io/alpaca_eval/
Table 3: Comparison of different instruction tuning datasets, showing that different instruction-tuning datasets can excel in different aspects, and mixtures perform best on average. Cells are blue if the finetuning boosts the vanilla LLAMA performance, and orange if the finetuning hurts the performance.
MMLU (factuality)	GSM (reasoning)	BBH (reasoning)	TydiQA (multilinguality)	Codex-Eval (coding)	AlpacaEval (open-ended)	Average
EM (0-shot)	EM (8-shot, CoT)	EM (3-shot, CoT)	F1 (1-shot, GP)	P@10 (0-shot)	Win % vs Davinci-003
Vanilla LLaMa 13B	42.3	14.5	39.3	43.2	28.6	-	-
+SuperNI	49.7	4.0	4.5	50.2	12.9	4.2	20.9
+CoT	44.2	40.0	41.9	47.8	23.7	6.0	33.9
+Flan V2	50.6	20.0	40.8	47.2	16.8	3.2	29.8
+Dolly	45.6	18.0	28.4	46.5	31.0	13.7	30.5
+Open Assistant 1	43.3	15.0	39.6	33.4	31.9	58.1	36.9
+Self-instruct	30.4	11.0	30.7	41.3	12.5	5.0	21.8
+Unnatural Instructions	46.4	8.0	33.7	40.9	23.9	8.4	26.9
+Alpaca	45.0	9.5	36.6	31.1	29.9	21.9	29.0
+Code-Alpaca	42.5	13.5	35.6	38.9	34.2	15.8	30.1
+GPT4-Alpaca	46.9	16.5	38.8	23.5	36.6	63.1	37.6
+Baize	43.7	10.0	38.7	33.6	28.7	21.9	29.4
+ShareGPT	49.3	27.0	40.4	30.5	34.1	70.5	42.0
+Human data mix.	50.2	38.5	39.6	47.0	25.0	35.0	39.2
+Human+GPT data mix.	49.3	40.5	43.3	45.6	35.9	56.5	45.2
Table 4: Performance of different base models after training on the Human+GPT data mixture.
MMLU (factuality)	GSM (reasoning)	BBH (reasoning)	TydiQA (multilinguality)	Codex-Eval (coding)	AlpacaEval (open-ended)	Average
EM (0-shot)	EM (8-shot, CoT)	EM (3-shot, CoT)	F1 (1-shot, GP)	P@10 (0-shot)	Win % vs Davinci-003
Pythia 6.9B	34.8	16.0	29.2	32.8	20.9	23.5	26.2
OPT 6.7B	32.6	13.5	27.9	24.1	8.9	25.9	22.2
LLAMA 7B	44.8	25.0	38.5	43.5	29.1	48.6	38.3
LLAMA-2 7B	49.2	37.0	44.2	52.8	33.9	57.3	45.7
Pairwise preference. We then ask humans to compare the outputs of two systems and select which one they think is more helpful. This is a 5-way decision, and the raters could select if one of the responses is "clearly" or "slightly" better than the other or if it is a tie implying that both responses were equally good or bad.
To get a more reliable evaluation, we recruited a group of 18 expert annotators who are researchers at AI2 or students at UW. All of them are fluent English speakers, holding bachelor's degrees or above.
5.1 Analysis of Instruction Tuning Datasets and Base Models
To understand how the instruction datasets listed in Table 1 contribute to model abilities, we evaluated LLaMa 13B models trained on these datasets using our evaluation suite. Table 3 shows the results on our benchmark evaluation set, with more extensive results in App. F. We find that:
There is not a single best instruction tuning dataset across all tasks. Different datasets enable different capabilities in the model. Noteworthy examples include training on CoT being particularly helpful for mathematical reasoning in GSM and Code-Alpaca being helpful for Codex-Eval. We hypothesize that success on these tasks, which are significantly different from the rest of the evaluation tasks, calls for training sets where these tasks are well-represented. Apart from constructing task-specific datasets manually, distilling task-specific data from large models also appears to be an effective way to ensure this (e.g., CodeAlpaca is distilled from Davinci-003).
Combining datasets results in the best overall performance on the benchmark tasks. While models trained on our combination datasets are often not the best model for a single task (being the best only in 2 out of 6 evaluation settings), they are the best when measuring average performance across tasks. This suggests that future work into better dataset mixing or instruction-tuning modular