SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Performance of TÜLU and other of our trained models to vanilla LLAMA models and the state-of-the-art proprietary models across evaluation settings. See Table 8 for a complete list.
MMLU(factuality)	GSM(reasoning)	BBH(reasoning)	TydiQA(multilinguality)	Codex-Eval(coding)	AlpacaEval(open-ended)	Average
EM(0-shot)	EM(8-shot, CoT)	EM(3-shot, CoT)	F1(1-shot, GP)	P@10(0-shot)	Win % vs Davinci-003
Vanilla LLaMa models ↓
LLaMa 7B	31.5	10.0	33.0	38.4	20.5	-	-
LLaMa 13B	42.3	14.5	39.3	43.2	28.6	-	-
LLaMa 30B	54.6	36.0	49.5	55.3	42.8	-	-
LLaMa 65B	58.7	50.0	58.1	56.8	46.9	-	-
LLaMa-2 7B	41.8	12.0	39.3	51.2	26.8	-	-
LLaMa-2 13B	52.0	25.0	48.9	56.5	32.5	-	-
65B models trained on alternate data mixtures ↓
ShareGPT 65B	61.3 (+2.6)	59.0 (+9.0)	55.8 (-2.3)	31.6 (-25.2)	56.2 (+9.3)	73.6	56.3
Human mix. 65B	60.4 (+1.7)	60.0 (+10.0)	54.8 (-3.3)	58.3 (+1.7)	44.6 (-2.3)	43.4	53.6
models trained on our final Human+GPT data mixture ↓
TÜLU 7B	44.8 (+13.3)	25.0 (+15.0)	38.5 (+5.5)	43.5 (+5.1)	29.1 (+8.6)	48.6	38.3
TÜLU 13B	49.3 (+7.0)	40.5 (+26.0)	43.3 (+4.0)	45.6 (+2.4)	35.9 (+7.3)	56.5	45.2
TÜLU 30B	57.7 (+3.1)	53.0 (+17.0)	51.9 (+2.4)	51.9 (-3.4)	48.0 (+5.2)	62.3	54.1
TÜLU 65B	59.2 (+0.5)	59.0 (+9.0)	54.4 (-3.7)	56.6 (-0.2)	49.4 (+2.5)	61.8	56.7
models trained on our final Human+GPT data mixture using LLaMA-2 ↓
TÜLU-1.1 7B	49.2 (+7.4)	37.0 (+25.0)	44.2 (+4.9)	52.8 (+1.6)	33.9 (+7.1)	57.3	45.7
TÜLU-1.1 13B	52.3 (+0.3)	53.0 (+28.0)	50.6 (+1.7)	58.8 (+2.3)	38.9 (+7.4)	64.0	52.9
Proprietary models ↓
ChatGPT	67.9	76.0	66.1	51.9	88.4	83.6	72.3
GPT-4	82.4	92.5	88.0	70.8	94.1	93.5	86.9
models (e.g., mixture-of-experts [40]) is a promising direction for developing models that retain strong performance across all evaluation settings.
Base model quality is extremely important for downstream performance. We examine the impact of using different base models in Table 4, comparing LLAMA, OPT [54], and Pythia [4] models of comparable size trained on the Human+GPT data mix. Across all evaluation settings, we find that using LLAMA performs best by a significant margin, likely due to the fact that LLAMA is pretrained on significantly more tokens than the other models (see Table 2). This suggests that models pretrained on larger (or potentially higher-quality) corpora are preferable as base models for instruction tuning. The later addition of LLAMA-2 confirms this finding by showing a significant improvement can come from only the base model upgrade.
Some datasets degrade vanilla model performance. Notably, most datasets we evaluate cause degradation in performance on GSM and TydiQA over the vanilla base model. We hypothesise this is due to data style and quality. Many of the datasets we examine contain little to no examples of chain-of-thought-style reasoning and contain little to no multilingual data. As such, training on these datasets likely results in some forgetting of the CoT or multilingual abilities previously held by the model, resulting in degraded performance. Additionally, we note that self-instruct appears to cause degradations across most tasks, which we hypothesise is due to the relatively poor quality of the original self-instruct data, being generated by a weaker model (base GPT-3) than the other GPT-distilled datasets.
5.2 Pushing the Limits of Open Models
Having established that (a) using a broad mix of data is best, and (b) using LLAMA as the base model is preferable to other open alternatives, we compare the performance of models trained on the Human+GPT data mix (TÜLU models) across all LLAMA sizes in Table 5. We find that:
Instruction tuning brings large benefits on top of LLAMA models at all sizes. On average, all LLAMA models improve considerably after instruction tuning.
Smaller models benefit most from instruction tuning. We find that relative improvements from instruction tuning are largest for the smallest models, and shrink as models get larger. Notably, the 65B LLAMA model performs comparably or better than the 65B TÜLU model on MMLU, BBH, and TydiQA. This suggests that instruction-tuning does not help to enhance strong capabilities already present in the original model, and also highlights that care must be taken during finetuning to avoid forgetting the base model's original capabilities.
TULU still lags behind state-of-the-art proprietary models. Despite the impressive performance of TULU 65B, it lags behind ChatGPT and GPT-4 in all evaluation settings, contrary to prior claims that models trained on these open resources can match ChatGPT [56, 8]. We note we cannot discount the possibility that either ChatGPT or GPT-4 was trained on significant portions of our evaluation suite. However, the presence of a significant gap between TULU models and ChatGPT matches our findings in the model and human-based evaluations, which are less likely to be compromised.
5.3 Evaluation of Potential Risks and Harms
ToxiGen (↓)	TruthfulQA (↑)
Model ↓	7B	13B	7B	13B
LLAMA	85.4	82.6	26.2	23.6
+ SuperNI	85.3	77.3	26.7	26.2
+ CoT	63.0	43.9	35.1	35.5
+ Flan V2	77.5	61.4	33.2	33.4
+ Dolly	72.1	78.9	30.1	32.9
+ Open Assistant 1	39.2	5.2	40.9	48.6
+ Self-instruct	89.0	89.3	22.4	22.4
+ Unnatural Inst.	35.8	55.7	27.3	31.7
+ Alpaca	63.2	58.1	33.5	39.8
+ Code-Alpaca	84.3	92.0	25.1	26.7
+ GPT4-Alpaca	3.9	1.2	51.2	56.7
+ Baize	77.2	41.2	42.4	43.9
+ ShareGPT	5.5	2.5	45.3	60.0
+ Human mix.	51.8	76.9	34.1	32.1
+ TÜLU	10.6	0.1	44.6	41.6
ChatGPT	27.7	75.2
GPT-4	10.6	82.3
Table 6: Performance of models on ToxiGen (% toxic generations, lower is better) and TruthfulQA (% truthful and informative answers, higher is better). See Table 9 and Table 10 for the full breakdown of these two evaluations.
We evaluate our models on ToxiGen and TruthfulQA to measure the degree to which different datasets are likely to yield models that generate toxic language or misinformation. We find that:
Trends remain similar to capability-focused benchmarks. Similarly to the results in Sec. 4.1, we find that GPT-distilled datasets yield the best overall performance and that there is a large variance in performance across datasets.
Models trained on GPT-sourced data yield less toxic generations than GPT. Larger models trained on GPT-distilled data appear to refuse to produce toxic generations almost entirely, despite the fact that ChatGPT and GPT-4 produce toxic generations a non-trivial amount of the time. We hypothesise this is due to our models overfitting on refusal-style behaviour, refusing to generate anything moderately toxic, while GPT models balance refusal behaviour with helpfulness to a greater extent.
TruthfulQA performance does not scale. Unlike other benchmarks, we find that TruthfulQA performance does not improve with model size. Further examining this, we find that larger models do output more correct facts, but also tend to
hedge and refuse to give informative answers more often, resulting in little to no overall improvements as model size increases.
5.4 Model-Based Evaluation Results for Open-Ended Generation
We report the AlpacaEval win-rates of our models in Table ??. We find that:
Models trained on mixtures based on traditional NLP datasets perform poorly. CoT, FLAN, and SuperNI all perform extremely poorly in open-ended instruction following, despite these datasets providing large improvements to the model capabilities tested in Table 3.
Datasets that encourage long, diverse generations perform best. Intrigued by ShareGPT's performance, we plot the average number of unique tokens in model generations against the AlpacaEval win-rate in Figure 2. We find that the evaluation is strongly correlated with the average number of unique tokens (Pearson correlation of 0.96,  $p \ll 0.05$ ). Given GPT-4's strong performance on other tasks, we do not believe that GPT-4 evaluation is merely counting unique tokens, but this result highlights how model preference scores do not necessarily reward only model capabilities.
Training Dataset ↓	7B	13B	30B	65B
SuperNI	2.9	4.2
CoT	5.0	6.0
Flan V2	3.1	3.2
Dolly	11.0	13.7
Open Assistant 1	51.4	58.1
Self-instruct	4.0	5.0
Unnatural Instructions	7.5	8.4
Alpaca	21.4	21.9
Code-Alpaca	15.3	15.8
GPT4-Alpaca	57.3	63.1
Baize	20.0	21.9
ShareGPT	62.4	70.5	69.1	73.6
Human mix.	28.7	35.0	38.3	43.4
TÜLU	48.6	56.5	62.3	61.8
Table 7: Win-rate  $(\%)$  of LLAMA models of varying sizes finetuned on the given dataset against Davinci-003 using AlpacaEval [27].
Figure 2: Win-rate scores of 13B models (trained on different datasets) given by GPT-4 strongly correlate with the average numbers of unique tokens in the model responses (Pearson  $r = 0.96$ ).
Figure 3: Human acceptance rates for four evaluated models.
Figure 4: Human preference rates for three comparison pairs of models.
ShareGPT performs best. We find that ShareGPT consistently performs best across all model sizes, including models trained on data mixes that include ShareGPT. Models trained on ShareGPT achieve higher win-rates than models over twice their size (e.g., 13B ShareGPT vs 65B TÜLU). We hypothesize this is due to ShareGPT's diversity, size, and the high average # tokens of target responses.
Overall, these results suggest that while model preference evaluation is important, it does not provide a holistic evaluation of these models. Instead, model preference evaluation should only be included as part of a larger, more comprehensive evaluation setup.
5.5 Human Evaluation Results for Open-Ended Generation
Finally, we show the human evaluation results in Figure 4 and we refer the reader to Appendix §G.2 for the inner-annotator agreement. We find that the human evaluation results largely correlate with the AlpacaEval and benchmark-based evaluation: all evaluations show that 65B TÜLU outperforms 7B TÜLU, suggesting making use of larger base models is important, and there is still a nontrivial gap in performance between 65B TÜLU and ChatGPT. We also find that making use of distilled datasets provides a large performance boost, suggesting that human-authored datasets are lacking in comparison. These observations are also consistent with the acceptability scores in Figure 3. However, we note that 7B TÜLU outperforms the human-mix 65B TÜLU in the model preference evaluation, but if we compare the acceptability scores in Figure 3, the opposite appears true. This is further evidence that model pairwise evaluation may not always reveal model deficiencies. In this case, the 65B human-mix model is more likely to yield acceptable (if not high-quality) responses than the 7B model.
Instruction Tuning of LMs Finetuning language models on diverse instruction sets alongside regular samples has been shown to greatly improve zero-shot performance on unseen tasks [39, 51, 49, 32, 9, 48], and serves as a good base for further finetuning in supervised settings [31]. Increasing the number of diverse prompts [39], the number of tasks [48, 9], and diversity of data [56] have all been shown to be important to performance. More recently, a growing number of models have made use of model-generated instruction-augmented data [47, 23, 25, 53], most often generated or collected from larger proprietary models such as ChatGPT or GPT-4 [8, 15, 43, 52, 36, inter alia]. Despite the explosion of models and datasets, evaluation remains inconsistent and difficult, with different evaluation setups used across models. Prior work has examined models trained on varying dataset sources with the aim of identifying 'the best mixture' [31, 24], but is often limited to examining only benchmark performance, and covers a smaller number of instruction sources than in this work. QLoRA [14] also explores (quantized and parameter-efficient) instruction-tuning of recent models and datasets, but explores a smaller range of models, datasets, and evaluations than this work.
Evaluation of LMs Given the success of LMs on NLP and instruction-following tasks, many evaluation frameworks have been proposed. Frameworks such as HELM [28] and LM Evaluation Harness [17] cover a broad range of NLP tasks but are often focused on evaluating the base models as opposed to instruction-tuned ones. Similar to our work, Chung et al. [9] focus on a series of benchmark evaluations focused around factuality and reasoning, but largely neglect open-ended instruction following abilities. Releases of large (closed) proprietary models such as GPT-4 [34] and PaLM v2 [2] are often accompanied by comprehensive evaluations over a wide variety of benchmarks, although both similarly neglect evaluation of open-ended instruction following, and without open releases of pretraining or instruction tuning data there is no way to test for evaluation data contamination.
Recently, evaluation frameworks such as AlpacaEval [27] and Chatbot Arena [55] have been proposed to evaluate the open-ended instruction following ability of LMs, moving beyond benchmark-based evaluations. These either make use of other models (in the case of AlpacaEval) or humans (in the case of Chatbot Arena) as annotators for judging model generations. We make use of this recent work and evaluate our models on traditional benchmarks, model-based evaluation, and human-based evaluation. Concurrent to this work, Gudibande et al. [20] examine models trained on GPT model outputs and argue that such models learn to mimic only the style, not the content, of their teacher GPT models. While we similarly find that existing datasets fail to train models close to strong proprietary models, the diversity of performance we observe across datasets suggests that significant performance improvements can be achieved through imitation data, so long as it contains a diverse and wide-ranging set of skills and domains.
In this work, we provide an extensive evaluation of a wide variety of publicly-available resources for instruction-tuning models, and compare them to the strongest proprietary models currently available. We find that using strong base models is vital to performance, combining datasets works best on average (but does result in slight performance drops compared to best performance in specific tasks), and our strongest open models do not yet match ChatGPT or GPT-4. Furthermore, we believe that our evaluation highlights the need for the continued development of strong base models and broader, diverse datasets. Finally, we hope that our evaluation and released code and models enable more comprehensive evaluations and spur research to close these gaps and shed insights on all large language models, closed or open.
Work at UW was partially supported by the Office of Naval Research under MURI grant N00014-18-1-2670, Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8650-23-C-7316 and MCS program through NIWC Pacific (N66001-19-2-4031), NSF IIS-2044660, and a gift from Apple. We thank colleagues at AI2 and UW NLP for their constructive feedback and intellectual support. We are particularly grateful to Tim Dettmers for his suggestions on efficient
inference techniques, and Artidoro Pagnoni for providing the reproduced FLAN V2 dataset. We also acknowledge support from AMD and CSC's LUMI cluster, and the Beaker team at AI2, which provided the essential computational infrastructure for our experiments. Finally, we are sincerely thankful for the following contributors to our human evaluation: Valentina Pyatkin, Clara Na, Yuling Gu, Yuchen Lin, Haiyan He, David Graham, Hao Peng, Hyunwoo Kim, Alisa Liu, Youngjae Yu, Tal August, and Egor Klevak.
[1] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance. Huggingface Model Release, 2023. URL https://huggingface.co/tiiuae/falcon-40b.
[2] R. Anil, A. M. Dai, O. First, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[4] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[5] T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023.
[6] S. Chaudhary. Code alpaca: An instruction-following llama model for code generation. GitHub repository, 2023. URL https://github.com/sahil280114/codealpaca.
[7] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[8] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with  $90\%$  chatgpt quality. Blog post, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
[9] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
[10] J. H. Clark, E. Choi, M. Collins, D. Garrette, T. Kwiatkowski, V. Nikolaev, and J. Palomaki. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. TACL, 2020. URL https://arxiv.org/abs/2003.05002.
[11] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
[12] Databricks. Free dolly: Introducing the world's first truly open instruction-tuned llm. Blog post, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
[13] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022.
[14] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
[46] P. Wang, L. Li, L. Chen, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023.
[47] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[48] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, et al. Super-Natural Instructions: Generalization via Declarative Instructions on 1600+ Tasks. In EMNLP, 2022.
[49] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations (ICLR), 2022.
[50] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[51] O. Weller, N. Lourie, M. Gardner, and M. E. Peters. Learning from task descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361-1375, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.105. URL https://aclanthology.org/2020.emnlp-main.105.
[52] C. Xu, D. Guo, N. Duan, and J. McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196, 2023.
[53] C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
[54] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[55] L. Zheng, Y. Sheng, W.-L. Chiang, H. Zhang, J. E. Gonzalez, and I. Stoica. Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings. Blog post, May 2023. URL https://lmsys.org/blog/2023-05-03-arena/.
[56] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.
Despite the comprehensiveness of our evaluations, we note that we did not exhaustively cover all possible evaluations: for example, we do not explicitly evaluate models on their multi-turn dialogue abilities nor their summarization abilities. Instead, we focus on a core set of capabilities we believe important, and cover broad open-ended tasks via our model and human preference-based evaluations.
We also note that we do not cover all possible instruction datasets and open models released recently, due to the computational cost of doing this. Instead, we focus on a wide set of datasets we believe are broadly representative of the type of open instruction datasets available (human-authored, skill-targeted, GPT-distilled, etc), and focused on the strongest base model widely available when performing experiments. Future work could investigate whether more recent strong base models (e.g., the Falcon model [1]), or other instruction datasets, perform significantly better or differently from the models explored in this work.
Finally, we note that open-ended instruction-based evaluation is highly subjective and difficult due to its extremely open-ended nature. There is likely no one answer that is definitively the best for any given query, and different annotators (whether they be human or model) will have different biases and preferences. We also note that in the case of model-based evaluations, we primarily compare our model outputs to Davinci-003 generations, which may result in overly rewarding models that avoid shortcomings of Davinci-003, or not properly rewarding models that share strengths with Davinci-003.
Despite not being completely exhaustive in this work, we believe that by covering a broad range of models, it still serves as a useful and important contribution in showing what type of open resources work, and where future community efforts should go (better base models, more diverse instruction-tuning datasets).
We believe that a rigorous evaluation of existing resources is broadly positive, exposing the strengths and deficiencies of currently widely-available resources. Furthermore, as all resources used are widely available, the harm posed by training these models is fairly small. We do note that training and releasing especially large instruction-tuned models without well-tested guides carries a degree of risk, and such initially release our largest models with a gated setup (requiring users to apply for access and be manually approved) to limit potential harms.
C Instruction Datasets Details
We provide a brief description of all the instruction datasets used (and licenses) below: