How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
  Resources
Authors: name: Yizhong Wang; name: Hamish Ivison; name: Pradeep Dasigi; name: Jack Hessel; name: Tushar Khot; name: Khyathi Raghavi Chandu; name: David Wadden; name: Kelsey MacMillan; name: Noah A. Smith; name: Iz Beltagy; name: Hannaneh Hajishirzi
Published: 2023-06-07 19:59:23

Abstract: In this work we explore recent advances in instruction-tuning language models
on a range of open instruction-following datasets. Despite recent claims that
open models can be on par with state-of-the-art proprietary models, these
claims are often accompanied by limited evaluation, making it difficult to
compare models across the board and determine the utility of various resources.
We provide a large set of instruction-tuned models from 6.7B to 65B parameters
in size, trained on 12 instruction datasets ranging from manually curated
(e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and
systematically evaluate them on their factual knowledge, reasoning,
multilinguality, coding, and open-ended instruction following abilities through
a collection of automatic, model-based, and human-based metrics. We further
introduce T\"ulu, our best performing instruction-tuned model suite finetuned
on a combination of high-quality open resources.
  Our experiments show that different instruction-tuning datasets can uncover
or enhance specific skills, while no single dataset (or combination) provides
the best performance across all evaluations. Interestingly, we find that model
and human preference-based evaluations fail to reflect differences in model
capabilities exposed by benchmark-based evaluations, suggesting the need for
the type of systemic evaluation performed in this work. Our evaluations show
that the best model in any given evaluation reaches on average 83% of ChatGPT
performance, and 68% of GPT-4 performance, suggesting that further investment
in building better base models and instruction-tuning data is required to close
the gap. We release our instruction-tuned models, including a fully finetuned
65B T\"ulu, along with our code, data, and evaluation framework at
https://github.com/allenai/open-instruct to facilitate future research.