INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large
  Language Models
Authors: name: Yew Ken Chia; name: Pengfei Hong; name: Lidong Bing; name: Soujanya Poria
Published: 2023-06-07 20:12:29

Abstract: Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as
conversational agents. These models, such as GPT-4, can not only master
language but also solve complex tasks in areas like mathematics, coding,
medicine, and law. Despite their impressive capabilities, there is still a lack
of comprehensive understanding regarding their full potential, primarily due to
the black-box nature of many models and the absence of holistic evaluation
studies. To address these challenges, we present INSTRUCTEVAL, a more
comprehensive evaluation suite designed specifically for instruction-tuned
large language models. Unlike previous works, our evaluation involves a
rigorous assessment of models based on problem-solving, writing ability, and
alignment to human values. We take a holistic approach to analyze various
factors affecting model performance, including the pretraining foundation,
instruction-tuning data, and training methods. Our findings reveal that the
quality of instruction data is the most crucial factor in scaling model
performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment. We
are encouraged by the rapid development of models by the open-source community,
but we also highlight the need for rigorous evaluation to support claims made
about these models. Through INSTRUCTEVAL, we aim to foster a deeper
understanding of instruction-tuned models and advancements in their
capabilities. INSTRUCTEVAL is publicly available at
https://github.com/declare-lab/instruct-eval.