FLASK: Fine-grained Language Model Evaluation based on Alignment Skill
  Sets
Authors: name: Seonghyeon Ye; name: Doyoung Kim; name: Sungdong Kim; name: Hyeonbin Hwang; name: Seungone Kim; name: Yongrae Jo; name: James Thorne; name: Juho Kim; name: Minjoon Seo
Published: 2023-07-20 14:56:35

Abstract: Evaluation of Large Language Models (LLMs) is challenging because aligning to
human values requires the composition of multiple skills and the required set
of skills varies depending on the instruction. Recent studies have evaluated
the performance of LLMs in two ways, (1) automatic evaluation on several
independent benchmarks and (2) human or machined-based evaluation giving an
overall score to the response. However, both settings are coarse-grained
evaluations, not considering the nature of user instructions that require
instance-wise skill composition, which limits the interpretation of the true
capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language
Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation
protocol that can be used for both model-based and human-based evaluation which
decomposes coarse-level scoring to an instance-wise skill set-level.
Specifically, we define 12 fine-grained skills needed for LLMs to follow
open-ended user instructions and construct an evaluation set by allocating a
set of skills for each instance. Additionally, by annotating the target domains
and difficulty level for each instance, FLASK provides a holistic view with a
comprehensive analysis of a model's performance depending on skill, domain, and
difficulty. Through using FLASK, we compare multiple open-sourced and
proprietary LLMs and observe highly-correlated findings between model-based and
human-based evaluations. FLASK enables developers to more accurately measure
the model performance and how it can be improved by analyzing factors that make
LLMs proficient in particular skills. For practitioners, FLASK can be used to
recommend suitable models for particular situations through comprehensive
comparison among various LLMs. We release the evaluation data and code
implementation at https://github.com/kaistAI/FLASK.