Prometheus: Inducing Fine-grained Evaluation Capability in Language
  Models
Authors: name: Seungone Kim; name: Jamin Shin; name: Yejin Cho; name: Joel Jang; name: Shayne Longpre; name: Hwaran Lee; name: Sangdoo Yun; name: Seongjin Shin; name: Sungdong Kim; name: James Thorne; name: Minjoon Seo
Published: 2023-10-12 16:50:08

Abstract: Recently, using a powerful proprietary Large Language Model (LLM) (e.g.,
GPT-4) as an evaluator for long-form responses has become the de facto
standard. However, for practitioners with large-scale evaluation tasks and
custom criteria in consideration (e.g., child-readability), using proprietary
LLMs as an evaluator is unreliable due to the closed-source nature,
uncontrolled versioning, and prohibitive costs. In this work, we propose
Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation
capabilities when the appropriate reference materials (reference answer, score
rubric) are accompanied. We first construct the Feedback Collection, a new
dataset that consists of 1K fine-grained score rubrics, 20K instructions, and
100K responses and language feedback generated by GPT-4. Using the Feedback
Collection, we train Prometheus, a 13B evaluator LLM that can assess any given
long-form text based on customized score rubric provided by the user.
Experimental results show that Prometheus scores a Pearson correlation of 0.897
with human evaluators when evaluating with 45 customized score rubrics, which
is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392).
Furthermore, measuring correlation with GPT-4 with 1222 customized score
rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask
Eval) shows similar trends, bolstering Prometheus's capability as an evaluator
LLM. Lastly, Prometheus achieves the highest accuracy on two human preference
benchmarks (HHH Alignment & MT Bench Human Judgment) compared to open-sourced
reward models explicitly trained on human preference datasets, highlighting its
potential as an universal reward model. We open-source our code, dataset, and
model at https://github.com/kaistAI/Prometheus.