Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models
Authors: Seungone Kim
Jamin Shin
Yejin Cho
Joel Jang
Shayne Longpre
Hwaran Lee
Sangdoo Yun
Seongjin Shin
Sungdong Kim
James Thorne
Minjoon Seo
Venue: ICLR 2024 poster (2024)
Keywords: automatic evaluation
large language models
llm-as-a-judge
Primary area: datasets and benchmarks
TL;DR: We are the first to train a model specifically for fine-grained evaluation capabilities that performs on-par with GPT-4.

Abstract: Recently, GPT-4 has become the de facto evaluator for long-form text generated by large language models (LLMs). However, for practitioners and researchers with large and custom evaluation tasks, GPT-4 is unreliable due to its closed-source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose PROMETHEUS a fully open-source LLM that is on par with GPT-4’s evaluation capabilities when the appropriate reference materials (reference answer, score rubric) are accompanied. For this purpose, we construct a new dataset – FEEDBACK COLLECTION – that consists of 1K fine-grained score rubrics, 20K instructions, and 100K natural language feedback generated by GPT-4. Using the FEEDBACK COLLECTION, we train PROMETHEUS, a 13B evaluation-specific LLM that can assess any given response based on novel and unseen score rubrics and reference materials provided by the user. Our dataset’s versatility and diversity make our model generalize to challenging real-world criteria, such as prioritizing conciseness, child-readability, or varying levels of formality. We show that PROMETHEUS shows a stronger correlation with GPT-4 evaluation compared to ChatGPT on seven evaluation benchmarks (Two Feedback Collection testsets, MT Bench, Vicuna Bench, Flask Eval, MT Bench Human Judgment, and HHH Alignment), showing the efficacy of our model and dataset design. During human evaluation with hand-crafted score rubrics, PROMETHEUS shows a Pearson correlation of 0.897 with human evaluators, which is on par with GPT-4-0613 (0.882), and greatly outperforms ChatGPT (0.392). Remarkably, when assessing the quality of the generated feedback, PROMETHEUS demonstrates a win rate of 58.62% when compared to GPT-4 evaluation and a win rate of 79.57% when compared to ChatGPT evaluation. Our findings suggests that by adding reference materials and training on GPT-4 feedback, we can obtain effective open-source evaluator LMs.

PROMETHEUS: INDUCING FINE-GRAINED EVALUATION CAPABILITY IN LANGUAGE MODELS
Seungone Kim
Jamin 2hin
Yejin Cho
Joel Jang"
Shayne Longpre
Hwaran Lee
Sangdoo Yun
Seongjin Shin
Sungdong Kim 1.2.3
James Thorne
Minjoon Seo
KAIST AI
NAVER AI Lab
NAVER Cloud
University of Washington
MIT
(seungone, yejin.cho, minjoon)@kaistt.ac ac.kr
jamin.
ABSTRACT
Recently, using powerful proprietary Large Language Model (LLM) (e.g., GPT- 4) evaluator for long-form responses has become the de facto standard. However, for practitioners with large-scale evaluation tasks and custom criteria in consideration (c.g., child-readability)) using proprietary LLMs as an evalua- tor is unreliable due to the closed- source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose PROMETHEUS, a fully open-source LLM that is on par with GPT-4's evaluation capabilities when the appropri- ate reference materials (reference answer, score rubric) are accompanied. We first construct the FEEDBACK COLLECTION, new dataset that consists of IK fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4. Using the FEEDBACK COLLECTION, PROMETHEUS, a 13B evaluator LLM that can assess any given long-form text based customized score rubric provided by the user. Experimental results show that PROMETHIBUS scores Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics, which is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392). Furthermore, measuring cor- relation with GPT-4 with 1222 customized score rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask Eval) shows similar trends, bolstering PROMETHEUS'S s capability as an evaluator LLM.. Lastly, PROMETHEUS achieves the highest accuracy on two human preference benchmarks (HHH Align- ment & MT Bench Human Judgment) compared to spurcsourced reward models explicitly trained on human preference datasets, highlighting its potential as universal reward model. We will open-source our code, dataset, and model!!
train
INTRODUCTION
Evaluating the quality of generreee text has been long-standing challenge in Natural Language Processing (NLP) and remains especially essential in the era of Large Language Models (LLMs) to understand their properties and behaviors (Liang et al., 2022: Chang et al., 2023; Zhong et al., 2023; Chia et al., 2023; Holtzman et al., 2023). Human evaluation has consistently been the predominant method, for its inherent reliability and capacity to assess nuanced and subjective di- mensions in texts. In many situations, humans assessment, such as brevity, creativity, tone, and cultural sensitivities. On the other hand, conven- tional automated evaluation metrics (e.g., BLEU, ROUGE) cannot capture the depth and granularity of human evaluation (Papineni et al., 2002: Lin, 2004b; Zhang et al., 2019; Krishna et al., 2021)
naturally discern the most important factors of
Applying LLMs (e.g. GPT-4) as an evaluator has received substantial attention due to its potential parity with human evaluation (Chiang & yi Lee, 2023; Dubois et al., 2023; Li et al., 2023; Liu et al.,
deeottes equal contribution. Work done while Seungone interning NAVER AI Lab. I 'Coresponning authors
'https://kati github. to/prometheus/

---

Conventional Evaluation
Coarse-Grained Evaluation
Fine-grained Evaluation with User-Defined Score Rubrics
<instruction Set>
Score Rubric: Does the response demonstrate sensitivity? Scare
MMLU, Big Bench, MATH, HumanEval,
Vicuna Bench, MT Bench, AlpacaFarm,
Vicuna MT Bench, AlpacaFarrm
Score
Reference Answer:
Accuracy, EM, Rouge
OPT-4 Evaluation
GPT-4 Evaluation
Prometheus Evaluation
Score specific domain or task
based Helpfuiness/Harmlesssnn
based Customized Score Criteria
Our Proposed Approach
source Nature
Fully Open-source
Only diagnoses about specific domain or task
Only diagnoses about general preference the public
Uncontrolled Versioning
Reproducible Evaluation
Prohibitive Costs
Inexpensive Costs
Is the LLM I'm developing inspiring while being culturally sensitive?
the most humorous one out there?
LLM User
LLM Developer
Figure 1: Compared to conventional, coarse-grained LLM evaluation, we propose a fine-grained approach that takes user-defined score rubrics as input.
2023; Peng et al., 2023; Zheng et al., 2023; Ye et al., 2023b; Min et al., 2023). Initial investigations and observations indicate that, when aptly prompted, LLMS can emulate the fineness of human evaluations. However, while the merits using proprietary LLMs as an evaluation tool are evident, there exist some critical disadvantages:
1. Closed-source Nature:
The proprietary nature of LLMs brings transparency concerns as internal workings are not disclosed to the broader academic community. Such lack of transparency hinders collective academic efforts to refine enhance its evaluation capa- bilities. Furthermore, this places fair evaluation, core tenet in academia, under control of for-profit entity and raises concerns about neutrality and autonomy.
2. Uncontrolled Versioning: Proprietary models undergo version updates that often be- yond the users' purview or control (Pozzobon et al., 2023). This introduces a reproducibil- ity challenge. As reproducibility is cornerstone of scientifie inquiry, any inconsistency stemming from version changes can undermine the robustness research findings that depend on specific versions of the model, especially in the context of evaluation.
3. Prohibitive Costs: Financial constraints associated with LLM APIs
trivial. For example, evaluating four L.MMs variants across four sizes (ranging from 7B to 65B) using GPT-4 on 1000 evaluation instances can cost over $2000. Such scaling costs can pro- hibitive, especially for academic institutions or researchers operating on limited budgets.
Despite these limitations, proprictary LLMs such as GPT-4 are able to evaluate scores based on customized score rubrics. Specifically, current resources are confined to generic, imgle-dimensional evaluation metrics that are either too Nomain/task-specift (e.g. EM, Rouge) or coarse soarse-grained (e.g. helpfuuless/harmlesnness (Dubois et al., 2023; Chiang al., 2023; Liu al., 2023) shown in left-
2

---

side of Figure 1. For instance, AlpacaFarm's (Dubois et al., 2023) prompt gives single definition of preference, asking the model to choose the model response that is generally preferred. However, response preferences are subject to variation based on specific applications and values. In real-world scenarios, users may he interested in customized rubric such as "Which L.LM generates responses that are playful and humorous" "Which LLM answers with particularly care for cultural sensitiv- ities?" Yet, in our initial experiments, we observe that even the largest open-source LLM (70B) is insufficient to evaluate based on a customized score rubric compared to proprietary LLMs.
To this end, we propose PROMETHBUS, a 13B LM that aims to induce fine-grained evaluation ca- pability of GPT-4, while being open-source, reproducible, and inexpensive. We first create the FEEDBACK COLLECTION, a new dataset that is crafted to encapsulate diverse and fine-grained user assessment score rubric that represent realistic user demands (example shown in Figure 2). We de- sign the FEEDBACK COLLECTION with the aforementioned consideration in mind, encompassing thousands of unique preference criteria encoded by user-injected score rubric. Unlike prior feed- back datasets (Ye et al., 2023a; Wang et al., 2023a), it uses custom, not generic preference score rubric, to train models to flexibly generalize to practical and diverse evaluation preferences. Also, to best of our knowledge, we first to explore the importance of including various reference materials particularly the 'Reference Answers' to effectively induce fine- grained evaluation capability.
We use the FEEDBACK COLLECTION to fine-tune Llama-2-Chen 13B in creating PROMETHEUS. On 45 customized score rubrics sampled across three test sets (MT Bench, Vicuna Bench, Feed- back Bench), PROMETHEUS obtains Pearson correlation of 0.897 with human evaluators, which is similar with GPT-4 (0.882), and has a significant gap with GPT-3.5-Turbo (0.392). Unex- pectely, when asking human evaluators to choose feedback with better quality in a pairwise set- ting, PROMETHEUSS was preferred over GPT-4 in 58.67% of the time, while greatly outperformed GPT-3.5-Turbo with a 79.57% win rate. Also, when measuring the Pearson correlation with GPT-4 evaluation across 1 1222 customized score rubrics across 4 test sets (MT Bench, Vicuna Bench, Feed- back Bench, Flask Eval), PROMETHRUS showed higher correlation compared to GPu--35-Turbo and Llama-2-Chat 70B. Lastly, when testing on 2 unseen human preference datasets (MT Bench Human Judgments, HHH Alignment). PROMETHEUS outperforms two state-of-the-ar reward models and GPT-3.5-Turbo, highlighting its potential an universal reward model.
2 RELATED WORK
Reference- based text evaluation Previously, model-free scores that evaluate machine-es geenrrteeee text based a golden candidate reference such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004a) scores were used as the dominant approach. In recent years, model-based approaches have been widely adopted such BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021) which are able to capture semantic information rather than only evaluating on lexical components. Recently, Krishna al. (2021) reported limitations in reference- based metrics, such as ROUGE, observing that they are not reliable for evaluation.
LLM-based text evaluation Recent work has used GPT-4 or fine-tuned critique LLM as an evaluator along a single dimension of "preference" (Chiang & yi Lee, 2023; Li et al., 2023; Dubois al., 2023; Zheng et al., 2023; Liu et al., 2023). For instance, AlpacaFarm (Dubois et al., 2023) asks the model to select "which response is better based on your judgment and based on your own pref- erence" Another example is recent work that showed ChatGPT can outperform crowd-workers for text-annotation tasks (Gilardi et al., 2023; Chiang & yi Lee, 2023). Wang et al. (2023b) introduced PandaLM, a fine-tuned LLM to evaluate the generated text and explain its reliability on various preference datasets. Similarly, Ye et al. (2023a) and Wang et al. (2023a) create critique LLMs. However, the correct preference is often subjective and depends on applications, cultures, and ob- jectives, where degrees of brevity, formality, honesty, creativity, and political tone, among many other potentially desirable traits that may vary (Chiang & yi Lee, 2023). While GPT-4 unreliable due to its close-source nature, uncontrolled versioning, and prohibitive costs, it was the only option explored for fine- mee-griined and customized evaluation based on the score rubric (Ye et al., 2023b). On the contrary, train, to best of our knowledge, the first evaluator sensitive to thousands of unique preference criteria, and show it significantly outperforms uni dimeensiiio preference evaluators in number of realistic settings. Most importantly, compared previous work, we strongly argue
3

---

Instruction
Response Evaluate
Banking FinTech Startup Statog//
Develop me a business strategy for creating FiniTchh startup that offers online baniing services.
Prometheus
Exacurive Sunmary. tus, "Digtalannc"
Industry providing innovative L--]
Customized Score Rubric
Doss the response consider regulations and compliance requirements?
Answer
regulations respense disregard
Exteblaatingg Fin Truch sla tup that offera unline canking servicass retpiiess compratennsove stratagy tackie
Feedloack
Becore 21 The reaco ise shows minimal
challenges osportunties. capitalze [d]
Induetry
The response provides comprehensive overview. L.d
awareness reculattry comp plannos,
[Scoee 3] The repoote recogrizes pertain regulatory requieeennt tts but backs depii.
Reguiesory Research: [.]
Overall, it includes a detalled pian of how start FinTech startup by presenting imaginary startup "Digital Bankk" I.d]
Product Development:
[Score 4)
considers
Service Oifferancatin: Lod
regulstory compii ance. with only minor eversights.
5. Markat Stretegy: Lol
7. Suctanebiilty and Truah [-]
However, deeper dive regulatory complence.
y socstunta and inoug regu wtory complience lance requiremnnts
[Soore
These items are importent aspects to
l.dd
condider when ri a
Score
online
out of
Figure 2: The individual components of the FEEDBACK COLLECTION. By adding the appropriate reference materials (Score Rubric and Reference Answer) and training GPT-4's feedback, show that we could obtain a strong open-source evaluator LM.
Table 1: Information about our training dataset FEEDBACK COLLECTION. Note that there are 20 instructions accompanied for each score rubric, leading to a total number of 20K. Also, there is a score 1-5 response and feedback for each instruction, leading to a total number of 100K.
Evaluation Mode
Data
# Score Rubrics
# Instructions & Reference Answer
Responses & Feedback
(Fine-grained & Castomized)
Total 20K
Total 100K15 for each instraction: 20K cach score within 1-5)
Absolute Evaluation
(20 for each score rubric)
the importance of appending reference materials (score rubric and reference answer) in addition to fine-tuning on the feedback in order to effectively induce fine-grained evaluation capability.
3 THE FEEDBACK COLLECTION DATASET
While previous work has demonstrated that fine-tuning on feedback is effective for improving LMs to function as a critique (Ye et al., 2023a; Wang et al., 2023a), the datasets used in previous work not directly applicable for improving LMs to function introduce the FREDBACK COLLECTION, a new dataset for the sole purpose of fine-tuning an open- sourced evaluator LLM. Our 4 main considerations during dataset construction are: (1) including as many reference materials (i.e. reference answer, and scoring rubric) as possible, (2) maintaining a uniform length among the reference answers for each score (1 5) to prevent undesired length bias, (3) maintaining a uniform score distribution to prevent undesired decision bias, and (4) limiting the scope of our instructions and responses to realistic situations where user is interacting with LLM.
fine- fine-grained evaluator. We thus
Taking these into consideration, we construct each instance within the FEEDBACK COLLECTION encompass four components for the input (instruction, response to evaluate, customized score rubric, reference answer) and two components in the output (feedback, score). An example of an instance is shown in Figure and the number of each component is shown in Table
The four components for the input are as follows:
1. Instruction: An instruction that a user would prompt to an arbitrary LLM.
2. Response to Evaluate: A response the instruction that the evaluator LM has to evaluate.
4

---

3. Customized Score Rubric: A specification of novel criteria decided by the user. The eval- uator should focus on this aspect during evaluation. The rubric consists of (1) a description of the criteria and (2) description of each scoring decision (I to 5).
4. Reference Answer: A reference answer that would receive a score of Instead of re- quiring the evaluator LM solve the instruction, it enables the evaluator use the mutual information between the reference answer and the response to make a scoring decision.
The two components for the output are as follows:
1. Feedback: A rationale of why the provided response would receive particular score. This is analogous to Chain-of-Thoopee (CoT), making the evaluation process interpretable.
2. Score: An integer for the provided response that ranges from 1 to 5.
Each instance has accompanying scoring rubric and reference answer upon the instruction in order to include as much reference material as possible. Also, we include an equal number of 20K instances for each score in the range of to 5, preventing undesired decision bias while training the evaluator LLM. A detailed analysis of the FEEDBACK COLLECTION dataset is in Appendix D.
3.1 DATASET CONSTRUCTION PROCESS
We construct large-scale FEEDBACK COLLECTION dataset by prompting GPT-4 Specifically, the collection process consists of (1) the curation of 50 initial seed rubrics, (2) the expansion of 1K new score rubrics through GPT-4, (3) the augmentation of realistic instructions, and (4) the augmentation of the remaining components in the training instances (i.c. responses including the reference answers, feedback, and scores). Figure shows the overall augmentation process.
Step 1: Creation of the Seed Rubrics We begin with the creation of a foundational seed dataset of scoring rubrics. Each author curates detailed and fine ane-grained scoring rubric that each personnel considers pivotal in evaluating outputs from LLMs. This results in an initial batch of 50 seed rubrics.
Step 2: Augmenting the Seed Rubrics with GPT-4 Using GPT-4 and initial seed rubrics, expand the score rubrics from the initial 50 to a more robust and diverse set of 1000 score rubrics. Specifically, by sampling 4 random score rubrics from the initial seed, we use them demonstra- tions for in-context learning (ICL), and prompt GPT-4 to brainstorm a new novel score rubric. Also, we prompt GPT-4 paraphrase the newly generated rubrics order to ensure PROMETHEUS could generalize to the similar score rubric that uses different words. We iterate the brainstorming - paraphrasing process for rounds. The detailed prompt used for this procedure is in Appendix
Step 3: Crafting Novel Instructions related to the Score Rubrics With a comprehensive dataset of 1000 rubrics our disposal, the subsequent challenge was to craft pertinent training instances. For example, a score rubric asking "Is it formal enough to send to my boss" is not related to math problem. Considering the need for a set of instructions closely related to the score rubrics, we prompt GPT-4 to generate 20K unique instructions that are highly relevant to the given score rubric.
Step Crafting Training Instances Lastly, we sequentially generate response evaluate and corresponding feedback by prompting GPT-4 to generate each component that will get score of i (1 i S 5). This leads to 20 instructions for each score rubric, and responses & feedback for each instruction. To eliminate the effect of decision bias when fine-tuning our evaluator LM, we generate an equal number of 20K responses for each score. Note that for the response with score of we generate two distinctive responses
could one of them as input (reference answer).
3.2 FINE-TUNING AN EVALUATOR LM
Using the FEEDBACK COLLECTION dataset, we fine-tune Llama-2-Chat (7B 13B) and obtain PROMETHEUS to induce fine-grained evaluation capability. Similar to Chain-of-Thought Fine- tuning (Ho et al., 2022: Kim et al., 2023a), we fine-tune to sequentially generate the feedback and then the score. We highlight that it is important to include a phrase such [RESUUT]]' in be- tween the feedback and the score to prevent degeneration during inference. We include the details of fine-tuning and inference in Appendix F.
5

---

Pearson Correlation Between LLM Evaliuators and Human Evailuators Scores
GPT-3.5 turbo
367
0:2
GPT-4
9.924
0.883
0.717
Prometheus
0.927
Overall
Feedback Collection (Unseon)
MT Bench
Vicuna Bench
Figure 3: The Pearson correlation between scores from human annotators and the score from GPT- 3.5-Turbo, Prometheus, and GPT-4 on 45 customized score rubries from the Feedback Bench, Vi- cuna Bench, and MT Bench. PROMETHEUS shows high correlation with human evaluators.
ChatOPT
Prometheus
Een aot
Pramethous
GPT-4
Court
Figure 4: Pairwise comparison of the quality the feedback generated by GPT-4, PROMETHEUS and GPT-3.5-Turbo. Annotators are asked to choose which feedback is better at assessing the given response. PROMETHEUS shows a win-rate of 58.62% over GPT-4 and 79.57% over EPT-3.5-Turbo
EXPERIMENTAL SETTING: EVALUATING AN EVALUATOR LM
In this section, we explain our experiment setting, including the list of experiments, metrics, and baselines that we use to evaluate fine-grained evaluation capabilities of an evaluator LLM. Compared to measuring the cstruuttionfoolloinng capability of a LLM, it is not straightforward to directly measure the capability to evaluate. Therefore, standard and measure how similarly our evaluator model and basclines could closely simulate them. We mainly employ two types of evaluation methods: Absolute Grading and Ranking Grading.
human evaluation and GPT-4 evaluation
Our experimental setting including "List Experiments", "Metrics" and "Baselines" are in Ap- pendix B.
EXPERIMENTAL RESULTS
5.1 CAN PROMETHEUS CLOSELY SIMULATE HUMAN EVALUATION?
Correlation with Human Scoring We first compare the correlation between human annotators and our baselines using 45 instances each with unique customized score rubric, namely the FEEDBACK BENCH (Unseen Score Rubric subset), MT Bench (Zheng et al., 2023), and Vicuna Bench (Chiang al., 2023). The results shown in Figure showing that PROMETHEUS is

---

GPT-4 Vs Prometheus
100
5.71%
condistent
too general and abstract
overly optimistic
14 00%
not relevant to the response
43%
overly cribical
unrelated to the score rubric
14. 00%
Percentage
6.0
18 00%
11.43%
34 29%
44 60%
20
14.29%
2.00%
2.86%
GPT-4
Prometheus
Figure The reason why GPT-4's or Prometheus's feedback was not chosen over the other. PROMETHEUS generates less abstract and general feedback, but tends to write overly critical ones.
on par with GPT-4 across all the three evaluation datasets, where PROMETHEUS obtains a 0.897 Pearson correlation, GPT-4 obtains 0.882, and GPT-3.5-Turbo obtains 0.392.
Pairwise Comparison of the Feedback with Human Evaluation To validate the effect of whether PROMETHEUS generates helpful/meaningguu feedback in addition to its scoring decision, ask human annotators to choose better feedback. The results are shown in Figure 4, showing that PROMETHEUS is preferred over GPT-4 88.62% of the times, and over GPT-3.5-Tun Turbbb 79.57% of the times. This shows that PROMETHEUS' 's feedback is also meaningful and helpful.
Analysis of Why Prometheus's Feedback was Preferred In addition to a pairwise comparison of the feedback quality, we also conduct an analysis asking human annotators to choose why they pre- ferred one feedback over the other by choosing at least one of the comprehensive 6 options ('rejected feedback is not consistent with its score" "too general and abstract" "overly optimistic" "not rel- evant to the response" / "overly critical" / "unrelated to the score rubric"). In Figure 5, we show the percentage why each evaluator LLM (GPT-4 and PROMETHEUS) was rejected. It shows that while GPT-4 was mainly not chosen due providing general or abstract feedback, PROMETHEUS was mainly not chosen because was too critical about the given response. Based on this result, we conclude that whereas GPT-4 tends to be more neutral and abstract, PROMETHEUS shows clear trend of expressing its opinion of whether the given response is good or not. We conjecture this an effect of directly fine-tuning PROMETHEUS ONLY perform fine-grained evaluation, essentially converting it to an evaluator rather than generator. We include ing "PROMETHEUS GPT-3.5-Turbo" and "GPT-4 vs GPT-3.5-Turbo" in Appendix H and (2) detailed explanation of the experimental setting of human evaluation in Appendix M.
(1) additional results analyz-
5.2 CAN PROMETHEUS CLOSELY SIMULATE GPT-4 EVALUATION??
Correlation with GPT-4 Scoring We compare the correlation between GPT-4 evaluation and our baselines using 1222 score rubrics across 2360 instances from the FEEDBACK BENCH (Seen and Unseen Score Rubric Subset), Vicuna Bench (Chiang al., 2023), MT Bench (Zheng al., 2023), and Flask Eval (Ye et al., 2023b) in an absolute grading scheme. Note that for the FEEDBACK BENCH, measure the correlation with the scores augmented from GPT-4-0613, and for the other 3 datasets, we measure the correlation with the scores acquired by inferencing GPT-4-0613,
The results these benchmarks shown across Table and Table 3.

---

Table 2: Pearson, Kendall-Tau Tau, Spearman correlation data generated by GPT-4-0611 0-143. All scores were sampled across inferences. The best comparable statistics are bolded and second best underlined.
FEEDBACK COLLECTION TEST SET (GENERATED BY GPT-4-0613)
Evaluator LM
SEEN CUSTOMIZED RUBRICS
UNSEEN CUSTOMIZED RUBRIC
Pearson
Kendall Tau
Spearman
Pearson
Kendall Tau
Spearman
LLAMA2-CHAT 7B
0.485
0.422
0.478
0.463
0.402
0.465
LLAMA2-CHAT 13B
0.387
0.452
0.450
0.379
0.431
LLAMA2-CHAT 70B
0.572
0.491
0,564
0.558
0.477
0.549
LLAMA2-CHAT 13B COARSE.
0.482
0.406
0.475
0.454
0.361
0.427
PROMETHEUS 7B
0.860
0.781
0.863
0.847
0.767
0.849
PROMETHEUS 13B GPT-3.5-TURBO-661
0.861
0.776
0.858
0.860
0.771
0.858
0.636
0.536
0.617
0.563
0.453
0,521
GPT-4-0314
0.754
0.671
0.762
0.753
0.673
0.761
GPT-4-0613
0.742
0.659
0.747
0.743
0,660
0.747
GPT-4 (RECENT)
0.745
0.659
0.748
0.733
0.641
0.728
Table 3: Pearson, Kendall-Tau. Spearman correlation with scores sampled from GPT-4-0613 across infer- ences. Note that GPT-4-0613 was sampled 6 times in total to measure self-consistency. The best comparable statistics are bolded and second best underlined among baselines. We include GPT-4 as reference to show it self-consistency when inferenced multiple times.
VICUNA BENCH
MT BENCH
FLASK EVAL
Evaluator I.
Pearson
Kendall Tau
Spearman
Pearson
Kendall Tau
Spearman
Pearson
Kendall Tau
Spearman
LLAMA2-CHAFF 7B
0.175
0.143
0.176
0.132
0.113
0.143
0.271
0.180
0.235
-0.000
40.029
LLAMA2-CHAF 70B
0.376
0,318
0.391
0,226
0.175
0.221
0,336
0.267
0.346
LLAMA2-CHAT 13B COARSE.
0.307
0.196
0.215
0.117
0.328
0.400
0,517
0.349
0.451
PROMETHEUS-7B
0.457
0.293
0.285
PROMETHEUS-138
0.466
0.346
0.429
0.473
0.341
0.451
0,467
0.345
0.455
0.270
0.187
0.232
0.275
0.202
0.267
0.422
0.299
0.371
GPT-4-0314
0.775
0.857
GPT-4-0613
0.925
0.783
0.864
0.952
0.834
0.927
0.835
0.672
0.798
GPT-4 (RECENT)
0.932
0.801
0.877
0.944
0.812
0.914
0.832
0.667
0.794
In Table 2, the performance of LLAMA-2-CIAT 13B degrades over the 7B model and slightly improves when scaled up to 70B size, indicating that naively increasing the size model does not necessarily improve an LLM's evaluation capabilities. On the other hand, PROMETHEUS 13B shows a +0.420 and +0.397 improvement over its base model LLAMA2-CHAT 13B in terms of Pearson correlation on the seen and unseen rubric set, respectively. Moreover, it outperforms LLAMA2-CHAT 70B, the high performance of PROMETHEUS is mainly because the instructions and responses within the test set might share a similar distribution with the train set we used (simulating a scenario where user interacting with LLM) even if the score rubric holds unseen. Also, training on feed- back derived from coarse -rarined score rubrics (denoted as LLAMA2-CHAA 13B + COARSE) only slightly improves performance, indicating the importance of training on wide range of score rubric is important to handle customized rubrics that different L.M user or researcher would desire.
and different versions of GPT-4. We conjecture
In Table 3, the trends of LLAMA2-CHE A-TT among different sizes hold similar; simply increasing size does not greatly improve the LLM's evaluation capabilities. On these benchmarks, PROMETHEUS shows a +0.255, +0.493, and +0.202 improvement over its base model LLAMA2-CHAN 13B in terms of Pearson correlation the Vicuna Bench, MT Bench, and Flask Eval dataset, respectively. While PROMETHEUS outperforms LLAMA2-CHA. 70B and GPT-3.5-TURBO- 0613, it lacks be- hind GPT-4. We conjecture that this might be because the instructions from the FEEDBACK COL- LECTION and these evaluation datasets have different characteristics, the FEEDBACK COLLECTION relatively long and detailed (e.g., I'm city planner ... I'm looking for novel and progressive
8

---

Table 4: Human Agreement accuracy among ranking datasets. The best comparable statistics are bolded.
HHH ALIGNMENT
MT BENCH HUMAN JUDG.
Evaluator LM
Help.
Harm
Hon.
Other
Total Avg.
Human Preference
RANDOM
50.00
50.00
50.00
50.00
50.00
34.26
STANFORDNLP REWARD MODEL
69,49
60.34
52,46
51.16
58,82
44.79
ALMOST REWARD MODEL
74.58
67.24
78.69
86,05
76,02
49.90
LLAMA2-CHAT
66.10
81.03
70.49
74.42
72.85
51.78
LLAMA2-CHAT 13B
74.58
87.93
55.74
79.07
73.76
52.34
LLAMA2-CHAY 70B
66.10
89.66
67.21
74.42
74.21
53.67
LLAMA2-CHAT 13B + COARSE.
68.74
68.97
65.57
67.44
67.42
46.89
76.27
87.93
67.21
86.05
78.73
57.12
PROMETHEU'S 7B
69.49
84.48
78.69
90,70
80.09
55.14
PROMETHEU' 13B
81.36
82.76
75.41
76.74
79.19
57.72
GPT-4-0613
91.53
93.10
85.25
83.72
88.69
63.87
solution to handle traffic congestion and air problems derived from population increase), while the datasets used for evaluation hold short (e.g., Can you explain about quantum mechanics?).
On the other hand, it is important note that the Flask Eval dataset, LLAMA2-CHAT 13B COARSE (specifically trained with the Flask Eval dataset) outperforms PROMETHEUS. This indi- cates that training directly the evaluation dataset might be the best option to acquire task-specific evaluator LLM, and we further discuss this in Section C.5.
CAN PROMETHEUS FUNCTION AS A REWARD MODEL.
We conduct experiments human preference datasets: HHH Alignment (Askell et al., 2021) and MT Bench Human Judgment (Zheng et al., 2023) that use ranking grading scheme. Table 4, results show that prompting LLAMA-2-0 2-CHAT surprisingly obtains reasonable performance, which we conjecture might be the effect of using a base model that is trained with Reinforcement Learn- ing from Human Feedback (RLHF). When training feedback derived from coarse -rarineg score rubrics (denoted as LLAMA2-CHA. 13B + COARSE), only hurts performance. On the other hand, PROMETHEUS 13B shows +5.43% and +5.38% margin over its base model LLAMA2-CHAT--I on the HHH Aligmment and MT Bench Human Judgement dataset, respectively. These results are surprising because they indicate that training on an absolute grading scheme could also improve performance on ranking grading scheme even without directly training on ranking evaluation in- stances. Moreover, shows the possibilities of using generative LLM (PROMETHEUS) as a reward model for RLHF (Kim et al., 2023b). We leave the exploration of this research to future work.
7 CONCLUSION
In this paper, we discuss the possibility of obtaining an open-source LM that is specialized for fine- grained evaluation. While text evaluation an inherently difficult task that requires multi-faceted considerations, we show that by incorporating the appropriate reference material, we can effectively induce evaluation capability into an LM. We propose a new dataset called the FEEDBACK COL- LECTION that encompasses thousands of customized score rubrics and train an open-source evalu- ator model, PROMETHEUS. Surprisingly, when comparing the correlation with human evaluators, PROMETHEUS obtains a Pearson correlation on par with GPT-4, while the quality the feedback was preferred over GPT-4 58.62% of the time. When comparing Pearson correlation with GPT-4, PROMETHEUS shows the highest correlation even outperforming GPT-3.5-Turbo. Lastly, we show that PROMETHEUS shows superior performance bility as an universal reward model. We hope that our work could stimulate future work on using open-source LLMs evaluators instead of solely relying proprietary L.MSs.
human preference datasets, indicating its possi-

---

ACKNOWLEDGMENTS
This work was partly supported by KAIST-NAVER Hypercreative AI Center and Institute of In- formation & communications Technology Planning & Evaluation (IIPP) grant funded by the Korea government (MSIT) 2N0.2022-0-00264, Comprehensive Video Understanding and Generation with Knowledge-based Deep Logic Neural Network, 40%; 1--2021--00208, Artificial Intelligence In- novation Hub, ).%). We thank Minkyeong Moon, Geonwoo Kim, Minkyeong Cho, Yerim Kim, Sora Lee, Seunghwan Lim, Jinheon Lee, Minji Kim, and Hyorin Lee for helping with the human evaluation experiments, We thank Se June Joo, Dongkeun Yoon, Doyoung Kim, Seonghycon Ye, Gichang Lee, and Yehbin Lee for helpful feedback and discussions.
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Jackson Kermion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant labora- tory for alignment, 2021.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv.2307.03109 2023.
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuncl large language models. arXiv preprint arXiv:2306.0475/ 2023.
Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative to human evalu- ations?, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
/ wilmsys.crg/bllee 2023- 03 30 vicuna/.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14388 2023.
Fabrizio Gilardi, Meysam Alizadeh, and Maäl Kubli. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint arXiv:2303. 15056, 2023.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. arXiv preprini arXiv:2212. 2022.
Ari Holtzman, Peter West, and Luke Zettlemoyer. Generative models as a complex systems science: How can we make sense of large language model behavior? arXiv preprint 00080, 2023.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.0966 2022.
Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305. 14045, 2023a.
Sungdong Kim. Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak. Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13335. 2023b.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles to progress in long-form forg-frmmat question answering. 2021.
arXiv preprint
10

---

Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction for.outtiioooooooiinng models. http:///...uu..cocoo/taaaaaaaaaa
evall 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar. et al. Holistic evaluation of language models. arXiv preprint arXi::2211.0911 2022.
Chin-Yew Lin, ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004a. Association for Computational Linguis- tics. URL https: : htt///acaaattooooooo wwgg//rgggeoo 00/0/004-00111
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004b.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303, 16634 2023,
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine- fine-grained atomic evaluation of factual precision long form text generation. arXiv preprint arXiv:2305. 14251, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318 2002.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint urXiv:2304003277, 2023.
Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. On the challenges of using black- box apis for toxicity evaluation in research, 2023.
Thibault Sellam, Dipanjan Das. and Ankur P. Parikh. Bleurt: Learning robust metrics for text generation. In Annual Meeting of the Association for Computational Linguistics, 2020. URL
https:/ /tap:///ppppi wwweeemaniiiie icscholarcc....occc
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O' Orire,n Ramakanth Pasunuru, Jane Dwivedi-YY Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308. 04592, 2023a.
Yidong Wang, Zhuohao Yu, Zhengran Zeng. Linyi Yang. Cunxiang Wang. Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang. Xing Xie, et al. Pandalm: An automatic evaluation benchmark for 11m instruction tuning optimization. arXiv preprint arXiv:2306.0500: 05087, 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi. and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint ar 222 2022.
11

---

Seonghyeon Ye, Yongrae Jo. Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising Ilm empowered by self-feedback generation. Blog post, May 2023a. URL https///aaastaa.iiiooa
SelFee/.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thome, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets, arXiv preprint arXiv:2307.10928, 2023b.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. BARTScore: Evaluating generated text as text generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan Advancess Neurul Information Processing Systems, 2021. URL hnttps://openreyee
Advances in cent/fooum
id=5Ya8Pbvp29.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 09675, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0568. 05685, 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXhv:2304.03366 2023.
12

---

Feedback Collection Score Rubrics (1K)
Eneckelly Yound Resorts
Seed Score Rubrics
Froxctiv Claniication Seeking
Is the answer explained like formal proof?
Tore Style Motthatian
any
formal
fo or Colead
Moraglion
li formal proct.
trador ally Gatd denate Comm aiiiii
Does the response uttiize appropetate profersional jargon and terminology sutte export
Pnoeutta Aekieeeeginn veeeeeeegnc
weniaeeniin Lamper
Aesich Rechenddig
[foor The repponte rituaee terms avoles professional
la anguage antiiely.
Tadneaa Naskly
perfectly professional
terms, oriu accuracy and comprsherssil belly for experts
Feedback Collection Instance Augmentation
Customizod Seore Rubric
Instruction
Referance Ansuer (Score 5) Responses Evaluate
Feedback
Is the answer written pr professioonaly nally and farmally,
am entry lovel emp
mu lination corpor alion have been aseed to write
you
The provides helpfu guide asgroaching task, bul caulc more pioflesione In tone and phrosing.
3
rsport
Find, reawarch the latest stuff in your industry. Than create en out to keep your thoughis organined in roport,
[Scove The ar tweer lacks any sonse of professionalsmm
om unsure how structurio the repori and level of formallly required The report will read by my pobentially
informal
pogo
read the super visce CEO.
supernisct,
conclusion,
It does, howerer provide clear cutine exampii
(score The enswer complenely professional and suliable format welling
keep your language formal, but don't too siift headings subhoodings, add charts
thould
Can you glvw me sample of
how the recort
Score
abould
Sc. the overall score
Figure 6: Overview of the augmentation process of the FEEDBACK COLLECTION. The keywords included within the score rubrics of the FEEDBACK COLLECTION also displayed.
A OVERALL ILLUSTRATION OF THE DATA AUGMENTATION PROCESS
Figure shows the augmentation process of the FEEDBACK COLLECTION.
B EXTENSION OF EXPERIMENTAL. SETTING: EVALUATING AN EVALUATOR
LLM
Detailed information the datasets used for the experiment is included in Table 5.
B.1 LIST OF EXPERIMENTS AND METRICS
Absolute Grading We first test in an Absolute Grading setting, where the evaluator LM should generate feedback and score within the range of to 5 given an instruction, response to evaluate, and reference materials (as shown in Figure 2). Absolute Grading is challenging compared to Rank- ing Grading since the evaluator LM does not have access to opponent to compare with and it is required to provide a score solely based on its internal decision. Yet, is more practical for users since it relieves the need to prepare opponent compare with during evaluation.
13

---

Table 5: Information about the datasets we use test evaulator LMs. Note that FEEDBACK BENCH is dataset that is crafted with the exact same procedure as the FEEDBACK COLLECTION as ex- plained in Section 3.1. We include additional analysis of FEEDBACK BENCH in Appendix E. Simu- lated GPT-4 denotes GPT-4 prompted to write score of (1 S S 5) during augmentation.
Evaluation Mode
Evaluation Data
Source/ Types of Score Rubric
Response LMs
PERDBACK BENCH (Seen Rubric)
Machine Generated
Simluated GPTT41
FREDBACK BENCH (Unseen Ruhric)
50 Hand Crafted
Absolute Evaluntion
Vicuna Bench
80 Hand Crafted
WizardLM. Vicuna, Llama2-Char, ChasGPT
MT Bench
Flask Eval
Logical Thinking (3), Background K Knowledge (2)
Problem Handling (4), User Alignmcct (3)
Alpaca, Vicuna, Band, ChaGGPT
MT Bench Human Judgments
Human Preference (Helpfulness)
Alpaca, Llama, Vicuna, CharGP Claude-v1, GPT-
Ranking Evaluation
Helpfalness, Harmlessness,
HHH Alignment
Honesty, Other
Human Ansotation
We mainly conduct three experiments in this setting: (1) measuring the correlation with human evaluators (Section 5.1), (2) comparing the quality of the feedback using human evaluation (Sec- tion 5, and (3) measuring the correlation with GPT-4 evaluation (Section 5.2). For the experiments that measure the correlation, we use 3 different correlation metrics: Pearson, Kdendall- Spearman. For measuring the quality of the generated feedback, we conduct a pairwise compari- son between the feedback generated by PROMETHEUS, GPT-3.5-Turbo, and GPT-4, asking human evaluators to choose which has better quality and why they thought so. Specifically, we recruited 9 crowdsource workers and split them into three groups: PROMETHEUS vs GPT-4, PROMETHEUS vs ChatGPT, and GPT-4 vs ChatGPT, The annotators are asked to answer the following three questions:
and
1. What score would you give to the response based on the given score rubric?
2. Among the two Feedback, which better for critiquing the given response?
3. Why did you reject that particular feedback?
We use the following four benchmarks to measure the correlation with human evaluation and GPT-4 evaluation. Note that FEEDBACK BENCH is a dataset generated with the same procedure as the FEEDBACK COLLECTION, and is divided into two subsets (Seen Rubric and Unseen Rubric).
The Seen Rubric subset shares the same IK score rubrics with the FEEDBACK COLLECTION across 1K instructions (1 per score rubric). The Unseen Rubric subset also consists of 1K new instructions but with 50 new score rubrics that are generated the same way as the training set. Details are included in Appendix
FEEDBACK BENCH:
Vicuna Bench: We adapt the 80 test prompt set from Vicuna (Chiang et al., 2023) and hand-craft customized score rubrics for each test prompt. In order to obtain reference answers, we concatenate the hand-crafted score rubric and instruction to prompt GPT-4
MT Bench:
We adapt the 80 test prompt set from MT Bench (Zheng al., 2023), a multi- turn instruction dataset. We hand-craft customized score rubrics and generate a reference answer using GPT-4 for each test prompt well. Note that we only use the last turn of this dataset for evaluation, providing the previous dialogue as input to the evaluator LM.
We adapt the 200 test prompt set from FLASK (Ye et al., 2023b), a fine- grained evaluation dataset that includes multiple conventional NLP datasets and instruction datasets. We use the 12 score rubrics (that are relatively coarse-graie saarse-rrained compared to the IK score rubrics used in the FEEDBACK COLLECTION) such as Logical Thinking, Background Knowledge, Problem Handling, and User Alignment.
FLASK Eval:
Ranking Grading To test an evaluator LM trained only on Absolute Grading could be utilized as universal reward model based on any criteria, we existing human preference benchmarks and use accuracy as our metric. Specifically, we check whether the evaluator LM could give a higher score to the response that is preferred by human evaluators. The biggest challenge of employing an
14

---

evaluator LM trained in Absolute Grading setting and testing it on Ranking Grading was that could give the same score for both candidates. Therefore, we use a temperature of 1.0 when evaluating each candidate independently and iterate until there is winner. Hence, it's noteworthy that the settings are not exactly fair compared to other ranking models. This setting is NOT designed to claim SOTA position in these benchmarks, but is conducted only for the purpose of checking whether an evaluator LM trained in an Absolute Grading setting could also generalize in a Ranking Grading setting according general human preference. Also, in this setting, we do not provide reference answer to check whether PROMETHEUS could function reward model. We use the following two benchmarks to measure the accuracy with human preference datasets:
MT Bench Human Judgement: This data is another version of the aforementioned MT Bench (Zheng et al., 2023). Note that it includes a tie option as well and does not require criteria.
iterative inference to obtain a clear winner. We use Human Preference
Introduced by Anthropic (Askell et al., 2021), this dataset (221 pairs) is one of the most widely chosen reward-model test-beds that measures preference accuracy in Helpfulness, Harmlessness, Honesty, and in General (Other) among two response choices.
НHH Alignment:
B.2 BASELINES
The following list shows the baselines we used for comparison in the experiments:
(Touvron et al., 2023): The base model of PROMETIEUS when fine-tuning on the FEEDBACK COLLECTION. Also, it is considered as the best option among the open-source LLMs, which we use as an evaluator in this work.
LLAMA2-CHAT-(7.13
To analyze the effectiveness of training on thousands of fine-grained score rubrics, we train a comparing model only using 12 coarse-grained score rubrics from Ye et al. (2023b). Detailed information on this model is in Appendix G.
LLAMA-2-CHAT-IE 13B COARSE:
GPT-3.5-TURBO-U 3: Proprietary LLM that offers cheaper price when employed as an evaluator LLM. While it is relatively inexpensive compared to GPT-4, it still has the issue of uncontrolled versioning and close-source nature.
GPT-4-(0311.0613 RECENT]: One of the most powerful proprictary LLM that is con- sidered the main option when using L.MS as evaluators. Despite its reliability as an eval- uator LM due to its superior performance, it has several issues of prohibitive costs, uncon- trolled versioning, and close-source nature.
STANFORDNLP REWARD MODEL
One of the state-of-the-art reward model directly trained on multiple human preference datasets in ranking grading setting.
ALMOST REWARD MODEL (Kim et al., 2023b): Another state-of-the-an reward model trained synthetic preference datasets in ranking grading setting.
C EXTENSION OF DISCUSSIONS AND ANALYSIS
C.I WHY IS IT IMPORTANT TO INCLUDE REFERENCE MATERIALS?
Evaluating a given response without any reference material is a very challenging task (i.e., Directly asking to decide score only when an instruction and response given), since the evaluation LM should be able to (1) know what the important aspects tailored with the instruction is, (2) inter- nally estimate what the answer of the instruction might be, and (3) assess the quality of responses based on the information derived from the previous two steps. Our intuition is that by incorporating each component within the reference material, the evaluator LM could solely focus on assessing the quality of the response instead of determining the important aspects or solving the instruction. Specifically, we analyze the role of each component as follows:
Score Rubric:
Giving information of the the pivotal aspects essential for addressing the given instruction. Without the score rubric, the evaluator LM should inherently know what details should be considered from the given instruction.
httpps://hugiiiggfffcs co/ wtt.c/////ooonnnnpoooooooocooc wtwprrddnnn//ttttssst-sese flan-t5-x1
15

---

Table Pearson, Kendall- Tau, Spearman correlation with data generated by GPT-4-061 4-0613 (Feedback Collec- tion Test set) and scores sampled from GPT-4-0613 across 3 inferences (Vicuna Bench).
FEEDBACK COLLECTION TEST SET
VICUNA BENCH
Evaluator LM
Seen Score Rubric
Unseen Score Rubric
Pearson
Pearson
Pearson
PROMETHEUS 7B
0.860
0.847
0.457
Training Ablation
w/O SCORE RUBRIC
0.837
0.745
0.355
W/O FEEDBACK DISTILLATION
0.668
0.673
0.413
W/O REFERENCE ANSWER
0.642
0.626
0.349
Model Ablation
LLAMA-2 2 7B BASELINE VICUNA-V1.5 7B BASELINE
0.839
0.818
0.404
0.860
0.829
0.430
7B BASELINE
0.823
0.761
0.470
Reference Answer: Decomposing the process of estimating a reference answer and evalu- ating it at the same time into steps. Since the reference answer is given additional input, the evaluator LM could only focus on evaluating the given response. This enables to bypass natural proposition that if evaluator LM doesn't have the ability to solve the problem, it's likely that it cannot evaluate different responses effectively as well.
As shown in Table 6, conduct an ablation experiment by excluding each reference material and also training only on the score rubric without generating feedback. Additionally, we also ablate the effect of using different model variants (Llama-2, Vicuna, Code-Llama) instead of Llama-2-Chat.
C.2 ABLATION EXPERIMENTS
Training Ablation The results indicate that each component contributes orthogonally to PROMETHEUS' 's superior evaluation performance. Especially, excluding the reference answer shows the most significant amount of performance degradation, supporting erence answer relieves the need for the evaluator LM to internally solve the instruction and only focus assessing the response. Also, while excluding the score rubric the FEEDBACK BENCH does not harm performance a lot, the performance drops lot when evaluating on Vicuna Bench. As in our hypothesis, conjecture that order generalize other datasets, the role of providing what aspect to evaluate holds relatively crucial.
claim that including ref-
Model Ablation To test the effect using LLAMA2- AAAAAA2---HA, model that has been instruction-tumi with both supervised tieettunng and RLHF, we ablate by using different models as a starting point. Results show that different model choices do not harm performance significantly, yet model trained with both supervised fine-tuning and RLHF shows the best performance, possibly due to additional training to follow instructions. However, find that using Code-Llama has some benefits when evaluating on code domain, and we discuss the effect on Section C.6.
C.3 NARROWING PERFORMANCE GAP TO GPT-4 EVALUATION
The observed outcomes, in which PROMETHEUS consistently surpasses GPT-4 based on human evaluations encompassing both scores and quality of feedback, well correlations in the FEED- BACK BENCH, are indeed noteworthy. We finmly posit that these findings are not merely serendipi- tous and offer the following justifications:
16

---

Regarding results on FEEDBACK BENCH, model is directly fine-tuned this data, so it's natural to beat GPT-4 on similar distribution test set it is well-trained. In addition, for GPT-4, compare the outputs of inferencing the instructions and augmenting new instances, causing the self-consistency to be lower.
Regarding score correlation for human evaluation, our model shows similar or slightly higher trends. First, our human evaluation set-up excluded all coding math-related ques- tions, which is where is non-trivial to beat GPTT-y yet. Secondly, there's 's always the margin highlight that
of human error that needs to be accounted for. Nonetheless, first work to argue that an open-source evaluator LM could closely reach GPT-4 evaluation only when the appropriate reference materials are accompanied.
the
As shown in Figure 5, PROMETHEUS tends be critical compared to GPT-4. We conjec- ture this is because since it is specialized for evaluation, it acquires the characteristics of seeking for improvement when assessing responses.
C.4 QUALITATIVE EXAMPLES OF FEEDBACK GENERATED BY PROMETHEUS
We present five qualitative examples to compare the feedback generated by PROMETHEUS and GPT- in Appendix L. Specifically, Figure 16 shows an example where human annotators labeled that GPT-4 generate
abstract/general feedback not suitable for criticizing the response. Figure 17 shows an example where human annotators labeled that PROMETHEUS generate overly critical feed- back. Figure 18 shows an example of human annotators labeled as a tie. In general, PROMETHEUS generates a detailed feedback criticizing which component within the response is wrong and seek improvement. This qualitatively shows that PROMETHEUS could function as an evaluator LM.
Moreover, we present an example of evaluating python code responses using PROMETHEUS, GPT- 4, and Code-Llama in Figure 19. We discuss the effect of using base model specialized on code domain for code evaluation in Section C.6.
C.5 A PRACTITIONER S GUIDE FOR DIRECTLY USING PROMETHIBUS EVALUATION
Preparing an Evaluation Dataset As shown in the previous sections, PROMETHEUS functions as good evaluator LM not only the FEEDBACK BENCH (a dataset that has a similar distribution with the dataset was trained on), but also on other widely used evaluation datasets such as the Vi- cuna Bench, MT Bench, and Flask Eval. As shown in Figure 1, users should prepare the instruction dataset they wish to evaluate their target LLM on. This could either be a widely used instruction dataset custom evaluation users might have.
Deciding Score Rubric to Evaluate on The next step is to choose the score rubric users would want to test their target LLM on. This could be confined to generic metrics such as helpfulness/harm- lessness, but PROMETHEUS also supports fine-grained score rubries such "Child-Safety", "Cre- ativity" or even "Is the response formal enough to send to my boss".
Preparing Reference Answers While evaluating without any reference is also possible, as shown in Table 6, PROMETHEUS shows superior performance when the reference answer is provided. Therefore, users should prepare the reference answer they might consider most appropriate based on the instructions and score rubrics they would want to test on. While this might require additional cost to prepare, there is a clear trade-off in order to improve the precision or accuracy of the overall evaluation process, hence it holds crucial.
Generating Responses using the Target LLM The last step is to prepare responses acquired from the target L.LM that users might want to evaluate. By providing the reference materials (score rubrics, reference answers) along with the instruction and responses to evaluate on, PROMETHEUS generates a feedback and a score. Users could use the score to determine how their target LLM is performing based on customized criteria and also refer to the feedback to analyze and check the properties and behaviors their target LLM. For instance, PROMETHEUS could be used as a good alternative for GPT-4 evaluation while training up with formalized procedure to decide the details of instruction-tuning or RLHF while developing new L.L.M This includes deciding how many training instances to use, how to systematically decide
LLM. Specifically, the field has not yet come
17

---

Distribution of ROUGE-L Scores Within Criteria Descriptions
175000
150000
125000
Frequency
100000
75000
50000
25000
ROUGE Score
Figure 7: Rouge-L score distribution among two randomly sampled score rubrics from the FEED- BACK COLLECTION. A left-skewed distribution with low values shows the score rubrics are diverse.
the training hyperparameters, and quantitatively analyzing the behaviors of L.MS across multiple versions. Most importantly, users might not want send the outputs generated by their LLMs to OpenAl API calls. In this regard, PROMETHEUS provides an appealing solution of having control over the whole evaluation process, also supporting customized score rubrics.
C.6 A PRACTITIONER'S 's GUIDE FOR TRAINING A NEW EVALUATION MODEL
Users might also want to train their customized evaluator LM PROMETHEUS for different use cases. As shown in Table 3, training directly on the Flask dataset (denoted as LLAMA2-CHAN 13B COARSE) shows higher correlation with GPT-4 on the Flask Eval dataset compared to PROMETHEUS that trained on the FEEDBACK COLLECTION. This implies that directly training on target feedback dataset holds the best performance when evaluating on it. Yet, this requires going through the process preparing a new feedback dataset (described in Section 3.1). This implies that there a trade-off between obtaining strong evaluator LM target task and paying the initial cost to prepare a new feedback dataset. this subsection, we provide some guidelines for how users could also train their evaluator LM using feedback datasets.
Preparing a Feedback Dataset to train on described Section 3, some important consider- ations to prepare new feedback dataset are: (1) including as many reference materials as possible, (2) maintaining a uniform length among the reference answers for each score to 5) to prevent un- desired length bias, (3) maintaining uniform score distribution to prevent undesired decision bias. While we did not explore the effect of including other possible reference materials such as a "Score 1 Reference Answer" or "Background Knowledge" due to limited context length, future work could also explore this aspect. The main intuition is that providing more reference materials could enable the evaluator LM solely focus evaluation instead of solving the instruction.
Choosing a Base Model Train an Evaluator LM As shown Figure 19, we find that training on CODE-LLAMA provides more detailed feedback and reasonable score decision when evaluating responses on code domains (7 instances included within the Vicuna Bench dataset). This indicates that choosing different base model based on the domain evaluate might be crucial when de- signing an evaluator LM. We also leave the exploration training an evaluator LM specialized on different domains (e.g., code and math) future work.
18

---

Box plat Average Sentiment by Score
Spearnan Correlation: 0.58
0.75
0.50
Avarage Sentiment
-0.25
-0.50
-1.00
Score
Figure 8: Box and whisker plot for average sentiment per score description. A linearly increasing trend is crucial for the evaluator LM to decide score in Absolute Scoring setting.
Feedback Collection Training Set
1000
Spearman Correlation: 0,12
800
Response Length
400
Score
Figure 9: Box and whisker plot plotting the length distribution responses on each score range. We check whether there is length bias (i.c., higher score given for longer responses).
D ANALYSIS THE FEEDBACK COLLECTION DATASET
In this section, we provide comprehensive analysis of the characteristics of the FEEDBACK COL- LECTION dataset. To ensure the quality, we answer each question one by one, emphasizing our main considerations during the creation of the dataset.
Are the Score Criteria Diverse Enough? Following previous work (Wang et al., 2022: Honovich et al., 2022), we plot the rouge-L distribution between two instances among our whole set of IK score rubrics. Specifically, we use the score criteria (description of the criteria) and measure the rouge-L value between the results indicate that each criteria does not overlap with one another, ensuring that we include many novel score rubries in our training set.
criteria. Figure shows the overall distribution plot. The
19

---

Table 7: Distinct N-gram measured on each component of the training instance. A higher diversity ratio indicates that each component tends to be more diverse.
Distinct N-gram
Bigram Diversity Ratio
Trigram Diversity Ratio
Instruction
0.43
0.79
Reference
0.43
0.82
Score Rubric
0.60
0.81
Responses
0.32
0.77
Feedback
0.26
0,66
Are the Score Descriptions Well Formulated? Another component in the score rubric is de- scription of each score (i.e., A comprehensive reason why score of i S i S 5 should be given). In an Absolute Scoring setting, it is important to evaluate the given response based on the score descriptions instead of giving score of 1 for all responses that lack a minor detail or giving a score of for all responses that seem to be good on the surface. Due these reasons, the role of the score descriptions hold crucial, where the main role is show monotonically increasing tendency of sentiment, not dramatically. Figure 8 shows that the FEEDBACK COLLECTION holds a smoothly in- creasing sentiment tendency for each score description. This ensures the quality of the score rubric, confirming that it plays a role deciding the score.
Is there a length bias among the Responses? Previous work has demonstrated that when LMs are used evaluators, they tend to give higher scores to longer responses (Li et al., 2023; Dubois al., 2023; Zheng et al., 2023). order minimize this effect during fine-tuning PROMETHEUS, one of our main consideration was to maintain a length distribution equal among the score range of to 5. As shown in Figure most of the responses within the FEEDBACK COLLECTION maintained similar length among different scores (near 200 tokens). We also include comprehensive analysis of whether PROMETHEUS possessed any length bias during evaluation in Appendix
Are the Instructions, Responses, and Feedback Diverse as Well? In addition the analysis of the score rubric and responses, we also analyze whether the instructions, responses, and feedback within the FEEDBACK COLLECTION are diverse enough. For this purpose, we examine the bigram and trigram ratios. The results are shown in Table 7, indicating a variety in how terms are expressed, and our findings suggest a moderate level of diversity. While there some term repetition, dataset also showcases notable range of expressions.
E ANALYSIS OF THE FEEDBACK BENCH EVALUATION DATASET
In this section, provide a analysis of whether the FEEDBACK BENCH consists of unseen score rubrics against the score rubrics from the FEEDBACK COLLECTION.
Does the testset maintain Unseen Score Rubrics? One of the main considerations of our exper- iments in Section 5.2 using the FEEDBACK BENCH was testing whether PROMETHEUS could gen- cralize to unseen customized score rubrics. For this purpose, we built an unseen customized rubric subset. We plot the rouge-L distribution between random score rubric within the FEEDBACK COL- LECTION and a random score rubric within the FEEDBACK BENCH. As shown in Figure 10, there low overlap among the train and test sets, confirming that the FEEDBACK BENCH is valid to be claimed as an unseen test set to measure the evaluation capability of evaluator LMs.
F FINE-TUNING AND INFERENCE DETAILS OF PROMETHEUS
We use 8xA 100 (80GB) GPUs to train our models with Py Torch Fully- cully-Sharded Data Parallel (FSDP) option. The code we used for training and inference is the official Llama2 fine-tuning code released we used are the basic settings in the fine-tuning code except
by Meta AF. The wtt.c/o.occcceeccee/e facebookreseareeeeoooooo 1ama recipes
hhttps://gi thub.
20

---

Distribution of ROUGE-L Scores Between Criteria and Test Criteria Descriptions
10000
Freuunccy
8000
2000
0.0
0.2
0.4
0.6
0.8
1.0
ROUGE
Figure 10: Rouge-L. score distribution among a randomly sampled score rubric from the FEEDBACK COLLECTION and rubric from the FEEDRACK BENCH. A left-skewed distribution with low values shows that they do not overlap with each other, hence meaning that a UNSEEN score rubric assumption is satisfied.
Table 8: Hyperparameters used for fine-tuning PROMETHEUS.
Schedaler
Optimizer
Length Linpur Ourput)
PROMETHEUS 7B
Liama 2 Ca-t-hh-- 7B
28
le-5
StepLR
AdamW
4096
Liana 2: 2-Char133
StepLR
AdamW
4096
Table Hyperparameters used for inferencing PROMETHEUS, GPT-3.5-Turbo, and GPT-4. Verbal- izer denotes accepting outputs such "[Score 5]" or "Score: 4 out of 5" whereas the exact format "[Result] 5" (format is mentioned concretely within the instruction given to the evaluator LM).. Even after applying verbalizer, Llama-2-Chat is not able generate easily be parsed, highlighting the benefits of fine-tuning it on feedback data.
decision that could
Params
Model
Temperature
Top-p
Repetition Penalty
Max Output Length
Verbalizer
7B
1.03
1024
Yes
13B
Hamma-2-Chat-a
1.0
0.9
1.03
1024
Yes
70B
Jama-22Chat--00
1.0
0.9
1.03
1024
Yes
7B
PROMETHEUS-7B
1.0
0.9
1.03
1024
No
13B
PROMETHEUS-I3B
0.9
1.03
1024
No
GPT-3,5-Turbo
0.9
1024
No
GPT-4
1.0
0.9
1024
No
for the training batch size which was set according to the model size: for 7B models we used 28 and for 13B models we used 20 to fully leverage GPU memory. Note that in the official Llama2 fine-tuning code, the loss is only calculated on the feedback and score decision, not the instruction. We empirically find that not masking out the instruction leads poor performance while evaluating responses. The detailed hyper- parameters are shown in Table 8.
For inference, Llama-2- 2-Chat model (not trained on the FEEDBACK COLLECTION) was extremely difficult to steer the model to generate final score in the form to be easily parsed (c.g., "[RESULT] 3"). While in-context learning (ICL) could solve this issue, most of our instances contained maximum of 3072 tokens, could not utilize demonstrations during inference. Therefore, empirically
the hyper-parameters as shown in Table 9. When inferencing with the naive
21

---

Prometheus vs GPT-3.5-Turke
100
not consistent score too general and abstract optimistic
80
not relevant to response everly critical
49.23%
59
20
Promethaus
GPT-3 GPT-3.5-Turbo S-Turbo
Figure 11: The reason why GPT--35-Turbb's or Prometheus's feedback was not chosen over the other. PROMETHEUS generates less abstract and general feedback, but tends to write overly critical ones.
found patterns such as "[SCORE 5]" "Score: 4 out 5" and applied verbalizer map those outputs to final score decision. This highlights the benefits of directly training to generate in structured format as PROMETHEUS. On the other hand, we also find that proprietary LLMs such as GPT-3.5-Turbo and GPT-4 excel at generating structured outputs when the prompt is adeptly given. Also, note that we found that we set the temperature to 0.0, evaluator LMs are able generate meaningful feedback compared to using temperature of 1.0.
G TRAINING A EVALUATOR LM ON COARSE-GRAINED SCORE RUBRICS
For the purpose of exploring the benefits of training thousands of fine-grained and customized score rubrics, we employ a baseline of only training on relatively coarse- poorse-grained score rubrics. Since the FEEDBACK COLLECTION'S instructions closely tied with the score rubrics during its creation process, we could not directly use and only change the score rubrics into coarse-grained ones.
So, we used the Flask dataset (Ye et al., 2023b) and split it into training data and evaluation data. The evaluation data is denoted Flask Eval throughout the paper. Specifically, the Flask dataset consists of 1.7K instructions acquired across conventional NLP datasets and instruction datasets. Also, there exists 76.5K responses acquired across 15 response LMs. Each instance has score rubric among 12 options (Logical Robustness, Logical Correctness, Logical Efficiency, Factuality, Commonsense Understanding. Harmlessness, Readability, Comprehension, Insightfulness, Com- pleteness, Metacognition, Conciseness). While these 12 score rubrics are more fine-grained and di- verse compared to previous works only using helpfulness and harmlessness, they are coarse-grained compared to the thousands of score rubrics included within the FEEDBACK COLLECTION, so denote as coarse-grained in this work.
Among the 1.5K instructions & 67.5K responses as training data, found that the score distribution extremely skewed towards the score of 5. We distributed the instances so that the number of instances within the range of to remains equal, which leads to 30K training instances. We trained the Llama-2- 2aCatt model on the Flask train set, which led to one of our baselines denoted as LLAMA-2-CHAN COARSE in Table 2, Table 3, Table 4, Table 6.
22

---

GPT-4 vs GPT-3.5-Turk3
100
not consistent score too general and abstract optimistic
0.00%
80
9.52%
13.33%
not relevant to response everly critical
19 05%
25.71%
23 81%
19 05%
20
24 29.
6.67%
GPT4
GPT-3 S-Turbo
Figure 12: The reason why GPT-4's or GPT-3.5- 11-33--Tubbbss 's feedback was not chosen over the other.
GPT-4 generates less abstract and general feedback, but tends to write overly critical ones.
H PAIRWISE COMPARISON OF THE QUALITY OF THE FEEDBACK
In this section, further explain the experimental setting and present additional results & analysis for the experiment of comparing the quality of the generated feedback (Section 5.
In addition to Figure 5, the reason why each annotator rejected the feedback from either PROMETHEUS, GPT-3.5-Turbo, GPT-4 is shown in Figure 11 and Figure 12.
The results further support our claim that PROMETHEUS tends to be critical over GPT-4 and GPT- 3.5-Turbo. Interestingly, GPT-4 was considered to be more critical compared to GPT-3.5-Turbo and the gap was even wider when comparing GPT-3.5-Turbo and PROMETHEUS. This indicates that PROMETHEUS can serve as a critical judge when evaluating responses generated by LLMs, but could also be biased towards not being optimistic generally. The degree of being critical could be useful or a limitation based on different use cases. For instance, we conjecture that it could be helpful when analyzing the limitations LLMs or providing feedback as supervision to further improve target LLM (e.g., RLHF), yet we leave this exploration to future work.
I IS THERE A LENGTH BIAS DURING EVALUATION?
One of the limitations of employing an LLM as an evaluator LM is that it could be vulnerable to various biases. In this work, we train/test on an Absolute Scoring evaluation setting, hence there exists no position bias. Yet, it is crucial to analyze whether PROMETHEUS showed any bias towards favoring longer responses. Hence, we conduct comprehensive analysis this section.
As shown in Figure 13, Figure 14, and Figure 15, both GPT-4 and PROMETHEUS and GPT-4 shows similar trend of not favoring longer responses (i.e., similar length distribution among different scores). However, as mentioned in Zheng et al. (2023), LLM evaluators might favor more verbose responses, yet the responses from our test instances (FEEDBACK BENCH and Vicuna Bench) did not include any adversarial examples to test this phenomenon. More extensive research on whether the length bias is also transferred to fine-tuned evaluator LMs should explored in future work.
J PROMPT FOR FEEDBACK COLLECTION CREATION
In this section, provide the extensive list of prompts used to create the FEEDBACK COLLECTION.
23

---

GPT 4 vs Prometheus Eval (feedback Collection Seen Rubric Testset)
GPS4 homeroa
8
500
Nesporss Lenghh
800
200
Score
Figure 13: Box and whisker plot describing a relationship between a given response and its corre- sponding score. We check if the response lengths correlate with its scores.
vs. Prometheus (Feedback Collection Unseen Rubric Testset)
1400
1,00
Response Longhh
800
600
400
200
Figure 14: Box and whisker plot describing relationship between given response and its corre- sponding score. We check if the response lengths correlate with its scores.
GPT-4 vs Prometheus Eval (Vicuna)
CPSA
1200
Rosponss Lergth
800
400
Score
Figure 15: Box and whisker plot describing a relationship between a given response and its corre- sponding score. We check if the response lengths correlate with its scores.
Note that in the prompt of generating response and feedback, use the sentence length of the reference answer and append it to "(SENT NUM]" within the prompt. This was crucial to make the length even across different scores shown in Figure 9. Also, note that for the 1K
24

---

score rubrics, 20K instructions & reference answers, and 100K responses & feedback within the FEEDBACK COLLECTION, each prompt was sequentially used. In early experiments, we found that generating every component all at once leads to very poor generation quality (i.e., similar responses & feedback across different score ranges). Yet, we found that grouping (1) the instruction and reference answer generation and (2) the response and feedback generation had positive synergy, leading to better generation quality and less amount of cost, Also, to the best of our knowledge, we first explore acquiring negative and neutral responses (Score 4 responses) through GPT-4 augmentation. We hope future work could also explore applying this strategy to different use cases.
Prompt for Brainstorming New Score Rubries
We are brainstorming criteria with which to grade language model on its responses in diverse situations.
A 'criteria' is some useful, real-world objective, and associated rubric for scores 1-5, that tests capability.
Here you will see 4 examples of 'criteria', and their scoring rubrics, formatted as ISON.
Criteria 1:
(JSON LIST 1}
Criteria 2: (JSON LIST 2)
Criteria 3: {JSON LIST 3}
Criteria 4: (JSON LIST 4)
Please brainstorm a new criteria and scoring rubrics. Be creative and create new but useful criteria that people in different settings industries might find practical.
Please format the output same the above examples with extra or surrounding text. Write [END] after you are done.
New Criteria:
Prompt for Paraphrasing New Score Rubric
Please paraphrase the sentences inside the dictionary below. Each paraphrase should not change the meaning or substance of the original sentence, be naturally written, but sufficiently diverse from one another. Diversity can come from differences in diction, phrasing, sentence structure, formality, detail, and/or other stylistic changes.
The dictionary: (CRITERIA)
Respond with only dictionary (same format the given dictionary) with extra surrounding text. Write [END] after you done.
Dictionary with Paraphrased Sentences:
25

---

Prompt for Generating an Instruction and Reference Answer
Your job generate a new novel problem and a response that related to the given score rubric.
The score rubric: {CRITERIA]
Problem
-The problem should inherently be related to the score criteria and score rubric given above. Specifically, the score criteria should be the core attributes required to solve the problem.
The problem itself should be too generic or easy solve. If the score rubric is related to logical abilities, generate problems that require math or coding abilities.
Try to make the person who might solve the problem not notice the existence of the score rubric by not explicitly mentioning it, and also provide additional inputs and options needed.
Assume situation where a user interacting with an model. The user would try to ask in first-person point of view, but not using terms like "T", "A User" "You" in the first sentence.
Do not give role to the AI, assume that the user is asking question from his point of view.
Do not include any phrase related AI model in the problem.
Response
The response should be response that would get score of 5 from the score rubric. The response should be as detailed as possible unless the score rubric is related to conciseness brevity. It should consist of multiple paragraphs, list of items, or step-by-step reasoning process.
The response should look like how well-prompted GPT-4 would normally answer your problem.
Format
DO NOT WRITE ANY GREETING MESSAGES, just write the problem and response only.
-In front of the problem, append the phrase "Problem:" and in front of the response, append the phrase "Response:".
Write in the order "Problem"- "Response", where the items are separated by the phrase "[NEXT]".
Write [END] after you are done.
Data Generation:
26

---

Prompt for Generating Responses and Feedback
Your job is to generate a response that would get score of (SCORE) and corresponding feedback based the given score rubric. For reference, reference response that would get a score of is also given.
Instruction:
[INSTRUCTION]
The score rubric: (CRITERIA)
Reference response (Score 5):
[REFERENCE]
Ressponee
The quality of the {SCORE} response should be determined based the score rubric, not by its length.
The score (SCORE) response should have the same length as the reference response, composed of {SENT NUM} sentences.
Do not explicitly state the keywords of the score rubric inside the response.
Feedback
'Tee score {SCORE} feedback should each be an explanation of why the response would get a score of (SCORE). should be written based on the generated response and score rubric.
The score {SCORE) feedback shouldn't 't just copy and paste the score rubric, but it should also give very detailed feedback the content of the corresponding response. The score (SCORE) feedback should include the phrase "So the overall score is {SCORE)" in the last sentence.
Format
DO NOT WRITE ANY GREETING MESSAGES, just write the problem and response only.
In front of the response, append the phrase "Response:" and in front of the feedback, append the phrase "Feedback:".
Write in the order of "Response" "Feedback", where the two items separated by the phrase "[NEXT]".
Write [END] after you done.
Data Generation:
27

---

K PROMPT USED FOR PROMETHEUS
In this section, we provide the prompt used for training/inferecingg PROMETHEUS. Note that after applying the prompt template shown below, we also apply Llama-2's basic conversation prompt template in order to minimize the discrepancy between the training process of Llama-2 and training on the FEEDBACK COLLECTION.
Prompt for Prometheus
###Task Description:
An instruction (might include Input inside it), a response evaluate, reference answer that gets score of 5, and score rubric representing an evaluation criterion is given. Write detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general.
After writing feedback, write a score that an integer between 1 and You should refer to the score rubric.
The output format should look as follows: Feedback: (write feedback for criteria) [RESULT] (an integer number between I and 5)"
Please do not generate any other opening, closing, and explanations.
##e instruction to evaluate: (instruction)
###Rese evaluate: {response}
###Reference Answer (Score 5): (reference answer)
###SSoore Rubrics:
{f criteria description H
Score I: {scorel description
Score 2: (score2 description
Score 3: (score3 description
Score 4: (score4 description)
Score 5:
{score5 description) }
###Feedback:
28

---

L QUALITATIVE EXAMPLES OF GENERATED FEEDBACK
Figure 16, Figure 17, Figure 18, Figure 19 shows a qualitative example of feedback generated by either GPT-4, PROMETHEUS (I3B), and Code-Llama trained on the FEEDBACK COLLECTION.
M EXPERIMENTAL DETAILS FOR HUMAN EVALUATION
The interface used for human evaluation shown in Figure 20, In order acquire score decision for the response, a decision of which feedback is better, and an annotation of why they made decision to choose one of the feedback, we constructed the user interface in sequential order. Each annotator was informed not to solve the questions below before solving the ones above them. The instances were randomly selected among FEEDBACK BENCH (Customized Unseen Rubric set), Vicuna Bench, and MT Bench except for math and coding instances. The experiments were conducted on LabelStudio, hosted on Huggingface Spaces. We asked each annotator to spend at least 3 hours annotating 45 instances each and paid $ 50. The 9 annotators were divided into groups: (1) PROMETIEUS vs GPT-4, (2) PROMETIEUS vs GPT-3.5-Turbo, and (3) GPT-4 vs GPT- 3.5-Turbo The order of the feedback was randomly decided in order prevent order bias during annotation.
29

---

Score Rubric:
Does the model demonstrate an ability to logically process and interpret given information, apply relevant knowledge, and present an insightful conclusion?
Instruction
Here is dialogue between an assistant and an user. Generate a human preferable response by the assistant.
One morning after sunrise, Suresh was standing facing pole. The shadow of the pole fell exactly to his right. Can you tell me the direction towards which the shadow was pointing east, south, west, or north? Explain your reasoning steps.
User:
Assistant- on the given information, Suresh was standing faciing pole, and the shadow of the pole exactly to his right. To determine the direction towards which the shadow was pointing, we need consider west. the morning, when Suresh was standing facing the pole, the would have been the eastern sky. Since the shadow of the pole fell exactly to Suresh's right, means that the shadow was pointing towards west.
the during day. IninThe
the
sets the
Therefore, the direction towards which the shadow pointing is west.
User: which direction was Suresh facing? How you solve this?
Assistant:
Response from ChatGPT
Apologies the confusion.
To determine the direction Suresh was facing, we need to consider the pasition of the shadow in relation to him. the shadow fell exactly to his right, it means that Suresh was facing towards the west. The shadow formed when object blocks the light from Suresh's right, indicates that the sun is in front of him, casting the shadow towards his back. Therefore, Suresh was facing towards the east.
In case, since the shadow is
GPT-4's Feedback
Prometheus's Feedback
The response provided by the model is inaccurate and doesn't demonstrate a clear understanding the given information According to the user's query, Suresh was standing facing a pole, and the shadow of the pole exactly to his right. This implies that Suresh facing towards the west, not the east as the model suggests. Furthermore, the model falled to properly explain the reasoning behind conclusion, leading conclusion that contradicts the initial information provided. The model didn't correctly apply the knowledge
The model interprets the information from the accurately and applies the concept of the sun's position and its effect on the formation shadows. However, it conclusion that
task.
fails reach the Suresh is facing the south. Instead, incorrectly concludes that Suresh is facing the east. The response demonstrates partial understanding of the problem but lacks complete logical coherence in the solution.
illogical
sun during the day, did take into account the position of the shadow in relation to Suresh.
movement
The final score is out of 5.
The final
of
Figure 16: An example of comparing the feedback generated by PROMETHEUS and GPT-4. GPT4 was labeled to generate an abstract feedback.
30

---

Score Rubric
Does the Response represent balanced view of the factors?
Instruction
Why might someone prefer shop at small, locally-owned business instead a large chain store, even if the prices are higher?
Response from Vicuna 13B
There are several reasons why someone might prefer to shop at a small, locally-owned business instead of large chain store, even the prices are higher:
Supporting the local economy: By shopping at small, locally-owned business, you are supporting the local economy and helping to keep money within the community.
Personalized service: Small, locally-owned businesses often offer personalized service and attention that may not be available at large chain store. The staff may more knowledgesble about the products sell and may be more willing to help you find what you're looking for.
3. Unique products: Small, locally-owned businesses often carry unique products that you may not be able find large chain store. This can be especially true for businesses that specialize in niche products or that offer custom- made items.
4, Community involvement: Small, locally-owned owned businesses are often more involved in the local community than They may sponsor local
large chain
support charities,
community
as whole.
locally-owned businesses often customers and may be more willing to above and beyond to ensure customer satisfaction.
Better customer service:
personal relationship their
Environmental ethical considerations: they believe that these businesses are more likely to have environmentally and socially responsible practices.
people prefer shop small, locally-owned businesses because
Overall, shopping at small, locally-owned business can be to support the local community, get personalized service, and find unique products that you may be able find at large store.
GPT-4's Feedback
Prometheus's Feedback
resporse does good job presenting of why sameone might prefer to shop at small, locally- local economy,
excellent job of presenting balanced view of the factors that might make someone prefer shopping at a small, locally-owned business
The response
owned business, including supporting community personalized service, unique products, involvement, better customer service, and environmental and ethical considerations. However, the response could improve in presenting all factors with equal rigor.
large chain store, even the prices higher. Six different reasons were given, each of them explained thoroughly in depth. given more emphasis than the others, and each explanation was fair and unbiased. However, it's worth noting that response failed to consider factors that might fevor large chain stores, the reference answer did.
reason was
For example, the response does not delve deeply enough into specilics of how shopping small, loaalyy- owned business could support the local economy, nor does explain how these businesses might involved in the community compared to large chain The
could also further discuss how these factors might not be as important to all consumers, similar how the reference response acknowledges importance of factors like price, convenlence, and product range large chain stores.
The final
4 out of 5.
final
An example of comparing the feedback generated by PROMETHEUS and GPT-4. PROMETHEUS was labeled generate an overly critical feedback.
Figure 17:
31

---

Rubric:
the process. Can the model's response understood by non-expert in the subject?
Instruction
Explain the process of photosynthesis plants in way that middle school student could understand.
Response from Simulated GPT-4
Photosynthesis, essentially, biochemical reaction in which plants use sunlight (solar energy), (H20), and dioxide (CO2) produce glucose (C6H1206) oxygen (02).
It's the key method that plants use for their nutritional requirements. To begin with, plants capture solar energy with chlorophyll in their leaves.
Then, with the absorbed solar energy, the plant's chloroplasts take water from the ground and CO2 from the atmosphere to perform the complex photosynthesis process.
The end result is glucose, which plants use as energy source for growth and development. Additionally, plants emit oxygen, which is beneficial for us as it's necessary for our respiration.
So, to simply, photosynthesis is the process in which plants transform sunlight, water, and carbon dioxide into glucose and oxygen.
GPT-4's Feedback
Prometheus's Feedback
The explanation given this response is generally understandable, several terms and concepts that might difficult for middle school student to grasp. instance, terms "chlorophyll," "chloroplasts", "photosynthesis", "solar energy", "glucose", and "oxygen" are used without sufficient contextual exp lanations. While the main ideas photosynthesis conveyed, the language used could potentially be confusing for mon-exxert, Additionally, concept plants using sunlight, water, and carbon dioxide produce glucose and exygen is described but lacks concrete, relatable analogies that would have helped simplify the explanation. The explanation could be improved by using simpler language and incorporating more accessible examples metaphors help
Your explanation of photosynthesis is decent and uses analogies to make some aspects relatable
expeex, middle school student. However, there are still areas that could be challenging for them. use the terms "biochemical reaction', 'chlorophyll, "hhoroppaasss, and the chemical formulas for water, carbon dioxide, and glucose could confuse non-expert, making harder them grasp overall concept. Instead of diving straight into the complex aspects of photosynthesis, you might started with simpler relatable analogy. Additionally, providing step step-by-st step description the process would have made your explanation more digestible.
So overall score 3.
So the overall score is 3.
Figure 18: An example of comparing the feedback generated by PROMETHEUS and GPT-4, Both evaluator LM generated good feedback, and hence was labeled tie.
32

---

Score Rubric
How well is the implementation in terms of its correctness, efficiency, documentation, and error handling?
Instruction
Develop a C++ program that reads a text line by line and counts number occurrences a specific word in the file.
Response from Wizard LM 13B
Here is a C++ program that reeds text file line by line and counts the number of occurrences of specific word in the file:
#include <lostreams
#inolude
#includo estring>
using namespace std;
int mainn {
File
string Mleeamm \'exemple..xi)";
exist,
(filenaat)
ofstream outifilename); out \oNo such file."" << endi;
Open the file read mode ifstream initllename, los:in); Count the
occurrences
int count D: string
woou__orr__eere"
Replace with the word want to count (in (word == word)
Print cout <c The word Y word ex appears ! <x count ec times in the file." <c endi;
Close
in.ciose():
return
Here's how the program works:
1. The first few lines include the necessary header files and define the 'main()" function. read.
"Ilename' variable defined store name the
3. The program checks if the file exists and creates it does not.
file opened
using 'ifstream'
The program enters loop that reads each line the file.
8. Inside the loop, the program checks if the current line contains the word to counted. If does, the 'count" variable incremented.
7, After the loop exits, the program prints the result to the console. Finally, the file closed using "lose() method
'ifstream' object.
To use this program, simply replace the 'fianame" variable with the name the file you want read, and nun the program. output
specified appears
33

---

Fee Feedback
The condition in while loop, word; overwrite the variable 'word' with the word read from the file. This makes the following 'if' statement to always evaluate to true. Therefore, the program counts all the words in the file, not just the occurrences a specific word. Also, the standard C+ and could lead documentation, but it does not address the aforementioned errors, making it somewhat misleading. Finally, there is no error handling case does not exist or cannot be opened. Hence, this response does not demonstrate and it does not handle possible errors properly.
implementation provided fully
'exists' function error. The description the code does provide some basic
understanding the problem the method,
final
of
Prometheus's Feedback
The provided C++ program correctly implements solution to read text file line by line and count the number of occurrences of specific word. It does so efficiently and well- documented, making it easy to understand and use. The program effectively handles errors and does not doesn't exist. The code is well-organized and well-structured, making it easy to follow and maintain. So the overall score is
error prone methods such checking fill existence creating the file
Code-Llama (7B) Fine-tuned the Feedback Collection
Feedback(nnnThhe provided response some the criteria areas for improvement. Here some specific feedback points:
rubric,
counting number occurrences of specific ward file. However, the code could benefit from additional error handling and handling of edge cases, such as dealing with files that do not exist or containing invalid data. (Score
Correctness: response correctly implements
Efficiency: The response relatively efficient terms memory usage processing time, but there are areas where optimization could be achieved, such as reducing the number of 'if' statements and using more efficient data structures. (Score 3)
* Documentation: The response could benefit from additional documentation, including comments and explanations of the code, to make easier for others to understand and use.
(Score 2)
The response could handle errors more robustly, such as providing error messages and handling edge cases, make it more rellable and user-friendly. (Score 3)
Error handling:
Overall, the response scores out of terms of evaluation criteria.
An example of comparing the feedback generated by PROMETHEBSS, GPT-4, and Code- Llama trained on the FEEDBACK COLLECTION. Compared to PROMETHEUS, using a base model specialized on the code domain also helps criticize and evaluate responses within the code domain.
Figure 19:
34

---

Please read through score rubric (Al 7| instruction (Al [ # A]A]), response (Al l Ert), and reference answer ( E
Plesse refer this rubric enauntion
modafs
understoed
Score The response is filed with jargon and comples language, making incomprei ensible for non-expert.
lelics
language.
Score The response is somewhat clear, but could stil chailinging ig for non expert to fully understand: cempreeensiilees
fow complex Score The response is complesaly clear and understandazle for non export, with no rellance on jargon or complex language.
The inser action walluate::
the photogyythesis
could undorstand.
Photosynthesis, essentaally, biochomical osygen (02), the method that pla use for their nusitiinnal requirements. To begin with, ants capture colar energy with the choooopyll their leaves. Then, with the ground 002 stmosphere perform complex photocyntheslo process.
plants use sunlight (solar energy), water (H20),
dioadee (CO2) produce glucoss (26H1206)
soler energy, stant's chloroplasts glucose, which plants use energy source for growth and development. Addtticallyy plants emit ooygen, which beneficial for us as necessary for our respiration. glupose orygen.
simply, photosyntes's process
sunlight,
Anower that get
Photosynthesis making process plants
simple.
Imagine plants are like littl chefs. They need certain ingredients make their food, much like how might noed flour, suçar and opps to bake cako. The ingredients photooynthesis sunlight, carbin dioxide, which
use
sunlight.
solar panels. ounlight energy plants start making food.
Water absorbed through the roats plant
roots straws, succing the
ground
Carbon d oxide comee from the air. Plants have thny openings on their leaves, called stomata, which act like litle doore let the carbon diowide in
Oria the plant has these three ingredients sunlight water, and carbon diocide it can start making its food This food is sugar swiled glaccon, which the plant uses to grow you photosyrthes's
develop,
elso eaygen, which
and other animala tool
So in short, photosynthesis the process where plants sunlight hii, water and corbon closide make their own food and give out oaygen. It's like little cooking show hoppenting every day gardens, parks,
35

---

What would you give the response above?
2H
4M
Please read through FeedbackA (Al l Bo C ø7))
Your explanation of shotoeyntesis is decent and uses analog make some aspects more relasable expert, such a micdle school student. However there blochemical reaetion, chlorophylf, 'hhlropllssss; and glucose could contuse non-expert making it harder for them grasp the overall concept: instead of diving stroight into the complex aspects of photosynthesi, might your explonation digestible.
shallenging
water carton
simpler
analogy Additionally providing step by desoription process
So overall ocoee is
The final acoro 3 out of
Please read through FeedbackB 2 Gf C ()))
that diffoutt middle school giasp. instance, terms like orostyll" "chloroolests" "shoooyythesii" "solar energy" "alucose' and 'oaygen' are used without suffic ent contextual explanatoon ons, While the main ideas photosynthesis dicalde procuce gl plucose and oargen desoribe but looks zonorete, re relatabre analogies that would have helped simplity the explanation. The explanation could be improved by simpler
explanation
generally
language potenticlly
Additionalty the
incorporating
metaphors Illustatee
Which feedback is better?
feedbackB betterraised, handedf!
Both are Back11f
Why diid you chose one of the feedback above? Choose more than one options.
The rejected leadback unnelated to the givan acere rubric
rejected
relevent respansse
The rejected feedback
and absttrest
The rejected leetbockk overly critice!
The rejected fredback optimistio X
The rejected feedback not consister innssttnntte with the given score it gave.
the
Figure 20: The annotation user interface for labeling the human scores, pairwise comparison of the two feedback and gathering labels of why one feedback was preferred over the other one.
36


Reviews:
[{"review_id": "Tc6AclxmTc", "reviewer": "Reviewer_6KgK", "summary": "This paper propose a method to automatically generate evaluation dataset using the few-shot capabilities of GPT-4 starting from \"50 initial seed rubrics\". The method shares some similarity to self-instruct (Wang et al., 2023). The main difference is apart from expanding seed rubrics, the proposed method also need to craft instructions and training instances. The resulting evaluation dataset (i.e., FEEDBACK COLLECTION) is further used to fine-tune a llama2-chat model, which is called PROMETHEUS. Essentially, PROMETHEUS tries to distill the evaluation capabilities of GPT-4.\n\nExperiments show PROMETHEUS correlates well with human judgements and GPT-4 references.", "questions": "- To what extent do the seed examples (and their generated instances) resemble examples in VICUNA, MT Bench, or FLASK EVAL?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- LLM based evaluation is important, since long text generation is becoming more difficult for human evaluators\n- Using a distilled evaluator can significantly reduce cost, compared to GPT-4\n- Results look good", "weaknesses": "- It is unclear why PROMETHEUS correlates better with human evaluation than GPT-4 on MT Bench (Figure 3), given the fact that PROMETHEUS distills from GPT-4. Further analysis is required.\n- There is no details on how the seed examples are constructed.\n- Generalization to different domains seems difficult. Note that this is not a weakness, since almost all LLMs have this problem. Let us assume that the GPT-4 evaluator is good enough (that is also the reason why the proposed method intend to distill from GPT-4). Now the task becomes how good can PROMETHEUS mimic GPT-4 evaluation.  It is shown in Tables 2 and 3 that the correlation between PROMETHEUS and GPT-4 reference is significantly higher in the in-domain dataset (data generated by GPT-4, as shown in Table 2) compared to out-of-domain datasets (refer to Table 3), with approximate values of 0.46 versus 0.86. Nevertheless, the development of an LLM evaluator that performs effectively on certain tasks is still meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4HwynR7s1", "reviewer": "Reviewer_B7Vr", "summary": "Paper presents a new benchmark for building evaluation systems with LLMs. Although the paper contribution is promising, there are some serious problems in the paper. Many of the figures are missing and unvisible. The paper contribution, whether this is a novel LLM, or a data set generated by gpt-4 is unclear. The model is advertised as open-source but how the data will be shared is unstated. If an LLM is built on this data, which is described as a 100K synthesized data set, how is it an 13B LM is unclear. Paper cannot be published in such state with so much missing information.", "questions": "Where is Figure 2?\nWhere is Figure 4?", "limitations": "", "rating": 1, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Proposes open-source LLM for evaluation", "weaknesses": "Model implementation is not described.\nExperimental methodology not clear or supported.\nMost figures missing.\nContribution too small (not any new data, model or any advertised contribution is clearly described).\nData is synthetic and not corrected by humans for any potential errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0U8KiCwQwC", "reviewer": "Reviewer_LTFo", "summary": "This paper presents Prometheus, an open-source language model that provides fine-grained evaluation capabilities comparable to GPT-4. The authors aim to overcome the challenges of using GPT-4 as an evaluator, such as its closed-source nature, uncontrolled versioning, and high cost. Prometheus is trained on a new dataset, the Feedback Collection, which includes a wide range of user-based evaluation criteria. The model shows strong correlation with GPT-4 evaluation on seven benchmarks and outperforms ChatGPT in human evaluation. Remarkably, Prometheus demonstrates a win rate of 58.62% when compared to GPT-4 evaluation and a win rate of 79.57% when compared to ChatGPT evaluation.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Prometheus can assess responses based on novel and unseen score rubrics and reference materials provided by the user. This flexibility makes it applicable to a variety of real-world criteria.\n2. Prometheus can be freely used and further enhanced by the academic community, facilitating transparency and reproducibility.\n3. Prometheus shows remarkable performance in comparison with GPT-4 in terms of evaluation capabilities and the quality of generated feedback.\n4. The creation of the Feedback Collection, a dataset designed specifically for the task of teaching fine-grained evaluation to language models, is a significant contribution.", "weaknesses": "1. One of my concerns about this work is whether can Prometheus be generalized to other fields since the downstream benchmarks are close the the training data. More results on unseen data and more specific domains can better improve this work. \n\n2. Potential bias of Prometheus. Can Prometheus be attacked by some adversarial attack methods? Does it have stronger biases like length bias compared with GPT-4?\n\n3. Dependency on GPT-4 Feedback: The training of Prometheus relies heavily on feedback generated by GPT-4. The model's ability to generalize beyond the feedback patterns of GPT-4 is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OSSEF7fGgI", "reviewer": "Reviewer_WpqT", "summary": "This paper introduces PROMETHEUS, an open-source large language model (LLM) that aims to provide evaluation capabilities on par with the proprietary GPT-4. To achieve this, the authors create a new dataset called FEEDBACK COLLECTION, containing diverse and fine-grained user assessment criteria. PROMETHEUS is trained using this dataset and demonstrates a strong correlation with GPT-4's evaluation capabilities, as well as human evaluators.\n\nThis paper addresses the limitations of using proprietary LLMs like GPT-4 for evaluation, such as closed-source nature, uncontrolled versioning, and prohibitive costs. The PROMETHEUS aims to offer an alternative that is open-source, reproducible, and cost-effective. The FEEDBACK COLLECTION dataset allows the model to generalize to various evaluation preferences and real-world scenarios. In tests, PROMETHEUS outperforms other baselines and shows potential as a universal reward model.", "questions": "see above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The organization of this paper is well-structured, making it easy to read and comprehend.\n2. This work presents the creation of a FEEDBACK COLLECTION dataset, which encompasses a diverse range of scoring criteria, reference answers, and feedback. Based on this, an evaluation LLM is trained for assessing the text generated by large language models.\n3. The analysis in this work is thorough, discussing the selection of base models, data construction, and demonstrating the importance of reference answers. This provides valuable insights for the evaluation of large models in the field.", "weaknesses": "1. The main contribution of this paper is the FEEDBACK COLLECTION dataset. However, the dataset has not been made publicly available, and the construction details are unclear. For instance, the content of Step 2 is incomplete, and Step 3 is overly simplistic. Furthermore, the prompts used during construction have not been disclosed.\n2. Assessing the consistency of scores alone is insufficient; it is also necessary to evaluate the feedback corresponding to these scores. On one hand, it is important to determine whether the feedback aligns with the scoring criteria. On the other hand, it should be examined if the feedback can be appropriately matched with the given scores. In fact, humans are not solely interested in obtaining a score; they are more concerned with the feedback associated with that score, which can further guide the large language model to generate desired answers.\n3. It is unclear whether the test data in Table 1 is manually constructed or generated by GPT4-0613. If it is generated by GPT4-0613, why are the Pearson/Kendall/Spearman evaluation metrics not equal to 1?\n4. For the Unseen FEEDBACK COLLECTION Testset, should all unseen instances be extracted? If the evaluation is conducted by combining the 50 Unseen samples with the 1000 samples similar to the training distribution, would this overshadow the true performance when facing unseen distribution during training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)