InstructEval: Instruction-Tuned Text Evaluator from Human Preference (2024)

Abstract: This paper explores to construct a general text evaluator based on open-source Large Language Models (LLMs), a domain predominantly occupied by commercial counterparts such as GPT-4. Recognizing the limitations of open-source models like Llama in evaluative tasks, we introduce InstructEval, a general multi-aspect text evaluator developed through instruction tuning of open-source LLMs. To overcome the shortage of annotated resources for multi-aspect evaluations, InstructEval combines extensive open Human Preference Modeling (HPM) datasets with a small set of multiaspect annotated data. This approach not only enhances effectiveness in overall evaluation tasks but also exhibits improved performance in multi-aspect evaluation tasks. As demonstrated by our extensive experiments, InstructEval achieves comparable or superior performance to commercial LLMs like ChatGPT or GPT-4 in terms of both overall and multi-aspect evaluation.

heading: 1 Introduction
text: Recent advancements in LLMs, exemplified by renowned models like ChatGPT, have showcased their impressive zero-shot capabilities in a generative manner, empowering them to effectively handle a wide range of arbitrary human instructions (Brown et al., 2020; Wei et al., 2022). Nonetheless, assessing the text quality presents a significant challenge due to the complexity of tasks and the necessity for multi-aspect evaluation (Ethayarajh and Jurafsky, 2021; Chang et al., 2023).
In recent research, leveraging the generalization capabilities of Large Language Models (LLMs) as evaluators for text generation tasks has gained popularity. For example, a popular method involves using pairwise comparisons, with models like GPT4 serving as benchmarks for identifying superior
∗Work is done during an internship at Baidu Inc. †Corresponding author.
samples (Zheng et al., 2023). However, the use of commercial LLMs, such as GPT-4, poses limitations due to their high cost and potentially slow and inconsistent response times, which impede their practical application in large-scale evaluations. In contrast, while open-source LLMs present a more accessible and efficient alternative, they tend to be less effective as evaluators, especially for multiaspect evaluation. For instance, in Table 3, we observed that without further training, Llama (Touvron et al., 2023) based GPTScore fails in effectively assessing dialog-level data.
In this paper, we aim at investigating the construction of a general multi-aspect text evaluator based on open-source LLMs. Constructing such an evaluator is challenging mainly due to the following two reasons: (1) Limited resources: Training a text evaluator is hindered by the scarcity of annotated samples. (2) Complexity of evaluation instructions: An effective text evaluator is required to comprehend intricate task instructions and as-
13462
sess text samples across multiple aspects. To address these challenges, we propose to fine-tune LLMs using publicly available Human Preference Modeling (HPM) datasets. These datasets are designed to capture human preferences regarding responses to various questions and instructions. Despite their human-assessed nature, HPM datasets represent valuable labeled resources, offering a diverse and rich set of instructions, as outlined in Table 1. These diverse and rich instructions empower language models not only to evaluate the overall quality of the target task but also to extend their capability for assessing various aspects and instructions in a zero-shot manner (Chung et al., 2022; Longpre et al., 2023).
To put this insight into practice, we propose InstructEval, a multi-aspect text evaluator that leverages instruction tuning on human preference datasets. Initially, we establish a standardized prompt format, termed “eval-instruct”, designed to unify Human Preference Modeling (HPM) data from diverse sources. This format is composed of three integral parts: the task definition, the aspect definition, and the target sample. Such a structure allows for the clear delineation and decomposition of both explicit and implicit evaluation criteria within the HPM data. Subsequently, we engage in the fine-tuning of LLMs using the entirety of available open-source HPM data structured in the eval-instruct format. This process substantially enhances the LLMs’ capabilities in assessing textual quality comprehensively. To further refine the LLMs’ proficiency in complex multi-aspect evaluations, we construct a small set of multi-aspect evalinstruct for augmentation. This set is generated through a methodical process of random sampling, combining various task instructions and aspect definitions, and employing GPT-4 for the creation of specific assessment preferences. Through joint finetuning of LLMs using both the publicly available HPM data and our constructed multi-aspect labeled data, InstructEval not only demonstrates improved efficacy in evaluating the overall quality of text samples but also exhibits enhanced performance in multi-aspect evaluation tasks.
Extensive experimentation in both overall helpfulness and multi-aspect evaluation demonstrate that our proposed method, InstructEval, achieves comparable or even superior performance to ChatGPT or GPT-4, despite fine-tuning with only 7B or 13B parameters of Llama. Furthermore, we con-
duct a thorough analysis to examine the contributions of each resource in enhancing the evaluation capabilities of InstructEval. This analysis allows us to gain valuable insights into the specific benefits provided by each resource. Our contributions can be summarized into three folds:
• We propose a novel instruction-tuning method to make full use of human preference modeling datasets in text evaluation. The model and dataset will be released to the community.
• A general and multi-aspect text evaluator InstructEval is implemented and achieves comparable or superior performance to commercial LLMs like ChatGPT or GPT-4.
• Our experiments reveal the relations between overall human preference and multi-aspect evaluations. Additionally, we provide a thorough analysis on how current available HPM resources benefit text evaluation.

heading: 2 Preliminaries and Related Works
text: Our work is closely related to the three domains discussed below, and there are subtle connections among these three domains as well.
Instruction tuning Instruction tuning is a technique employed to assist LLM in better comprehend and respond to a wide range of diverse instructions provided by humans (Brown et al., 2020). Formally, with each task t ∈ T written in explicit instruction (may also include a task input), instruction tuning trains a model by maximizing:
E t∼T ,y∼t [p(y|t)] , (1)
where y is either written by human, such as Flan (Longpre et al., 2023), Supernatrual (Wang et al., 2022), etc., or generated by other LLMs, such as Alpaca (Taori et al., 2023).
Human Preference Modeling (HPM) HPM, often referred to as Reward Modeling (RM) in the literature on Reinforced Learning from Human Feedback (RLHF), leverages human preference samples to train an evaluation model (Christiano et al., 2017; Stiennon et al., 2020). The main goal of HPM is to ensure that the evaluation model aligns with human judgment and accurately assesses the quality or performance of given samples. HPM is usually performed in a pairwise manner: given a pair of output
samples (yw,yl), where yw is the sample human preferred over yl, HPM trains the evaluation model by maximizing:
E t∼T ,(yw,yl)∼t [p(yw ≻ yl|t)] , (2)
where p(yw ≻ yl|t) denote the probability of evaluation model select yw over yl. Current HPM in text usually involves finetuning multiple domains and instructions (Ouyang et al., 2022).
Text Evaluation Models Previous text evaluation mainly focus on assessing a single task (Kryscinski et al., 2020) or a set of tasks in generation (Zhang et al., 2020; Wu et al., 2023). Benefiting from the powerful generalization ability of LLMs, current text evaluation models have the capacity to assess any given task. An evaluation model M should evaluate the quality of sample y based on task requirement t and the required evaluation aspect ai by outputting a score sta = M(y|t, ai). Recent works focus on how to prompting general LLMs for text evaluation. GPTScore (Fu et al., 2023) and G-EVAL (Liu et al., 2023) study how to better use GPTs for multi-aspect text evaluation. Zheng et al. (2023) investigate the possibility of using GPT-4 as a replacement for human experts as evaluators and discovered its positional biases.

heading: 3 InstructEval
text: In our proposed InstructEval, our target is to construct a LLM with the ability to evaluate not only from overall quality but also based on specific aspects defined. LLMs should provide objective evaluations even in situation where a particular task and aspect that never appeared in training set.
To achieve this objective, our initial step proposes a standardized prompt format called "evalinstruct", which explicitly defines evaluation task and aspects. Subsequently, we finetune a base LLM using HPM samples to equip the LLM with the capability to evaluate the overall aspect of the given task. In order to further augment the LLM’s capacity for specific aspect assessments, we create an additional set of samples evaluated in diverse aspects. These additional samples are then added to our training set, contributing to the training of our final InstructEval. Intuitively, the design of an InstructEval can be deconstructed into three fundamental components: input format, scoring format and instruction tuning. In the subsequent section, we will provide a detailed introduction to each part.

heading: 3.1 Input Format: Eval-Instruct
text: Previous approaches have employed either pointwise evaluation prompts, which involves a single target sample y in its input, or pairwise evaluation prompts, which involves a pair of target samples. In InstructEval, we choose to employ a pointwise prompt instead of a pairwise prompt, and this choice is motivated by the following reasons: 1. Flexibility in inference: Many application and evaluation scenarios require list-wise ranking, which can be computationally expensive with pairwise estimation due to the O(n2) number of comparisons involved. 2. Memory cost in training: When using pairwise comparisons, including an additional variable y in the prompts significantly increases their length. This can present memory challenges during the fine-tuning LLMs. For point-wise prompt, it is essential to include three essential components:
1. Instruction xt: This defines the requirement of task t in text.
2. Evaluation aspect a: This defines and explains the required evaluation aspect in textual form.
3. Target sample y : This represents the textual output that needs to be evaluated.
Based on the above three components, we propose a standard point-wise input format, called "evalinstruct", for our instruction tuning. This format composes the required components by concatenating them as follows:
eta = xt ⊕ a⊕ y. (3)
In this format, the requirements for evaluation, xt and a, are both defined in natural language. The concatenations ⊕ are performed using a specific template. As illustrated in Figure 1, the evalinstruct includes special tokens: "##Instruction" placed before xt to indicate the start of the instruction. "##Aspect {aspect name}:{aspect definition}" is used to format a and its definition. "##Model Output" and "##Score Output" respectively indicate start of y and score output.

heading: 3.2 Scoring Format
text: Given an eval-instruct as input, the evaluation model first generates its last hidden state h ∈ Rn. We employ and compare two different methods of scoring based on h, namely regression score and expected Likert score.
Regression Score applies a newly initialized linear head W ∈ Rn on the top of pre-trained LLM and output the final logits score by s = W Th, This scoring format has been widely used in the reward models of recent works in RLHF (Ouyang et al., 2022; Touvron et al., 2023). However, this format shifts the model from generating tokens to classifying representations, which introduce inconsistencies between the pre-training and finetuning.
Expected Likert Score (ELS) Instead introducing new parameters, ELS generates number tokens (0-9) from the word list. To ensure efficient gradient back-propagation during training, we use the expected score of generating numbers:
sta = ∑
w∈Vn w ∗ p(w|eta)/
∑
w∈Vn p(w|eta), (4)
where Vn = {0, . . . , 9} is the list of number tokens in the word list, p(w|eta) denotes the likelihood of generating token w. The denominator of this equation normalizes the probability distribution over w ∼ Vn. The benefit of using ELS is that it maintains the same generation paradigm as in the pre-training phase. However, it is also important to consider that ELS constrains the score within the range of [0, 9]. This limitation may restrict the expressive range of the scores, potentially limiting the evaluator’s ability to provide nuanced assessments. We further compare with these two scoring formats through experiments.

heading: 3.3 Instruction Tuning With Aspects
text: We now delve into explaining the training process of the evaluator. Following HPM, given a pair of eval-instruct (ewta, e l ta) for task t, where e w ta is preferred by human than elta under the aspect a, we first calculate their evaluated score by model as (swta, s l ta) and train the model by:
L = − ∑
t∼T ,a∈A log σ(swta − slta), (5)
where T and A are the set of ranking tasks and aspects, σ is the sigmoid function.
The remaining challenge lies in collecting a sufficient number of eval-instructs that are evaluated under diverse aspects. We apply two types of data to reach this goal: HPM datasets and our constructed Multi-aspect eval-instruct data.
HPM Data with Overall Aspect Reported in first block in Table 1, publicly available HPM datasets offer a substantial number of samples on overall aspects evaluation. As a result, fine-tuning with these datasets leads to an evaluation model that closely aligns with human preferences, enabling comprehensive evaluations of the samples. In practice, as illustrated on the right side of Figure 1, we set the aspect name of these samples to "Overall" and provide a comprehensive aspect definition.
Multi-aspect Eval-instruct Construction We construct an addition multi-aspect eval-instruct to ensure the ability of multi-aspect evaluation. Although the possibilities for task instructions are infinite, commonly used evaluation aspects are often limited and shared across different tasks. For instance, summary evaluation commonly involves 4 aspects (Fabbri et al., 2021; Zhang et al., 2019), while dialogue evaluation employs 7 aspects (Mehri and Eskenazi, 2020), with coherence, consistency, and relevance being shared aspects between them.
Leveraging this characteristic, we first list out a full aspect list involving all the commonly used evaluation aspects and their corresponding definitions. For convenience and flexibility, we prompt ChatGPT to generate this full aspect list, as it has demonstrated great proficiency in generating a wide array of aspects and their definitions. Next, for a given HPM dataset (first block of Table 1) with a task type t, we carefully select a candidate aspect set At from the previously generated full aspect list. This selection process ensures that the chosen aspects are relevant and appropriate for evaluating the task t defined in the dataset. Subsequently, we randomly sample several data points from task t and assign them with an aspect randomly selected from the aspect set At. In the final step, we utilize GPT-4 (OpenAI, 2023) to relabel these samples with various assigned aspect.
We select 500 samples from each dataset, excluding Harmful-HH, and filter out some samples to ensure label balance. Additionally, we incorporate human-annotated, multi-aspect data from OpenAISum, referred to as “OpenAI-Sum aspect”. The detailed statistics of this multi-aspect eval-instruct data are presented in the second block of Table 1. We combine these data with HPM dataset, reformulating them into eval-instruct format for fine-tuning. For further details on this process, please refer to Appendix 7.1.

heading: 4 Training and Experiment Setup

heading: 4.1 Training Settings
text: Training Datasets The overall statistics of preprocessed training datasets are listed in Table 1. The open-sourced HPM datasets are Stanford Human Preference (SHP), WebGPT, OpenAI-Sum, Helpful-HH and Harmful-HH, Instruct-GPTJ, and an additional Openai-Sum aspect. More details of our training data is provided in Appendix 7.2.
Model Settings We select Llama v2, current state-of-the-art open-source pretrained language model, as the base model for fine-tuning. Two versions of the model are trained, one with 7 B parameters and a larger variant with 13 B parameters. Detailed hyperparameters for finetuning are provided in Appendix 7.3.

heading: 4.2 Experiment Settings
text: We conducted validation experiments on two main categories of evaluation tasks. Notice: We have conducted a careful examination to ensure that there are no instances of overlapping samples between our training data and the testing benchmark.1
Multi-aspect evaluation This evaluation category focuses on assessing generated samples from various specific aspects. Each task within this category typically focuses on a single type of evaluation. For this evaluation, we selected three tasks: SummEval (Fabbri et al., 2021) for text summarization, FED (Mehri and Eskenazi, 2020) for dialoglevel and turn-level dialogue evaluation, we denote them as FED-D and FED-T, respectively. These tasks require ranking multiple samples based on their evaluation scores and measure the correlation
1For instance, the articles in OpenAI-Sum aspect are sourced from Reddit posts, whereas the articles in SumEval are derived from CNN/DM news sources.
between these scores and human judgments using metrics like Spearman, Pearson, and Kendall.
Overall aspect evaluation This evaluation category involves assessing the overall quality of generated samples for arbitrary instructions. We chose MT-benchmark (Zheng et al., 2023) for validation, which includes up to 7 types of instructions and model-generated samples, covering areas such as writing, mathematical reasoning, and code generation. In this benchmark, pairs of samples with human expert preferences are provided, and the evaluation model is tasked with determining which sample is better from the aspect of overall helpfulness. Accuracy is used as the metric to judge the agreement between model judgments and the majority of human expert votes. MT-benchmark includes two stages of evaluation: Stage-1, which consists of a one-turn dialogue, and Stage-2, which involves a second-turn dialogue for evaluation.
Baselines For the detailed introduction to the baseline methods, please refer to Appendix 7.4

heading: 5 Results

heading: 5.1 Multi-aspect Evaluation
text: Overall Performance From results from Table 2, 3, and 4, it is clear that InstructEval, with both 7B and 13B parameters, exhibits competitive performance and often surpasses other commercial LLMs (GPTs) across a range of datasets on average. In the case of SummEval, the best-performing InstructEval surpasses ChaGPT-based G-EVAL by 14% and 12% in terms of Spearman and Kendall correlations, respectively. It also performs on par with GPT-4 based G-EVAL-4. On FED-D and FED-T, the best InstructEval outperforms the highest-performing GPTScore by 2.7% and 2.8%, respectively. Increasing the model’s parameter size from 7B to 13B
yields a significant improvement in average performance. Specifically, there is a 1.9% and 2.5% increase in Spearman and Kendall on the SummEval, and a 2.5% and 1.7% increase in Spearman on the FED-D and FED-T respectively.
Regression Score vs Expected Likert Score Despite the continued training with the pretrained language model head, the expected Likert score (InstructEval-ELS) consistently exhibits inferior performance compared to the regression score (InstructEval-reg). We believe the reason behind this is that the limitations imposed by restricting the output spaces outweigh the benefits gained from reusing the language model head.
Finetuning with Overall vs Multi-aspects It is interesting to observe that w/o aspects (InstructEvals only trained with HPM data ) are able to generalize to specific aspect definitions. Additionally, we discovered w/o aspects perform well on some aspects that correlates with informativeness. For instance, the performance of InstructEval-reg (13B, w/o aspects) on the relevance of the summary surpasses that of GPT-4. However, it is important to note that the improvements achieved in multiaspect evaluation are limited. By incorporating
annotated multi-aspect data, InstructEval improves by 5.3%, 1.6%, 2.6% in terms of Spearman on SummEval, FED-D and FED-T, respectively.
Zero-shot Ability on Aspects and Tasks In our research, while certain tasks and aspects, such as coherence and relevance in summarization, are directly annotated in the training set (e.g., OpenAIsum aspect), most task and aspect combinations in the benchmarks are predicted in a near zero-shot manner. Specifically, for the finetuned aspects in SummEval, we noted average improvements of 4.1% and 6.2% in Spearman scores for the 7B and 13B versions of InstructEval-reg, respectively. It’s important to note that while training samples from Helpful-HH also involve dialogue, their task formats differ from our evaluation methods in FED-D and FED-T. As indicated in Table 3 and 4, the annotated aspects show average improvements of 3.0% and 3.1% for the 13B model, respectively. Improvements for zero-shot aspects were observed at 0.1% and 2.2%, respectively. Overall, our findings suggest that multi-aspect eval-instruct not only significantly enhances directly annotated aspects but also improves zero-shot capabilities for unseen aspect definitions.

heading: 5.2 Overall Aspect Evaluation
text: The main focus of this evaluation is the generalization ability of diverse instructions, and we select the strong ChatGPT as our primary baseline. To utilize ChatGPT as an evaluator, we implement four different approaches: 1. ChatGPT-point: This approach applies a point-wise scoring method similar to InstructEval. We ask ChatGPT to predict a Likert score between 1 and 10. 2. Chat-pair: This approach employs a pairwise comparison method. We present ChatGPT with two options (sample A and sample B) and ask it to choose the better option. 3. ChatGPT-reverse: In this approach, we reverse the order of the options presented in Chat-pair. ChatGPT is asked to predict the preferred option between sample B and sample A. 4. ChatGPT-jointly combines both ChatGPT-pair and ChatGPT-reverse to eliminate position bias, similar to the approach used by Zheng et al. (2023).
Based on the results presented in Table 5, it is
evident that InstructEval consistently demonstrates its superiority over ChatGPT-point in all settings using the same point-wise scoring approach. When comparing with the pairwise methods, InstructEval initially underperforms compared to ChatGPTs in Stage-1 but surpasses them in Stage-2. Increasing the parameter size only improves the performance in Stage-2 while being detrimental to Stage-1. We speculate that this is because the main training dialogue samples, Helpful-HH and Harmful-HH, primarily consist of multi-turn dialogues. Increasing the parameter size causes the model to focus more on capturing multi-turn features while potentially neglecting single-turn interactions. After adding multi-aspect eval-instruct, we observe that they are beneficial to the Stage 1 of overall aspects. Additionally, by only using the best contributing Helpful-HH dataset (according to Table 6), InstructEval-HH gains an further improvements in Stage-1. This highlights the untapped potential
in optimizing the data distribution of HPM, which we will leave for future work.

heading: 5.3 Ablation Study on Training Data
text: To gain a comprehensive understanding of how different types of data impact the performance, we conducted a comprehensive ablation study on the composition of the training set. The first block of Table 6 presents the independent performance of training Llama using each HPM dataset individually. In the second block, "ALL" represents the utilization of a mixture of all the datasets in the first block. "+Sum-asp" further incorporate "OpenAI-sum asp" alongside the "ALL" dataset, and "+ALL-asp" adds both "OpenAI-sum asp" and our constructed "Multi-aspect eval-instruct". The results of SumEval(Sum), FED-T, and FED-D are average Spearman of different aspects. MT reports the average performance of stage-1 and stage-2
without "tie" labels.
Ablation on HPM Data Results in Table 6 reveals significant variations across datasets in performance, despite all samples being labeled under the general aspect of "overall helpfulness." Merely having a large amount of data, "SHP" for example, does not guarantee superior performance. Surprisingly, OpenAI-Sums, trained exclusively on summarization data, exhibits strong performance across multiple tasks, including summarization, turn-level dialogue, and overall aspect evaluation. However, its performance in dialog-level evaluation is comparatively weaker due to the absence of multi-turn dialogue in training. On the other hand, training with "All" datasets achieves a well-balanced performance across all evaluation criteria, indicating its effectiveness in handling diverse tasks. In realworld scenarios, where testing samples often exhibit greater diversity than the benchmarks, training with a comprehensive datasets encompassing all relevant aspects is likely to lead to improved performance and significance.
Multi-Aspect Data The incorporation of specific aspect datasets consistently enhances performance in both overall and specific aspects evaluation (second block vs first block in Table 6 ). Notably, the inclusion of constructed "Multi-aspect instruct" data demonstrates a stable improvement in all multiaspect evaluation ("+All-asp" vs "+Sum-asp").

heading: 6 Conclusion and Future Research
text: This paper has addressed the challenges of text evaluation with LLMs. The major insight of this study lies in leveraging HPM resources for text evaluation. By utilizing the abundant HPM datasets, the study demonstrates the potential of fine-tuning LLMs to evaluate specific aspects and overall quality in a zero-shot manner. Additionally, this paper propose to use a multi-aspect instruct set constructed by GPT-4, to enhance the performance of multi-aspect evaluation.
The insights gained from the analysis of resource contributions in this paper can guide future research in designing more effective evaluation methodologies. Understanding the specific benefits provided by different resources can inform the selection and utilization of appropriate datasets for evaluation tasks.
Limitations
The evaluation of LLMs heavily relies on the availability of labeled samples for fine-tuning. In this study, we focus on leveraging human preference modeling (HPM) resources for evaluation. However, the HPM datasets used may not cover the full range of evaluation aspects required for comprehensive text evaluation. The reliance on specific HPM datasets limits the generalizability of the proposed approach to evaluate a broader range of tasks and aspects. Future research should explore strategies to incorporate more diverse and representative labeled samples to enhance the evaluation capabilities.

heading: 7 Appendix

heading: 7.1 Multi-aspect Data Construction
text: Table 8 lists all the aspects and their definitions along with their appeared datasets. These aspect
definitions were initially generated by ChatGPT, as described in Table 7, and subsequently revised and curated by human experts.

heading: 7.2 Training Data Details
text: The overall statistics of preprocessed training datasets are listed in Table 1. The open-sourced HPM datasets are Stanford Human Preference (SHP)2, WebGPT3, OpenAI-Sum4, Helpful-HH and Harmful-HH5, Instruct-GPTJ 6, and an additional Openai-Sum aspect. More details of our training data is provided in Appendix 7.2.
For SHP, which additionally provide confidence annotations, we apply a filtering criterion where we exclude samples with confidence ratios smaller than 5 to ensure data quality. For our constructed Multi-aspect instruction samples, the instructions are randomly selected from SHP, WebGPT and Helpuful-HH and. For OpenAI-sum, since it provide additional labeled specific aspect preferences as a validation split, we directly incorporate it in training set. Overall, the training set contains 727,070 eval-instruct, 655,101 of them are overall aspect, only 3,500 of them are specific aspects annotated by GPT-4.

heading: 7.3 Model Training Details
text: We fixed the hyper-parameters for all models with a learning rate of 1e−5 with no weight decay and linear warmup for 3% of the total training steps. We use a batch size of 256, and a maximum sequence length of 1024 and truncate exceeded tokens. The overall dataset is trained for only one epoch on 8 NVIDIA A800 gpus with 80GB of VRAM.

heading: 7.4 Baseline Methods
text: Our primary baselines consist of evaluation models built on large language models (LLMs), with a particular focus on GPTs. For instance, GPTscore (Fu et al., 2023) is used as a baseline for text generation evaluations, while G-EVAL (Liu et al., 2023) serves as a baseline for text summarization. Also, we apply our base model, Llama, as baseline.
2https://huggingface.co/datasets/stanfordnlp/ SHP
3https://huggingface.co/datasets/openai/ webgpt_comparisons
4https://huggingface.co/datasets/openai/ summarize_from_feedback
5https://huggingface.co/datasets/Anthropic/ hh-rlhf
6https://huggingface.co/datasets/Dahoas/ synthetic-instruct-gptj-pairwise
Because Llama without instruction tuning cannot follow instructions to evaluate, we adopt the scoring approach of GPT-score, i.e. scoring the target output by its perplexity. We also test Llama-Chat, the version of Llama trained with RLHF, in both GPTScore and ELS way of scoring.

heading: Future Work
text: The insights gained from the analysis of resource contributions in this paper can guide future research in designing more effective evaluation methodologies. Understanding the specific benefits provided by different resources can inform the selection and utilization of appropriate datasets for evaluation tasks. Limitations
The evaluation of LLMs heavily relies on the availability of labeled samples for fine-tuning. In this study, we focus on leveraging human preference modeling (HPM) resources for evaluation. However, the HPM datasets used may not cover the full range of evaluation aspects required for comprehensive text evaluation. The reliance on specific HPM datasets limits the generalizability of the proposed approach to evaluate a broader range of tasks and aspects. Future research should explore strategies to incorporate more diverse and representative labeled samples to enhance the evaluation capabilities