Venue: NAACL 2024

# GPTScore: Evaluate as You Desire

Jinlan Fu1, See-Kiong Ng1, Zhengbao Jiang2, Pengfei Liu3,

1National University of Singapore, 2Carnegie Mellon University

3Shanghai Jiao Tong University

jinlanjonna@gmail.com, pengfei@sjtu.edu.cn

###### Abstract

Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., in-context learning, zero-shot instruction) of **g**enerative **pre**-trained models to **score** generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., Flan-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, \(22\) evaluation aspects, and corresponding \(37\) datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation-how to achieve customized, multi-faceted evaluation without model training. We make our code publicly available. 1

Footnote 1: [https://github.com/jinlanfu/GPTscore](https://github.com/jinlanfu/GPTscore)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 6556–6576 June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

The advent of generative pre-trained models, such as GPT3 Brown et al. (2020), has precipitated a shift from _analytical_ AI to _generative_ AI across multiple domains Sequoia (2022). Take text as an example: the use of a large pre-trained model with appropriate prompts Liu et al. (2021) has achieved superior performance in tasks defined both in academia Sanh et al. (2021) and scenarios from the real world Ouyang et al. (2022). While text generation technology is advancing rapidly, techniques for evaluating the quality of these texts lag far behind. This is especially evident in the following ways:

(1) Existing studies Ghazarian et al. (2022); Ye et al. (2021) evaluate text quality with limited aspects (e.g., fluency) (Fig. 1-(a)), which are usually customized prohibitively, making it harder for users to evaluate aspects _as they need_Freitag et al. (2021). (2) A handful of studies Yuan et al. (2021); Scialom et al. (2021); Zhong et al. (2022); Li et al. (2021); Mehri and Eskenazi (2020) have examined multi-aspect evaluation but lack carefully studied aspects' definitions and their relationship. Moreover, the specific aspect evaluations are empirically bound with metric variants (Fig. 1-(b)). (3) Rely on annotated samples and model training. Most of the above methods necessitate complicated supervised training or costly manual annotation of samples (Fig. 1-(a,b)). This makes these methods hard to use in industrial settings and adapt to new evaluation aspects required by users.

In this paper, we demonstrated the talent of the super large pre-trained language model (e.g., GPT-3) in achieving multi-aspect, customized, and training-free evaluation (Fig. 1-(c)). Essentially, it skillfully utilizes the pre-trained model's zero-shot instruction Chung et al. (2022) and in-context learning Brown et al. (2020); Min et al. (2022) ability to deal with complex and ever-changing evaluation needs while solving multiple evaluation challenges that have plagued many years. Specifically, given a text generated from a specific context (e.g., source text in text summarization) and a desirable evaluation aspect (e.g., fluency), the high-level idea of the proposed framework is that

Figure 1: An overview of text evaluation approaches.



the higher-quality text for a specific aspect will be more likely generated than unqualified ones, where the "likely" can be measured by the conditional generation probability.

**How to perform an evaluation as the user desires?** As illustrated in Fig. 2, to capture users' true desires, an _evaluation protocol_2 will be initially established based on (a) the _task specification_, which typically outlines how the text is generated (e.g., generate a response for a human based on the conversation); (b) _aspect definition_ that documents the details of desirable evaluation aspects (e.g., the response should be intuitive to understand); (c) _demonstrated samples_: a handful of well-labeled samples are required to teach the model which sample is qualified. Subsequently, each evaluation sample will be presented with the evaluated protocol with optionally moderate exemplar samples, which could facilitate the model's learning. Lastly, a generative pre-trained model will be used to calculate how likely the text could be generated based on the above evaluation protocol, thus giving rise to our model's name: GPTScore. Given the plethora of pre-trained models, we instantiate our framework with different backbones: GPT2 [15], OPT [30], FlanT5 [47], and GPT3 (instruction-based [68]) due to their superior capacity for _zero-shot instruction_ and their aptitude for _in-context learning_.

Footnote 2: To better understand how to design the evaluation protocols, we give all the evaluation protocols for the different tasks and aspects studied in this work in the Appendix F.

Experimentally, we ran through almost all common natural language generation tasks in NLP, and the results showed the power of this new paradigm. The main observations are listed as follows: (1) GPTScore performs better when instructed by the definition of task and aspect. Furthermore, incorporating suitable exemplified samples with incorrect learning will further enhance the process. (2) Different evaluation aspects exhibit certain correlations. By incorporating definitions with other highly correlated aspects (e.g., interesting and engaging), the performance of the smaller model (GPT3-curie, 6.7B) can surpass the larger model (GPT3-davinci, 175B). (3) The GPTscore performs better than fine-tuned models across tasks such as text summarization, data-to-text, and dialogue response generation. (4) The performance of _GPT3-text-davinci-003_, which is tuned based on human feedback, is inferior to _GPT3-text-davinci-001_ in the majority of the evaluation settings.

Our main **contributions** in this paper are:

(1) We propose a newly generated text scoring framework, GPTScore, which utilizes the emergent ability of large language model to achieve multi-aspect, customized, and training-free evaluation.

(2) We comprehensively explore GPTScore, studying 19 language models (ranging in size from 80M to 175B) and four popular text generation tasks. Experiments demonstrate that training-free GPTScore outperforms fine-tuning model and achieves higher human correspondence.

(3) We design and demonstrate the feasibility of custom evaluation for a new aspect by the GPTScore framework with training-free.

(4) We summarize some observations of the opaque GPT3 family and other backbone models in the evaluation and try to give explanations.

## 2 Related Work

**Similarity-based Metrics** measures the similarity between the generated text and the reference text. It includes two types: (1) lexical overlap-based

Figure 2: The framework of GPTScore. We include two evaluation aspects _relevance (REL)_ and _informative (INF)_ in this figure and use the evaluation of _relevance (REL)_ of the text summarization task to exemplify our framework.

 

metrics, e.g., BLEUU Papineni et al. (2002) and ROUGE Lin (2004); (2) embedding-based metrics, e.g., BERTScore Zhang et al. (2020) and MoverScore Zhao et al. (2019).

**Single-aspect Evaluator** refers to evaluators designed to evaluate the quality of a specific aspect or overall of the generated text. For example, DEAM Ghazarian et al. (2022) and QuantiDCE Ye et al. (2021) were proposed for the evaluation of the _coherence_ of the dialogue system; several evaluators Cao et al. (2020); Durmus et al. (2020); Wang et al. (2020) are designed for the evaluation of the _consistency_ of text summarization.

**Multi-aspect Evaluator** refers to one evaluator handle several evaluation aspects by using different input and output text pair Yuan et al. (2021), different prompt designed by the aspect name Zhong et al. (2022); Mehri and Eskenazi (2020), Mehri and Eskenazi (2020), different formulas Scialom et al. (2021). Unlike Zhong et al. (2022); Mehri and Eskenazi (2020) which only consider the vague aspect description, we fully considered exhaustive aspect definition and their relationship.

**Emergent Ability** Recent works have revealed various emergent abilities of generative pre-trained language models, such as, in-context learning Min et al. (2022), chain-of-thought reasoning Wei et al. (2022), and zero-shot instruction Ouyang et al. (2022). These abilities allow large language models to achieve good performance without training.

What is the difference between our work and BARTScore?Yuan et al. (2021) demonstrates the feasibility of using the probability of text generation as a text quality score, and fine-tuning is required to achieve better performance. However, the model fine-tuning cost a lot, and it is hard for us to fine-tune a scoring model for each task and each domain. In this work, we focus on proposing a new framework that allows the generated text evaluation to achieve customizable, multi-faceted, and train-free evaluation. To achieve this target, (1) we utilized the emergent ability of language models, such as in-context learning, zero-shot instruction et al., to build the GPTScore. (2) By studying the GPTscore framework on 19 language models covering four backbones, we demonstrate that (a) GPTScore outperforms the fine-tuned BARTScore; (b) GPTScore can be customized for a new evaluation aspect with a labeled handful of samples while BARTScore cannot do this.

## 3 Generative Pretraining Score (GPTScore)

The core idea of GPTScore is that a generative pre-training model will assign a higher probability of high-quality generated text following a given instruction and context. In our method, the instruction is composed of the task description \(d\) and the aspect definition \(a\). Specifically, suppose that the text to be evaluated is \({\mathbf{h}}=\{h_{1},h_{2},\cdots,h_{m}\}\), the context information is \(\mathcal{S}\) (e.g., source text or reference text), then GPTScore is defined as the following conditional probability:

\[\mathrm{GPTScore}({\mathbf{h}}|d,a,\mathcal{S})=\\ \sum\nolimits^{m}_{t=1}w_{t}\log p(h_{t}|{\mathbf{h}}_{<t},T(d,a, \mathcal{S}),\theta),\]

where \(w_{t}\) is the weight of the token at position \(t\). In our work, we treat each token equally. \(T(\cdot)\) is a prompt template that defines the evaluation protocol, which is usually task-dependent and specified manually through prompt engineering.

**Few-shot with Demonstration** The generative pre-trained language model can better perform tasks when prefixed with a few annotated samples (i.e., demonstrations). Our proposed framework is flexible in supporting this by extending the prompt template \(T\) with demonstrations.

**Choice of Prompt Template**Prompt Liu et al. (2021); Fu et al. (2022) templates define how task description, aspect definition, and context are organized. In this work, for the GPT3-based model, we opt for prompts that are officially provided by OpenAI.3 For instruction-based pre-trained models, we use prompts from NaturalInstruction Wang et al. (2022) since it's the main training source for those instruction-based pre-train models. Taking the evaluation of the fluency of the text summarization task as an example, based on the prompt provided by OpenAI,4 the task prompt is "{Text} TI;dr {Summary}", the definition of fluency is "Is the generated text well-written and grammatical?" (in Tab. 1), and then the final prompt template is "Generate a fluent and grammatical summary for the following text: {Text} Tl;dr {Summary}", where demonstrations could be introduced by repeating instantiating "{Text} Tl;dr {Summary}" In Appendix F, we list the prompts for various aspects of all tasks studied in this work and 

leave a more comprehensive exploration on prompt engineering as a future work.

**Selection of Scoring Dimension GPTScore exhibits different variants in terms of diverse choices of texts being calculated. For example, given a generated hypothesis, we can calculate GPTScore either based on the source text (i.e., _src->hypo_, \(p({\rm hyppo|src})\)) or based on the gold reference (i.e., _ref->hypo_, \(p({\rm hypo|ref})\)). In this paper, the criteria for choosing GPTScore variants are mainly designed to align the protocol of human judgments Liu et al. (2022) that are used to evaluate the reliability of automated metrics. We will detail this based on different human judgment datasets in the experiment section.

## 4 Experimental Settings

### Meta Evaluation

Meta evaluation aims to evaluate the reliability of automated metrics by calculating how well automated scores (\(y_{\text{auto}}\)) correlate with human judgment (\(y_{\text{human}}\)) using correlation functions \(g(y_{\text{auto}},y_{\text{human}})\) such as spearman correlation. In this work, we adopt two widely-used correlation measures: (1) **Spearman** correlation (\(\rho\)) Zar (2005) and (2) **Pearson** correlation (\(r\)) Mukaka (2012).

### Tasks, Datasets, and Aspects

To achieve a comprehensive evaluation, in this paper, we cover a broad range of natural language generation tasks: _Dialogue Response Generation_, _Text Summarization_, _Data-to-Text_, and _Machine Translation_, which involves \(37\) datasets and \(22\) evaluation aspects in total. Tab. 8 summarizes the tasks, datasets, and evaluation aspects considered by each dataset. The definition of different aspects can be found in Tab. 1. More detailed illustrations about the datasets can be found in Appendix D.

(1) **Dialogue Response Generation** aims to generate an engaging and informative response based on the dialogue history. We adopt the FED Mehri and Eskenazi (2020) datasets and consider both turn-level and dialogue-level evaluations. (2) **Text Summarization** is a task of automatically generating informative summary for a given long text. We adopt SummEvalBhandari et al. (2020), REALSummBhandari et al. (2020), NEWSROOM Grusky et al. (2018), and QAGS_XSUM Wang et al. (2020) datasets. (3) **Data-to-Text** aims to generate a fluent and factual description for a given table. We consider BAGEL Mairesse et al. (2010) and SFRES Wen et al. (2015) datasets. (4) **Machine Translation** aims to translate a sentence from one language to another. We consider a subdatasets of Multidimensional Quality Metrics (MQM) Freitag et al. (2021), namely, MQM-2020 Chinese->English).

### Scoring Models

**ROUGE Lin (2004)** is a popular automatic generation evaluation metric. We consider three variants ROUGE-1, ROUGE-2, and ROUGE-L. **PRISM Thompson and Post (2020)** is a reference-based evaluation method designed for machine translation with pre-trained paraphrase systems.

\begin{table}
\begin{tabular}{l l p{196.7pt}} \hline \hline
**Aspect** & **Task** & **Definition** \\ \hline Semantic Coverage (COV) & Summ & How many semantic content units from the reference text are covered by the generated text? \\ Factuality (FAC) & Summ & Does the generated text preserve the factual statements of the source text? \\ Consistency (CONN) & Summ, Diag & Is the generated text consistent in the information it provides? \\ Informativeness (INFN) & Summ, DIZ, Diag & How well does the generated text capture the key ideas of its source text? \\ Coherence (COH) & Summ, Diag & How much does the generated text make sense? \\ Relevance (REL) & Ding, Summ, DTT & How well is the generated text relevant to its source text? \\ Fluency (FLU) & Diag, Summ, DIZT, MT & Is the generated text-written and grammatical? \\ Accuracy (ACC) & MT & Are there inaccuracies, missing, or unfatcal context in the generated text? \\ MQM & MT & How is the overall quality of the generated text? \\ Interest (INT) & Diag & Is the generated text interesting? \\ Engagement (ENG) & Diag & Is the generated text engaging? \\ Specific (SPE) & Diag & Is the generated text generic or specific to the source text? \\ Correctness (CORR) & Diag & Is the generated text correct or was there a misunderstanding of the source text? \\ Semantically & Diag & Is the generated text semantically appropriate? \\ Appropriate (SEM) & Diag & Is the generated text understandable? \\ Understandingability (UND) & Diag & Is the system able to recover from errors that it makes? \\ Error Recovery (ERR) & Diag & Is there diversity in the system responses? \\ Diversity (DIV) & Diag & Does the system discuss topics in depth? \\ Depth (EPP) & Diag & Does the system display a likeable personality? \\ Likeability (LIK) & Diag & Does the system display a likeable personality? \\ Flexibility (FILE) & Diag & Is the system flexible and adaptable to the user and their interests? \\ Imquisitions (INGQ) & Diag & Is the system insensitive throughout the conversation? \\ \hline \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: The definition of aspects evaluated in this work. _Semantic App._ denotes _semantically appropriate_ aspect. _Diag_, _Summ_, _D2T_, and _MT_ denote the _dialogue response generation_, _text summarization_, _data to text_ and _machine translation_, respectively. “MQM” is the short name of _Multidimensional Quality Metrics_.

 



**BERTScore**Zhang et al. (2020) uses contextual representation from BERT to calculate the similarity between the generated text and the reference text. **MoverScore**Zhao et al. (2019) considers both contextual representation and Word Mover's Distance Kusner et al. (2015). **DynaEval**Zhang et al. (2021) is a unified automatic evaluation framework for dialogue response generation tasks on the turn level and dialogue level. **BARTSCore**Yuan et al. (2021) is a text-scoring model based on BART Lewis et al. (2020) without fine-tuning. **BARTScore+CNN** and **BARTSCore+CNN+Para** are the variants of **BARTSCore**, the former is fine-tuned on the CNNDM dataset Hermann et al. (2015), and the latter is fine-tuned on CNNDM and Paraphrase2.0 Hu et al. (2019). **GPTSCore** is our evaluation method, designed based on 19 pre-trained language models, covering GPT3, OPT, Flan-T5, and GPT2 backbones. Tab. 2 shows model variants used in this paper and their number of parameters.

### Scoring Dimension

Specifically, (1) For aspects INT, ENG, SPC, REL, COR, SEM, UND, and FLU of FED-Turn datasets from the open domain dialogue generation task, we choose the _src->hypo_ variant since the human judgments of the evaluated dataset (i.e., FED-Turn) are also created based on the source. (2) For aspects COH, CON, and INF from SummEval and Newsroom, since data annotators labeled the data based on source and hypothesis texts, we choose _src->hypo_ for these aspects. (3) For aspects INF, NAT, and FLU from the data-to-text task, we choose _ref->hypo_. Because the source text of the data-to-text task is not in the standard text format, which will be hard to handle by the scoring function.

### Evaluation Dataset Construction

Unlike previous works Matiana et al. (2021); Xu et al. (2022a, b); Castricato et al. (2022) that only consider the overall text quality, we focus on evaluating multi-dimensional text quality. In this work, we studied 37 datasets according to 22 evaluation aspects. Since each sample needs to evaluate the generated text of dozens of systems, to reduce the API cost of GPT3, we randomly sample 40 samples for each text summarization dataset and 100 samples for each dialogue response generation and data-to-text dataset. For example, in the Newsroom dataset with 60 samples, 40 samples (accounting for 60% of the samples) are randomly selected to construct the evaluation set.

## 5 Experiment Results

In this work, we focus on exploring whether language models with different structures and sizes can work in the following three scenarios. (a) **vanilla****(VAL)**: with non-instruction and non-demonstration; (b) **nstruction****(IST)**: with instruction and non-demonstration; (c) **instruction+demonstration****(IDM)**: with instruction and demonstration. We studied four text generation tasks introduced in Sec. 4.2. Due to the limited space, we moved the results and analysis of the machine translation task into the Appendix A.

**Significance Tests** To examine the reliability and validity of the experiment results, we conducted the significance test based on bootstrapping.5 Our significance test is to check (1) whether the performance of IST (IDM) is significantly better than VAL, and values achieved with the IST (IDM) settings will be marked \({\dagger}\) if it passes the significant test (p-value <0.05). (2) whether the performance of IDM is significantly better than IST, if yes, mark the value with IDM setting with \({\ddagger}\).

Footnote 5: [https://en.wikipedia.org/wiki/Bootstrapping_](https://en.wikipedia.org/wiki/Bootstrapping_) (statistics)

**Average Performance** Due to space limitations, we keep the average performance of GPT3-, GPT2-, OPT-, and FT5-based models. The full results of various variants can be found in Appendix G.

### Text Summarization

The evaluation results of \(28\) (9 baseline models (e.g., ROUGE-1) and 19 variants of GPTScore (e.g., GPT3-d01) scoring functions for the text summarization task on SummEval and RealSumm datasets are shown in Tab. 3. Due to the space limitation,

\begin{table}
\begin{tabular}{l c c c}
**GPT3** & **Param.** & **OPT** & **Param.** \\ \hline GPT3-a01 (text-ada-001) & 350M & OPT350M & 350M \\ GPT3-b01 (text-babbage-001) & 1.3B & OPT-1.3B & 1.3B \\ GPT3-c01 (text-curie-ou01) & 6.7B & OPT-1.3B & 6.7B \\ GPT3-d01 (text-davi nc-ou100) & 175B & OPT-13B & 13B \\ GPT3-d03 (text-advinci-003) & 175B & OPT-66B & 66B \\ \end{tabular} 
\begin{tabular}{l c c}
**Flan-T5** & **Param.** & **GPT2** & **Param.** \\ \hline FT5-small & 80M & GPT2-M & 355M \\ FT5-base & 250M & GPT2-L & 774M \\ FT5-L & 770M & GPT2-XL & 1.5B \\ FT5-XL & 3B & GPTJ-f6B & 6B \\ FT5-XXL & 11B & & \\ \end{tabular}
\end{table}
Table 2: A summary of pre-trained language models studied in this work. _Param._ denotes _Parameter_.



we move the results of the REALSumm, NEWSMOOM, and QXSUM datasets to the Appendix G. Fig. 3 shows the evaluation results of five GPT3 variant models on four text summarization datasets, where QXSUM uses the Pearson correlation and other datasets use the Spearman correlation metric. The main observations are summarized as follows:

(1) **Evaluator with instruction significantly improves the performance**. For the 4 aspects of SummEval datasets, 19 instruction-enhanced variants of GPTScore significantly outperform models without instruction (values with \({\dagger}\) in Tab. 3) and almost 7 unsupervised baseline methods. (2) **Most GPT3- and FT5-based models equipped with instructions outperform supervised methods**. For example, equipped with instructions, FT5-L, FTS-XL, and FT5-XXL significantly outperform the supervised model BARTSc+CNN+Pa for all four aspects of the SummEval dataset. (3) As for the GPT3-based models, (a) **the performance of GPT3-d01 is barely significantly better than GPT3-c01**, which tries to balance power and speed. (b) GPT3-d03 performs better than GPT3-d01 significantly. Both conclusions have passed the significance test at \(p<0.05\) and can be seen in Fig. 3.

### Data to Text

We consider the BagEL and SFRES datasets for the evaluation of data to text task. The average Spearman correlations of the GPT3-based, GPT2-based, OPT-based, and FT5-based models are listed in Tab. 4. VAL, IST, and IDM denote the vanilla, using instruction, and using both instruction and demonstration settings, respectively. Due to the space limitations, detailed results for each evaluator are moved to Appendix G (Tab. 15 and Tab. 16). The main observations are listed as follows:

(1) **Instruction (IST) improves performance, and combining it with demonstrations (IDM) further enhances it.** In Tab. 4, the average performance on the three aspects is significantly improved when adapting to the instruction, and the performance of using demonstration on NAT and FLU has further significantly improved. (2) **Many GPTScore variants enhanced by instruction and demonstration (IDM) outperform the fine-tuned

Figure 3: Experimental results for GPT3-based variants in text summarization task. Here, blue, orange, green, pink, and cyan dot denote that GPTScore is built based on GPT3-a01 (�⃝), GPT3-b01 (⃝), GPT3-c01 (⃝), GPT3-d01 (⃝), and GPT3-d03 (◃�), respectively. The red lines (——) denote the average performance of GPT3-based variants.

 

model**, namely BARTSCORE+CNN+Para. For example, in Tab. 16 regarding NAT and FLU of the SFRES dataset, most of the 19 variants of GPTScore with instructions and demonstrations outperform fine-tuned BARTSCORE+CNN+Para.

(3) **The choice of samples for demonstration impacts the evaluation performance a lot.** On the BAGEL and SFRES datasets, when equipped with instruction and demonstration (IDM), the average performance of the four backbones performs much worse than backbone equipped with instruction (IST) only. (4) **Equipped with instruction and demonstration, the performance of a GPT3 family model with a small model size can surpass that of large models.** In Fig. 4, the performance of GPT3-c01 with IDM always outperforms GPT3-d03, which holds for both datasets.

### Dialogue Response Generation

To test if GPTScore can generalize to more aspects, we choose dialogue response generation task as a testbed, which usually requires evaluating generated texts from a variety of dimensions (i.e., "interesting" and "fluent"). To reduce the computational cost, we focus on GPT3-based metrics only since they have achieved superior performance as we observed in the previous experiments. Tab. 5 shows the perfromance in dialogue response generation task, where both baseline FED and DE (DynaEval) are fine-tuned on a large dialogue corpus. The main observations are listed as follows:

(1) **The performance of GPT3-d01 is much better than GPT3-d03, even though both of them have the same model size.** The average Spearman correlation of GPT3-d01 outperforms GPT3-d03 by **40.8** on the FED Turn-level dataset, and **5.5** on the FED dialogue-level. (2) **The GPT3-based model demonstrate stronger generalization ability.** BART-based models failed in evaluating the dialogue generation task. The GPT3-a01 with 350M parameters achieved comparable performance to FED and DE models, which are fine-tuned on dialogue corpus.

## 6 Ablation Study

### Effectiveness of Demonstration

To investigate the relationship between the demonstration sample size (denote as K) and the evaluation performance, we choose the machine translation task and the GPT3-based variants with model sizes ranging from 350M to 175B for further study.

The change of Spearman correlation on the MQM-2020 dataset with different demonstration sample size are shown in Fig. 5. The main observations are summarized as follows: (1) The utilization of demonstration significantly improves the evaluation performance, which holds for these three aspects. (2) There is an upper bound on the performance gains from the introduction of the demonstration. For example, when K>4, the performance of ACC is hard to improve further. (3) When demonstration has only a few samples (such as K=1), small models (e.g., GPT3-a01) are prone to performance degradation due to the one-sidedness of the given examples.

### Partial Order of Evaluation Aspect

To explore the correlation between aspects, we conducted an empirical analysis with INT _(interesting)_

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{BASEL} & \multicolumn{2}{c}{SFRES} \\ \cline{1-1} \cline{2-6} INF & NAT & FLU & INF & NAT & FLU \\ \hline \end{tabular}
\end{table}
Table 4: The average of Spearman correlation of the models based on GPT3, GPT2, OPT, and FT5 on BAGEL and SFRES datasets in the data-to-text task.

Figure 4: Experimental results for GPT3-based variants in the data-to-text task. Here, blue, orange, green, pink, and cyan dots denote that GPTSCore is built based on GPT3-a01 (�⃝), GPT3-b01 (⃝), GPT3-c01 (⃝), GPT3-d01 (⃝), and GPT3-d03 (⃝). The red lines (—) denote the average performance of GPT3-based variants.

 

on the dialogue response generation task of the FED-Turn dataset. Specifically, take INT as the target aspect and then combine the definitions of other aspects with the definition of INT as the final evaluation protocols. The x-axis of Fig. 6-(a) is the aspect order achieved based on the Spearman correlation between INT and that aspect's human score. Fig. 6-(b) is the Spearman correlation o INT as the modification of the INT definition, and the scoring function is GPT3-c01 with 6.7B parameters.

The following table illustrates the definition composition process, where Sp denotes Spearman.

\begin{tabular}{l l c 



## 8 Limitations

The limitations of this work include: (1) The pre-trained language models considered in our work were released before GPT-3.5 (included), while some recently released popular LLMs (such as ChatGPT and GPT-4) are not studied in this work. (2) GPT3-text-davinci-003 performs worse than GPT3-text-davinci-001, which holds in many evaluation settings. However, we cannot explain this conclusion well until OpenAI discloses the model and training in more details. (3) Due to the cost limitation of using the OpenAI API, we only consider evaluating four traditional NLP generation tasks. The evaluation of some complex text generation tasks (e.g., story generation, a long text generation task) can be studied in the future.

There are some risks associated with model-based evaluation. For example, (1) GPT3-based models work well as text evaluators, but their internal structure, training data, and training process are opaque, which makes it hard to explain the phenomenon of model performance. (2) Training and human feedback datasets influence language model behavior a lot. During the evaluation process, language models may inject bias and risky behaviors that are hard to identify, sending dangerous information to humans. The safety and risk of model-based evaluation should be carefully studied, and we will comprehensively explore the risk for our GPTScore like Wang et al. (2023) in the future.

## Acknowledgements

We thank Chen Zhang for helpful discussion and feedback. This research / project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

## References

* Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. _CoRR_, abs/2001.09977.
* Bhandari et al. (2020) Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Reevaluating evaluation in text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 9347-9359. Association for Computational Linguistics.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pfarulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _CoRR_, abs/2005.14165.
* Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 6251-6258. Association for Computational Linguistics.
* Castricato et al. (2022) Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anhang Ye, Ian Yang, Spencer Frazier, and Mark O. Riedl. 2022. Robust preference learning for storytelling via contrastive reinforcement learning. _CoRR_, abs/2210.07792.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Durmus et al. (2020) Esin Durmus, He He, and Mona T. Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 5055-5070. Association for Computational Linguistics.
* Freitag et al. (2021) Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. _CoRR_, abs/2104.14478.
* Fu et al. (2022) Jinlan Fu, See-Kiong Ng, and Pengfei Liu. 2022. Polyglot prompt: Multilingual multitask promptraining. _arXiv preprint arXiv:2204.14264_.
* Ghazarian et al. (2022) Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022. DEAM: dialogue coherence evaluation using amr-based semantic manipulations. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 771-785. Association for Computational Linguistics

Max Grusky, Mor Naaman, and Yoav Artzi. 2018: Newrsoom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pp. 708-719. External Links: Link Cited by: SS4.
* K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom (2015) Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1693-1701. External Links: Link Cited by: SS4.
* J. H. Hu, A. Singh, N. Holzenberger, M. Post, and B. Van Durme (2019) Large-scale, diverse, paraphrastic bitexts via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, pp. 44-54. External Links: Link Cited by: SS4.
* M. J. Kusner, Y. Sun, N. I. Kolkin, and K. Q. Weinberger (2015) From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, Vol. 37, pp. 957-966. External Links: Link Cited by: SS4.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020) BERT: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7880. External Links: Link Cited by: SS4.
* Z. Li, J. Zhang, Z. Fei, Y. Feng, and J. Zhou (2021) Conversations are not flat: modeling the dynamic information flow across dialogue utterances. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Volume 1: Long Papers, Virtual Event, August 1-6, 2021, pp. 128-138. External Links: Link Cited by: SS4.
* C. Lin (2004) Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81. External Links: Link Cited by: SS4.
* P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig (2021) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. External Links: Link Cited by: SS4.
* Y. Liu, A. R. Fabbri, F. Liu, Y. Zhao, L. Nan, R. Han, S. Han, S. Joty, C. Wu, C. Xiong, et al. (2022) Revisiting the gold standard: grounding summarization evaluation with robust human evaluation. arXiv preprint arXiv:2212.07981. External Links: Link Cited by: SS4.
* F. Mairesse, M. Gasic, F. Jurcicek, S. Keizer, B. Thomson, K. Yu, and S. J. Young (2010) Phrase-based statistical language generation using graphical models and active learning. In ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden, pp. 1552-1561. External Links: Link Cited by: SS4.
* S. Matiana, J. R. Smith, R. Teehan, L. Castricato, S. Biderman, L. Gao, and S. Frazier (2021) Cut the CARP: fishing for zero-shot story evaluation. CoRRabs/2110.03111. External Links: Link Cited by: SS4.
* Sh. Mehri and M. Eskenzazi (2020a) Unsupervised evaluation of interactive dialog with dialogpt. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020, pp. 225-235. External Links: Link Cited by: SS4.
* S. Mehri and M. Eskenzazi (2020b) USR: an unsupervised and reference free evaluation metric for dialog generation. CoRRabs/2005.00456. External Links: Link Cited by: SS4.
* S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022) Rethinking the role of demonstrations: what makes in-context learning work?. CoRRabs/2202.12837. External Links: Link Cited by: SS4.
* M. Mukaka (2012) A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal24 (3), pp. 69-71. External Links: Document, ISSN 1558-1212, Link Cited by: SS4.
* L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Salama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. External Links: Link Cited by: SS4.
* Bo Pang, E. Nijkamp, W. Han, L. Zhou, Y. Liu, and K. Tu (2020) Towards holistic and automatic evaluation of open-domain dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 3619-3629. External Links: Link Cited by: SS4.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002) Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. External Links: Link Cited by: SS4.

 

Maja Popovic. 2015. chrf: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@EMNLP 2015, 17-18 September 2015, Lisbon, Portugal, pp. 392-395. External Links: Link Cited by: SS4.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)Language models are unsupervised multitask learners. OpenAI blog1 (8), pp. 9. Cited by: SS1.
* R. Rei, C. Stewart, A. C. Farinha, and A. Lavie (2020)COMET: a neural framework for MT evaluation. CoRRabs/2009.09025. External Links: Link Cited by: SS1.
* V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafaei, A. Chaffin, A. Stiegler, Y. Le Scao, A. Raja, et al. (2021)Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. Cited by: SS1.
* T. Le Scaco, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagne, A. Sasha Luccioni, F. Yvon, M. Galle, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. Sasanka Ammanamachi, T. Wang, B. Sagot, N. Muennighoff, A. Villanova del Moral, O. Ruwase, R. Bauden, S. Bekeman, A. McMillan-Major, L. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. Ortiz Suarez, V. Sanh, H. Laurencion, Y. Jenrite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. Fikri Aji, A. Alfassy, A. Rogers, A. Kreisberg, N. Xuzav, C. Xu, C. Mou, C. Emzeue, C. Klamm, C. Leong, D. van Strien, D. Hedouva Adelman, and et al. (2022)BLOOM: a 176b-parameter open-access multilingual language model. CoRRabs/2211.05100. External Links: Link Cited by: SS1.
* T. Scalom, P. Dray, S. Lamprier, B. Piowowski, J. Staiano, A. Wang, and P. Gallinari (2021)Questeval: summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6594-6604. External Links: Link Cited by: SS1.
* T. Sellam, D. Das, and A. P. Parikh (2020)BLEURT: learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7881-7892. External Links: Link Cited by: SS1.
* T. Sequoia (2022)Generative ai: a creative new world. Note: [https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/](https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/) Cited by: SS1.
* B. Thompson and M. Post (2020)Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 90-121. External Links: Link Cited by: SS1.
* A. Wang, K. Cho, and M. Lewis (2020)Asking and answering questions to evaluate the factual consistency of summaries. CoRRabs/2004.04228. External Links: Link Cited by: SS1.
* A. Wang, K. Cho, and M. Lewis (2020)Making and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5008-5020. External Links: Link Cited by: SS1.
* P. Wang, L. Li, L. Chen, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui (2023)Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. Cited by: SS1.
* Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. Schwan, D. Kanahanexaran, A. Naik, D. Stap, et al. (2022)Super-natural instructions: generalization via declarative instructions on 1600+ nlp tasks. URL https://arxiv. org/abs/2204.07705. Cited by: SS1.
* J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Quoc Le, and D. Zhou (2022)Chain of thought prompting elicits reasoning in large language models. CoRRabs/2201.11903. External Links: Link Cited by: SS1.
* Tsung-Hsien Wen, Milica Gasic, N. Mrksic, Peihao Su, D. Vandyke, and S. J. Young (2015)Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1711-1721. External Links: Link Cited by: SS1.
* W. Xu, X. Qian, M. Wang, L. Li, and W. Yang Wang (2022)SecScore2: retrieval augmented pretraining for text generation evaluation. CoRRabs/2212.09305. External Links: Link Cited by: SS1.
* W. Xu, Y. Yuan, Y. Lu, M. Savon, L. Li, and W. Yang Wang (2022)Not all errors are equal: learning text generation metrics using stratified error synthesis. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6559-6574. External Links: Link Cited by: SS1.
* Z. Ye, L. Lu, L. Huang, L. Lin, and X. Liang (2021)Towards quantifiable dialogue coherence evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational 

Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 2718-2729. Association for Computational Linguistics.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. _Advances in Neural Information Processing Systems_, 34:27263-27277.
* Zar (2005) Jerrold H Zar. 2005. Spearman rank correlation. _Encyclopedia of biostatistics_, 7.
* Zhang et al. (2021) Chen Zhang, Yiming Chen, Luis Fernando D'Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021. Dynaeval: Unifying turn and dialogue level evaluation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 5676-5689. Association for Computational Linguistics.
* Zhang et al. (2022a) Chen Zhang, Luis Fernando D'Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. 2022a. Finedeval: Fine-grained automatic dialogue-level evaluation. _CoRR_, abs/2210.13832.
* Zhang et al. (2022b) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022b. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 563-578. Association for Computational Linguistics.
* Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. _CoRR_, abs/2210.07197.

## Appendix A Machine Translation Scoring Dimension

For aspects ACC, FLU, and MMM from the machine translation task, we also choose _ref-\(>\)hypo_. Because the source text of the machine translation is a different language from the translated text (hypo). In this work, we mainly consider the evaluation of the English text. In the future, we can consider designing a scoring function based on BLOOM Scao et al. (2022) that can evaluate texts in a cross-lingual setting.

The average sample-level Spearman (\(\rho\)) scores of GPT3-based, GPT2-based, OPT-based, and FT5-based models on the MQM-2020 machine translation dataset are shown in Tab. 6, where values with \({\dagger}\) denote that the evaluator equipped with IST (or lDM) significantly outperforms the VAL setting, and \({\ddagger}\) indicate that the evaluator equipped with lDM (the combination of IST and DM) significantly outperforms the IST setting. The Spearman correlations for the GPT3-based variants are shown in Fig. 7. For the full evaluation results of \(28\) models (including \(9\) baseline scoring models, such as ROUGE-1) can be found in Tab. 14. Following Thompson and Post (2020) and Yuan et al. (2021), we treat the evaluation of machine translation as the paraphrasing task. The main observations are listed as follows:

(1) **The introduction of instruction (IST) significantly improve the performance in three different aspects of ACC, FLU, and MQM.** In Tab. 6, the average performance of \(19\) GPTScore based evaluators with instruction (IST) significantly outperforms vanilla (VAL). (2) **The combination of instruction and demonstration (IDM) brings gains for the evaluator with different model structures.** In Tab. 6, the performance of GPT3, GPT2, OPT, and FT5 improves a lot when instruction and demonstration (lDM) are introduced. (3) **The evaluator built based on GPT3-c01 achieves comparable performance with GPT3-d01 and GPT3-d03.** This can be found in Fig. 7. Since the GPT3-d01 and GPT3-d03 are most expensive variant of GPT3, the cheaper and comparative GPT3-c01 is a good choice for machine translation task.

 

generation task), there are \(J\) system outputs \({\mathbf{h}_{i,j}}\), where \(j\in[1,2,\cdots,J]\). \(f_{\text{auto}}\) is an automatic scoring function (e.g., ROUGE Lin (2004)), and \(f_{\text{human}}\) is the gold human scoring function. For a given evaluation aspect \(a\), the meta-evaluation metric \(F\) can be formulated as follows.

Sample-leveldefines that a correlation value is calculated for each sample separately based on outputs of multiple systems, then averaged across all samples.

\[F^{\text{sample}}_{f_{\text{auto}},f_{\text{human}}}=\frac{1}{n} \sum_{i=1}^{n}\Big{(}g\big{(}\left[f_{\text{auto}}({\mathbf{h}_{i,1}}),\cdots,f_{ \text{auto}}({\mathbf{h}_{i,J}})\right]\] \[\qquad\qquad\quad\left[f_{\text{human}}({\mathbf{h}_{i,1}}),\cdots,f_{ \text{human}}({\mathbf{h}_{i,J}})\right]\big{)}\Big{)}\]

where \(g\) can be instantiated as Spearman or Pearson correlation.

Dataset-levelindicates that the correlation value is calculated on system outputs of all \(n\) samples.

In this work, we select the evaluation strategy for a specific task based on previous works Yuan et al. (2021); Zhang et al. (2022). We use the sample-level evaluation strategy for text summarization, data-to-text, and machine translation tasks. For the dialogue response generation task, the dataset-level evaluation strategy is utilized.

## Appendix C Metric Comparison

Tab. 7 summarize several popular generated text evaluation methods.

## Appendix D Tasks, Datasets, and Aspects

To achieve a more comprehensive evaluation, in this paper, we cover a broad range of natural language generation tasks: _Dialogue Response Generation_, _Text Summarization_, _Data-to-Text_, and _Machine Translation_, which involves \(9\) datasets and \(22\) evaluation aspects in total. Tab. 8 summarizes the tasks, datasets, and evaluation aspects considered by each dataset. The definition of different aspects can be found in Tab. 1.

Dialogue Response Generationaims to automatically generate an engaging and informative response based on the dialogue history. (1) FED Mehri and Eskenazi (2020) collects 124 conversations, including both human-machine Meena Adiwardana et al. (2020), Mitsuku6 and human-human dialogues, and manually annotated 9 and 11 evaluation aspects at the turn- and dialogue-level, respectively.

Footnote 6: [https://medium.com/pandorabots-blog/mitskuk-wins-loebner-prize-2018-3e8d98c5fza7](https://medium.com/pandorabots-blog/mitskuk-wins-loebner-prize-2018-3e8d98c5fza7)

Text Summarizationis a task of automatically generating an informative and fluent summary for a given long text. Here, we consider the following four datasets covering 6 evaluation aspects: _semantic coverage, informativeness, relevance, fluency, coherence_, and _factuality_. (1) SummEvalBhandari et al. (2020) collects human judgments on 16 model-generated summaries on the CNN/Daily Mail dataset, covering aspects of coherence, consistency, fluency, and relevance. (2) REALSummBhandari et al. (2020) evaluates the reliability of automatic metrics by measuring the pyramid recall of text generated by \(25\) systems. (3) NEWROOM Grusky et al. (2018) covers news, sports, entertainment, finance, and other topics and evaluates the quality of summaries generated by 7

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{ACC} & \multicolumn{2}{c}{FLU} & \multicolumn{2}{c}{MQM} \\ \cline{2-5}  & VALIST & IDM & VALIST & IDM & VALIST & IDM \\ \hline GPTS & 27.2 & 27.1 & 29.7\({}^{+1.1}\) **11.3** & 10.4 & **16.4\({}^{+1.4}\)** 30.3 & 31.2\({}^{\dagger}\) 32.3\({}^{\dagger}\) \\ GPT & 25.8 & 27.0\({}^{\circ}\) **30.3\({}^{\dagger}\)** 1.9 & 8.0 & 18.1\({}^{\circ}\) 15.8\({}^{\dagger}\) 30.1 & 30.3\({}^{\dagger}\) 33.5\({}^{\dagger}\) 1.2\({}^{\dagger}\) \\ OPT & **28.7** & **29.4\({}^{\dagger}\)** 30.3\({}^{\dagger}\) 1.0 & 10.2 & **12.1\({}^{\ddagger}\)** 16.3\({}^{\dagger}\)** 32.5 & **34.6\({}^{\dagger}\)** 35.1\({}^{\dagger}\)** 1.1\({}^{\dagger}\) \\ FT5 & 27.7 & 27.8\({}^{\dagger}\) 28.3\({}^{\dagger}\) 1.9 & 6.1 & 11.0\({}^{\dagger}\) 15.4\({}^{\dagger}\) 31.0 & 32.3\({}^{\dagger}\) 32.3\({}^{\dagger}\) \\ \hline \hline Avg. & 27.4 & 27.8 & 29.7\({}^{\ddagger}\) 10.2 & 11.4\({}^{\dagger}\) 16.0\({}^{\ddagger}\) 31.0 & 32.1\({}^{\dagger}\) 33.4\({}^{\ddagger}\) \\ \hline \end{tabular}
\end{table}
Table 6: The average Spearman correlation of the GPT3-based, GPTT-based, OTP-based, and FT5-based models in machine translation task of MQM-2020 dataset.

Figure 7: Experimental results for GPT3-based variants in the machine translation task. Here, blue, orange, green, pink, and cyan dot denote that GPTScore is built based on GPT3-a01 (), GPT3-b01 (), GPT3-c01 (), GPT3-d01 (), and GPT3-d03 (), respectively. The red lines () denote the average performance of GPT3-based variants.

 

systems, including informativeness, relevance, fluency, and coherence. (4) QAGS_XSUMWang et al. (2020b) is another dataset focusing on the factuality aspect. It has 239 samples from XSUM and their summaries are generated by a fine-tuned BART model.

Data-to-Textaims to automatically generate a fluent and factual description for a given table. (1) BadELMairesse et al. (2010) contains \(202\) samples about restaurants in Cambridge. (2) SFRESWen et al. (2015) contains \(581\) samples about restaurants in San Francisco. These two datasets consider three evaluation aspects: _informativeness_, _naturalness_ (relevance), and _quality_ (fluency).

Machine Translationaims to translate a sentence from one language to another. We consider a sub-datasets of Multidimensional Quality Metrics (MQM)Freitag et al. (2021), namely, MQM-2020 (Chinese->English). Due to limited annotations, here, we only consider three evaluation aspects: _accuracy_, _fluency_, and _MQM_ with diverse scores.

## Appendix E Ablation Study

### Effectiveness of Demonstration

The in-context learning helps a lot to achieve a good performance. However, how does the number of samples in the demonstration impact the performance? We conduct a case study on the five GPT3-based models explored in this work. The experimental results are shown in Fig. 5, and the specific performance values can be seen in Tab. 9.

### Partial Order of Evaluation Aspect

We have investigated the combination of different evaluation aspects to achieve further performance gains in SS 6.2. Tab. 10 summarizes the aspect definition and Spearman correlation changes for INT, with the introduction of other aspects.

## Appendix F Prompt Design

In this work, we have studied four popular text generation tasks: text summarization, machine translation, data-to-text, and dialogue response generation. The instructions for these tasks on different evaluation aspects are summarized in Tab. 11 and Tab. 12. Here, we convert the dialogue response generation task as a boolean question-answering task and in

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Metrics**} & \multirow{2}{*}{**Custom**} & \multicolumn{2}{c}{**Function (\(f\))**} & \multicolumn{2}{c}{**Additional text (\(\mathcal{S}\))**} \\ \cline{3-7}  & & **Representation** & **Formulation** & **Source** & **Reference** \\ \hline ROUGELin (2004) & ✗ Token & Matching & No & Required & ✓ & SUM \\ BLEU Papineni et al. (2002) & ✗ Token & Matching & No & Required & ✓ & MT \\ CHRF Popovic (2015) & ✗ Character & Matching & No & Required & ✓ & MT \\ BERTScore Zhang et al. (2020) & ✗ BERT & Matching & No & Required & ✓ & UML(2) \\ MOverScore Zhao et al. (2019) & ✗ BERT & Matching & No & Required & ✓ & MULT(4) \\ BLEURT Sellam et al. (2020) & ✗ BERT & Regression & No & Required & ✓ & MT \\ \hline PRISM Thompson and Post (2020) & ✗ Embedding & Paraphrase & Optional & Optional & Optimal & ✓ & MT \\ UNIEVA Zhong et al. (2022) & ✗ T5 & Boolean QA & Optional & Optional & ✗ & UML(2) \\ COMET Rei et al. (2020) & ✗ BERT & Regress, Rank & Optional & Optional & ✗ & MT \\ BARTScore Yuan et al. (2021) & ✗ BART & Generation & Optional & Optional & ✓ & MUL(3) \\ FED Mehri and Eskénazi (2020a) & ✗ DialoGPT & Generation & Required & Optional & ✓ & Dialogue \\ HolisticEval Pang et al. (2020) & ✗ GPT2 & Generation & Optional & Optional & ✓ & Dialogue \\ \hline GPTscore & ✓ & GPT3/OPT & Any & Optional & Optional & ✓ & MUL(5) \\ \hline \hline \end{tabular}
\end{table}
Table 7: A comprehensive comparison of existing research on automated evaluation of generated texts. MUL(k) denotes multiple (k) applications explored. _Custom Aspects_.

\begin{table}
\begin{tabular}{p{56.9pt} p{34.1pt} p{34.1pt}} \hline \hline
**Tasks** & **Dataset** & **Aspect** \\ \hline \multirow{4}{*}{Diag} & FAD-Diag & COH, DIV, FLE, UND,INQ \\  & FCJ-Turn & CON, INF, LIK, DEP, ERR \\ \hline \multirow{4}{*}{Summ} & SummEval & ENT, ENG, SPE, REL, COR, SEM, UND, FLU \\  & Newsroom REALSumm & COH, CON, FLU,REL, INF, COH \\  & Q-XSUM & FIXAC \\ \hline \multirow{2}{*}{D2T} & BagEL & FLU, REL, INF \\  & SFRES & FLU, REL, INF \\ \hline MT & MQM-2020 & FLU, COH, INF \\ \hline \hline \end{tabular}
\end{table}
Table 8: An overview of tasks, datasets, and evaluation aspects. _Summ._ denote the text summarization task, _D2T_ denotes the Data-to-Text task, _MT_ denotes the machine translation. Tab. 1 summarized the definitions of the aspects explored in this work.

 

corporate the aspect definition into the question of the boolean question-answering task.

## Appendix G Experiment Results

This section lists the full experimental results for the explored text generation tasks. The models considered here include the 9 baseline models: ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, MoverScore, PRISM, BARTSCORE, BARTSCORE+CNN, and BARTSCORE+CNN+Para, and \(19\) GPTScore models built based on the GPT3-based, GPT2-based, OPT-based, and Flan-T5-based pre-trained models.

Tab. 13 lists the results of the text summarization datasets. Tab. 14 lists the results of the machine translation datasets. Tab. 15 shows the results of the data-to-text task on the BAGEL dataset. Tab. 16 shows the results of the data-to-text task on the 



\begin{table}
\begin{tabular}{l l p{142.3pt} p{142.3pt}} \hline
**X** & **Aspect** & **Aspect Definition** & **Sear** \\ \hline
1 & Interesting (INT) & Is this response interesting to the conversation? & 36.9 \\
2 & Engaging (ENG) & Is this an interesting response that is engaging? & 40.7 \\
3 & Specific (SPE) & Is this an interesting response that is specific and engaging? & 48.6 \\
4 & Correct (COR) & Is this an interesting response that is engaging, specific, and correct? & 50.0 \\
5 & Relevant (REL) & Is this an interesting response that is specific, engaging, relevant, and correct? & **51.3** \\
6 & Understandable (UND) & Is this an interesting response that is specific, engaging, relevant, correct, and understandable? & 50.9 \\
7 & Semantically appropriate (SEM) & Is this an interesting response that is specific, engaging, relevant, correct, understandable, and semantically appropriate? & 51.4 \\
8 & Fluent (FLU) & Is this an interesting response that is specific, engaging, relevant, correct, understandable, semantically appropriate, and fluent? & 50.3 \\ \end{tabular}
\end{table}
Table 10: The aspect definition and Spearman correlation of INT. \(X\) denotes the number of aspects combined with the INT. The scoring model is GPT3-c01.

\begin{table}
\begin{tabular}{l l} \hline
**Aspect** & **Function** & **Instruction** \\ \hline \multicolumn{3}{l}{_Text Summarization_} \\ \hline FAC & src–hyppo & Generate a summary with consistent facts for the following text: {src}@nTh;dr[hypo] \\  & ref!e\textless{}hyppo & Rewrite the following text with consistent facts. {ref/hypo} In other words, {hypo/ref} \\ COV & src–hyppo & Generate a summary with as much semantic coverage as possible for the following text: {src}@nTh;dr[hypo] \\  & ref!e\textless{}hyppo & Rewrite the following text with the same semantics. {ref/hypo} In other words, {hypo/ref} \\ CON & src–hyppo & Generate factually consistent summary for the following text: {src}@nTh;dr[hypo] \\  & ref!e\textless{}hyppo & Rewrite the following text with consistent facts. {ref/hypo} In other words, {hypo/ref} \\ INF & src–hyppo & Generate an informative summary that captures the key points of the following text: {src}@nTh;dr[hypo] \\  & ref!e\textgreater{}hyppo & Rewrite the following text with its core information. {ref/hypo} In other words, {hypo/ref} \\ COH & src–hyppo & Generate a coherent summary for the following text: {src}@nTh;dr[hypo] \\  & ref!e\textless{}hyppo & Rewrite the following text into a coherent text. {ref/hypo} In other words, {hypo/ref} \\ REL & src–hyppo & Generate a relevant summary with consistent details for the following text: {src}@nTh;dr[hypo] \\  & ref!e\textless{}hyppo & Rewrite the following text with consistent details. {ref/hypo} In other words, {hypo/ref} \\ FLU & src–hyppo & Generate a fluent and grammatical summary for the following text: {src}@nTh;dr[hypo] \\  & ref!e\textless{}hyppo & Rewrite the following text into a fluent and grammatical text. {ref/hypo} In other words, {hypo/ref} \\ \hline \multicolumn{3}{l}{_Machine Translation_} \\ \hline Acc & ref!e\textless{}hyppo & Rewrite the following text with its core information and consistent facts:{ref/hypo} In other words, {hypo/ref} \\ FLU & ref!e\textless{}hyppo & Rewrite the following text to make it more grammatical and well-written:{ref/hypo} In other words, {hypo/ref} \\ MQM & ref!e\textless{}hyppo & Rewrite the following text into high-quality text with its core information:{ref/hypo} In other words, {hypo/ref} \\ \hline \multicolumn{3}{l}{_Data to Text_} \\ \hline INF & ref!e\textless{}hyppo & Convert the following text to another expression that preserves key information:uTh;hepo] In other words, {hypo/ref} \\ NAT & ref!e\textless{}hyppo & Convert the following text into another expression that is human-like and natural:uTh;deh;hepo] In other words, {hypo/pref} \\ FLU & ref!e\textless{}hyppo & Convert the following text into another expression that preserves key information and is human-like and natural:uTh;deh in other words, {hypo/ref} \\ \hline \end{tabular}
\end{table}
Table 11: Instruction design on different aspects for text summarization, machine translation, and data-to-text tasks. _src_, _hypo_, and _ref_ denote the _source text_, _hypothesis text_, and _reference text_, respectively. _a_->b_ (_a_<-_b_) denotes to evaluate the quality of \(b\) (_a_) text based on the given \(a\) (_b_) text.





\begin{table}
\begin{tabular}{l l} \hline \hline
**Aspect** & **Instruction** \\ \hline _FED Turn-Level_ \\ \hline INT & Answer the question based on the conversation between a human and AI.unQuestion: Are the responses of AI interesting? (a) Yes. (b) No.UnConversation: [History]uAnswer: Yes. \\ \cline{2-3}  & Answer the question based on the conversation between a human and AI.unQuestion: Are the responses of AI engaging? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3}  & Answer the question based on the conversation between a human and AI.unQuestion: Are the responses of AI understandable? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} REL & Answer the question based on the conversation between a human and AI.unQuestion: Are the responses of AI relevant to the conversation? (a) Yes. (b) No.backslashConsorversation: [History]uAnswer: Yes. \\ \cline{2-3} SPE & Answer the question based on the conversation between a human and AI.nQuestion: Are the responses of AI generic or specific to the conversation? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} COR & Answer the question based on the conversation between a human and AI.unQuestion: Are the responses of AI correct to conversations? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes.] \\ \cline{2-3} SEM & Answer the question based on the conversation between a human and AI.nQuestion: Are the responses of AI semantically appropriate? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} FLU & Answer the question based on the conversation between a human and AI.nQuestion: Are the responses of AI fluently written? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \hline \hline _FED Dialog-Level_ \\ \hline COH & Answer the question based on the conversation between a human and AI.nQuestion: Is the AI coherent and maintains a good conversation flow throughout the conversation? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} DIV & Answer the question based on the conversation between a human and AI.unQuestion: Is there diversity in the AI responses? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} FLE & Answer the question based on the conversation between a human and AI.nQuestion: Is the AI flexible and adaptable to human and their interests? (a) Yes. (b) No. unConversation: [History]uAnswer: Yes. \\ \cline{2-3} UND & Answer the question based on the conversation between a human and AI.nQuestion: Does the AI seem to understand the human? (a) Yes. (b) No. UnConversation: [History]uAnswer: Yes. \\ \cline{2-3} INQ & Answer the question based on the conversation between a human and AI.nQuestion: Is the AI inquisitive throughout the conversation? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} CON & Answer the question based on the conversation between a human and AI.nQuestion: Are the responses of AI consistent in the information it provides throughout the conversation? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} INF & Answer the question based on the conversation between a human and AI.nQuestion: Are the responses of AI informative throughout the conversation? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} LIK & Answer the question based on the conversation between a human and AI.nQuestion: Does the AI display a likeable personality? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} DEP & Answer the question based on the conversation between a human and AI.nQuestion: Does the AI discuss topics in depth? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \cline{2-3} ERR & Answer the question based on the conversation between a human and AI.nQuestion: Is the AI able to recover from errors that it makes? (a) Yes. (b) No.unConversation: [History]uAnswer: Yes. \\ \hline \hline \end{tabular}
\end{table}
Table 12: Instruction design on various aspects for dialogue response generation task at the turn- and dialogue-level. _History_ indicates the conversation history. We convert the evaluation of the response generation task as a questionnaire task, and the aspect definition is incorporated into the question of the question-answering task.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**NEWSROOM**} & \multicolumn{3}{c}{**QXSUM**} & \multicolumn{2}{c}{RSumm} \\ \cline{2-14}
* & 26.1 &
- & 25.9 &
- & 34.4 &
- & 3.6 &
- & **46.4** &
- \\ ROUGE-2 & 10.9 &
- & 11.7 &
- & 11.2 &
- & 14.4 &
- & 9.9 &
- & 37.3 &
- \\ ROUGE-L & 24.7 &
- & 25.7 &
- & 24.4 &
- & 32.5 &
- & 5.2 &
- & 45.1 &
- \\ BERTScore & 31.7 &
- & 31.7 &
- & 27.2 &
- & 33.7 &
- & -4.6 &
- & 38.4 &
- \\ MoverScore & 17.7 &
- & 14.2 &
- & 16.0 &
- & 18.9 &
- & 5.4 &
- & 34.4 &
- \\ PRISM & 60.7 &
- & 56.5 &
- & 59.2 &
- & 61.9 &
- & 2.5 &
- & 32.3 &
- \\ BARTSCORE & 70.3 &
- & 67.2 &
- & 63.1 &
- & 68.8 &
- & 0.9 &
- & 43.1 &
- \\ +CNN & 68.5 &
- & 64.9 &
- & 60.4 &
- & 66.3 &
- & 18.4 &
- & 42.9 &
- \\ +CNN+Para & 69.0 &
- & 65.5 &
- & 62.5 &
- & 67.3 &
- & 6.4 &
- & 40.9 &
- \\ \hline
**GPT3** & & & & & & & & & & & & \\ \hline GPT3-a01 & 71.6 & 71.9\({}^{\dagger}\) & 69.7 & 70.0\({}^{\dagger}\) & 66.0 & 67.0\({}^{\dagger}\) & 69.6 & 69.2 & 10.3 & 9.2 & 29.5 & 29.8\({}^{\dagger}\) \\ GPT3-b01 & 73.6 & 72.9 & 70.2 & 70.3 & 66.8 & 68.3\({}^{\dagger}\) & 71.5 & 71.2 & 8.5 & 14.2 & 35.0 & 35.2\({}^{\dagger}\) \\ GPT3-c01 & **73.8** & 72.8 & **70.5** & **70.9\({}^{\dagger}\)** & 65.9 & 68.6\({}^{\dagger}\) & 71.0 & 71.1 & 15.2 & 22.1\({}^{\dagger}\) & **36.1** & **45.1\({}^{\dagger}\)** \\ GPT3-d01 & 72.6 & **73.4\({}^{\dagger}\)** & 68.5 & 70.0\({}^{\dagger}\) & 65.9 & 66.9\({}^{\dagger}\) & 71.1 & 72.1\({}^{\dagger}\) & **24.0** & **22.7** & 36.0 & 33.9 \\ GPT3-d03 & 73.8 & 73.1 & 70.4 & 70.0 & **67.4** & **68.9\({}^{\dagger}\)** & **74.1** & **73.3** & 21.7 & 22.0\({}^{\dagger}\) & 35.2 & 38.0\({}^{\dagger}\) \\ \hline
**Avg** & 73.1 & 72.8 & 69.9 & 70.2\({}^{\dagger}\) & 66.4 & 67.9\({}^{\dagger}\) & 71.4 & 71.4 & 15.9 & 18.0\({}^{\dagger}\) & 34.4 & 36.4\({}^{\dagger}\) \\ \hline \hline
**GPT2** & & & & & & & & & & & & \\ \hline GPT2-M & 68.9 & 71.7\({}^{\dagger}\) & 66.4 & 68.0\({}^{\dagger}\) & 61.1 & 62.3\({}^{\dagger}\) & 67.0 & 66.8 & 18.1 & 18.7\({}^{\dagger}\) & 41.8 & 43.3\({}^{\dagger}\) \\ GPT2-L & 70.5 & **72.3\({}^{\dagger}\)** & 66.6 & 68.3\({}^{\dagger}\) & 60.2 & 61.4\({}^{\dagger}\) & 66.8 & 67.8\({}^{\dagger}\) & 19.2 & 19.6\({}^{\dagger}\) & 39.6 & 41.3\({}^{\dagger}\) \\ GPT2-XL & 71.0 & 70.5 & 66.6 & 66.6 & 61.4 & 60.7 & 67.2 & 66.9 & 21.2 & 21.2 & 40.4 & 41.0\({}^{\dagger}\) \\ GPTJ-6B & **71.8** & 71.4 & **69.8** & **69.5** & **65.5** & **65.5** & **69.4** & **69.3** & **21.6** & **22.0\({}^{\dagger}\) & **42.8** & **43.7\({}^{\dagger}\)** \\ \hline
**Avg** & 70.5 & 71.5\({}^{\dagger}\) & 67.4 & 68.1\({}^{\dagger}\) & 62.0 & 62.5\({}^{\dagger}\) & 67.6 & 67.7 & 20.0 & 20.4\({}^{\dagger}\) & 39.8 & 41.1\({}^{\dagger}\) \\ \hline \hline
**OPT** & & & & & & & & & & & \\ \hline OPT-350M & 70.6 & 71.5\({}^{\dagger}\) & 69.2 & 69.9\({}^{\dagger}\) & 67.3 & 68.1\({}^{\dagger}\) & 70.8 & 71.6\({}^{\dagger}\) & 13.5 & 13.3 & 40.2 & 42.3\({}^{\dagger}\) \\ OPT-1.3B & **73.2** & **73.6\({}^{\dagger}\)** & **70.9** & **71.3\({}^{\dagger}\)** & 67.2 & **67.8\({}^{\dagger}\)** & **72.5** & **72.4** & 21.1 & 19.9 & 42.0 & 39.7 \\ OPT-6.7B & 71.9 & 71.9 & 69.0 & 69.0 & **67.7** & 67.1 & 71.7 & 71.3 & 21.2 & 19.9 & 38.0 & 41.9\({}^{\dagger}\) \\ OPT-13B & 71.9 & 71.9 & 68.9 & 69.6\({}^{\dagger}\) & 65.4 & 66.0\({}^{\dagger}\) & 71.2 & 71.5\({}^{\dagger}\) & 23.1 & 22.1 & 37.6 & 41.0\({}^{\dagger}\) \\ OPT-66B & 72.8 & 72.8 & 70.0 & 69.5 & 66.0 & 65.9 & 71.9 & 71.9 & **24.0** & **23.1** & **40.3** & 41.3\({}^{\dagger}\) \\ \hline \hline
**Avg** & 72.1 & 72.3\({}^{\dagger}\) & 69.6 & 69.9\({}^{\dagger}\) & 66.7 & 67.0\({}^{\dagger}\) & 71.6 & 71.8\({}^{\dagger}\) & 20.6 & 19.6 & 39.6 & 41.2\({}^{\dagger}\) \\ \hline \hline
**Flan-T5** & & & & & & & & & & \\ \hline FT5-S & 68.3 & 69.2\({}^{\dagger}\) & 64.6 & 64.1 & 59.8 & 60.4\({}^{\dagger}\) & 64.6 & 65.5\({}^{\dagger}\) & 14.4 &



\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**ACC**} & \multicolumn{3}{c}{**FLU**} & \multicolumn{3}{c}{**MQM**} \\ \cline{2-10}

- & 1.7 &
- & 17.5 &
- \\ ROUGE-2 & 15.0 &
- & 5.8 &
- & 15.4 &
- \\ ROUGE-L & 16.6 &
- & 8.7 &
- & 15.7 &
- \\ BERTScore & 26.1 &
- & 8.2 &
- & 23.6 &
- \\ MoverScore & 18.2 &
- & 1.2 &
- & 17.2 &
- \\ PRISM & 25.9 &
- & 9.1 &
- & 27.4 &
- \\ BARTSCORE & 26.1 &
- & 8.2 &
- & 23.6 &
- \\ +CNN & 26.2 &
- & 8.1 &
- & 28.7 &
- \\ +CNN+Para & 31.0 &
- & 10.8 &
- & 29.9 &
- \\ \hline
**GPT3** & & & & & & & & & \\ \hline GPT3-a01 & 24.9 & 23.7 & 27.9\({}^{\dagger,\ddagger}\) & 5.9 & 6.3\({}^{\dagger}\) & 11.6\({}^{\dagger,\ddagger}\) & 27.0 & 24.1 & 24.4\({}^{\ddagger}\) \\ GPT3-b01 & 25.9 & 25.0 & 29.8\({}^{\dagger,\ddagger}\) & 10.7 & 10.8 & 14.0\({}^{\dagger,\ddagger}\) & 29.4 & 29.6 & 31.2\({}^{\dagger,\ddagger}\) \\ GPT3-c01 & **29.4** & **30.3\({}^{\dagger}\)** & 30.2\({}^{\dagger}\) & 10.7 & 9.3 & 17.9\({}^{\dagger,\ddagger}\) & **33.3** & 34.8\({}^{\dagger}\) & 34.5\({}^{\dagger}\) \\ GPT3-d01 & 28.6 & 26.5 & **31.2\({}^{\dagger,\ddagger}\)** & 11.3 & 8.6 & 17.5\({}^{\dagger,\ddagger}\) & 32.0 & 32.5\({}^{\dagger}\) & **38.3\({}^{\dagger,\ddagger}\)** \\ GPT3-d03 & 27.2 & 30.1\({}^{\dagger}\) & 29.5\({}^{\dagger}\) & **18.0** & **17.1** & **21.3\({}^{\dagger,\ddagger}\)** & 29.9 & **34.8\({}^{\dagger}\)** & 32.8\({}^{\dagger}\) \\ \hline
**Avg.** & **27.2** & **27.1** & **29.7\({}^{\dagger,\ddagger}\)** & 11.3 & 10.4 & 16.4\({}^{\dagger,\ddagger}\)** & **30.3** & 31.2\({}^{\dagger}\) & 32.3\({}^{\dagger,\ddagger}\)** \\ \hline
**GPI2** & & & & & & & & & \\ \hline GPT2-M & 25.7 & 24.6 & 29.6\({}^{\dagger,\ddagger}\) & 8.6 & 9.4\({}^{\dagger}\) & 15.1\({}^{\dagger,\ddagger}\) & 32.1 & 29.4 & 34.1\({}^{\dagger,\ddagger}\) \\ GPT2-L & 27.2 & 28.5\({}^{\dagger}\) & 32.2\({}^{\dagger,\ddagger}\) & 11.1 & 10.4 & 14.9\({}^{\dagger,\ddagger}\) & 31.2 & 30.9 & 33.9\({}^{\dagger,\ddagger}\) \\ GPT2-XL & 24.2 & 27.6\({}^{\dagger}\) & 29.7\({}^{\dagger,\ddagger}\) & 9.4 & 12.0\({}^{\dagger}\) & 17.4\({}^{\dagger,\ddagger}\) & 28.6 & 32.2\({}^{\dagger}\) & 35.8\({}^{\dagger,\ddagger}\) \\ GPTJ-6B & 26.2 & 27.2\({}^{\dagger}\) & 29.5\({}^{\dagger,\ddagger}\) & 9.9 & 11.2\({}^{\dagger}\) & 15.9\({}^{\dagger,\ddagger}\) & 28.5 & 28.8\({}^{\dagger}\) & 30.3\({}^{\dagger,\ddagger}\) \\ \hline
**Avg.** & 25.8 & 27.0\({}^{\dagger}\) & 30.3\({}^{\dagger,\ddagger}\) & 9.8 & 10.8\({}^{\dagger}\) & 15.8\({}^{\dagger,\ddagger}\) & 30.1 & 30.3\({}^{\dagger}\) & 33.5\({}^{\dagger,\ddagger}\) \\ \hline
**OPT** & & & & & & & & \\ \hline OPT-350M & 29.3 & 28.1 & 28.6\({}^{\ddagger}\) & 11.7 & 11.9 & 15.7\({}^{\dagger,\ddagger}\) & 31.5 & 32.5\({}^{\dagger}\) & 31.8 \\ OPT-1.3B & 27.9 & 27.7 & 28.0\({}^{\ddagger}\) & 8.8 & 13.3\({}^{\dagger}\) & 15.9\({}^{\dagger,\ddagger}\) & 32.6 & 33.6\({}^{\dagger}\) & 32.9\({}^{\dagger}\) \\ OPT-6.7B & 29.6 & 30.7\({}^{\dagger}\) & 30.6\({}^{\dagger}\) & 10.7 & 12.2\({}^{\dagger}\) & 15.0\({}^{\dagger,\ddagger}\) & 34.2 & 36.4\({}^{\dagger}\) & 36.9\({}^{\dagger,\ddagger}\) \\ OPT-13B & 27.5 & 29.5\({}^{\dagger}\) & 30.8\({}^{\dagger,\ddagger}\) & 9.6 & 11.7\({}^{\dagger}\) & 17.9\({}^{\dagger,\ddagger}\) & 31.9 & 35.5\({}^{\dagger}\) & 37.5\({}^{\dagger,\ddagger}\) \\ OPT-66B & 29.5 & 31.0\({}^{\dagger}\) & 33.4\({}^{\dagger,\ddagger}\) & 9.1 & 12.1\({}^{\dagger}\) & 16.8\({}^{\dagger,\ddagger}\) & 32.1 & 35.3\({}^{\dagger}\) & 36.4\({}^{\dagger,\ddagger}\) \\ \hline \hline
**Avg.** & 28.7 & 29.4\({}^{\dagger}\) & 30.3\({}^{\dagger,\ddagger}\) & 10.0 & 12.2\({}^{\dagger}\) & 16.3\({}^{\dagger,\ddagger}\) & 32.5 & 34.6\({}^{\ddagger}\) & 35.1\({}^{\dagger,\ddagger}\) \\ \hline
**Flan-T5** & & & & & & & & \\ \hline FT5-S & 27.6 & 28.7\({}^{\dagger}\) & 27.0 & 12.6 & 9.4 & 15.0\({}^{\dagger,\ddagger}\) & 33.5 & 33.3 & 31.3 \\ FT5-B & 25.5 & 25.4 & 27.4\({}^{\dagger,\ddagger}\) & 10.4 & 10.2 & 15.9\({}^{\dagger,\ddagger}\) & 29.8 & 29.6 & 30.0\({}^{\ddagger}\) \\ FT5-L & 28.5 & 28.5 & 28.8\({}^{\dagger,\ddagger}\) & 7.9 & 13.0\({}^{\dagger}\) & 15.6\({}^{\dagger,\ddagger}\) & 30.7 & 31.6\({}^{\dagger}\) & 32.1\({}^{\dagger,\ddagger}\) \\ FTS-XL & 28.1 & 27.0 & 28.1\({}^{\ddagger}\) & 9.4 & 10.2\({}^{\dagger}\) & 14.0\({}^{\dagger,\ddagger}\) & 30.4 & 33.5\({}^{\dagger}\) & 34



\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**INF**} & \multicolumn{4}{c}{**NAT**} & \multicolumn{2}{c}{**FLU**} \\ \cline{2-10}

- & 5.0 &
- & 8.3 &
- \\ ROUGE-2 & 24.0 &
- & 15.2 &
- & 16.0 &
- \\ ROUGE-L & 26.3 &
- & 10.5 &
- & 11.0 &
- \\ BERTScore & 37.2 &
- & 16.0 &
- & 18.7 &
- \\ MoverScore & 30.7 &
- & 20.4 &
- & 14.8 &
- \\ PRISM & 36.8 &
- & 28.7 &
- & 34.4 &
- \\ BARTSCORE & 29.5 &
- & 24.0 &
- & 29.7 &
- \\ +CNN & 37.7 &
- & 30.1 &
- & 34.4 &
- \\ +CNN+Para & 39.2 &
- & 31.0 &
- & 44.9 &
- \\ \hline \multicolumn{10}{l}{**GPT3**} \\ \hline GPT3-a01 & 33.3 & 37.0† [FOOTNOTE:†]Footnote †: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin: margin margin: margin: margin: margin: margin margin: margin: margin: margin margin: margin: margin margin: margin: margin margin: margin: margin margin: margin margin: margin margin: margin margin: margin margin:




\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**INF**} & \multicolumn{3}{c}{**NAT**} & \multicolumn{3}{c}{**FLU**} \\ \cline{2-10}

- & 24.2 &
- & 15.1 &
- \\ ROUGE-2 & 21.9 &
- & 25.9 &
- & 11.4 &
- \\ ROUGE-L & 18.5 &
- & 20.2 &
- & 1.7 &
- \\ BERTScore & 25.8 &
- & 28.0 &
- & 11.8 &
- \\ MoverScore & 17.9 &
- & 24.4 &
- & 5.0 &
- \\ PRISM & 27.4 &
- & 33.1 &
- & 14.2 &
- \\ BARTSCORE & 22.4 &
- & 25.5 &
- & 6.9 &
- \\ +CNN & 24.2 &
- & 30.6 &
- & 17.2 &
- \\ +CNN+Para & 25.0 &
- & 30.2 &
- & 19.5 &
- \\ \hline \multicolumn{10}{l}{**GPT3**} \\ \hline GPT3-a01 & 25.4 & 19.1 & 25.6\({}^{\ddagger}\) & **28.7** & **34.0\({}^{\dagger}\)** & **37.7\({}^{\dagger,\ddagger}\)** & 30.7 & 27.0 & 26.6 \\ GPT3-b01 & **37.5** & 28.4 & 26.5 & 21.5 & 30.6\({}^{\dagger}\) & 26.1\({}^{\dagger}\) & 24.6 & 28.9\({}^{\dagger}\) & 21.1 \\ GPT3-c01 & 29.8 & 21.3 & 33.7\({}^{\dagger,\ddagger}\) & 24.7 & 28.5\({}^{\dagger}\) & 28.6\({}^{\dagger}\) & 31.1 & 27.1 & 27.6\({}^{\ddagger}\) \\ GPT3-d01 & 32.6 & 27.0 & 33.9\({}^{\dagger,\ddagger}\) & 27.3 & 31.7\({}^{\dagger}\) & 21.9 & **35.8** & **39.7\({}^{\dagger}\)** & 27.1 \\ GPT3-d03 & 26.6 & **29.6\({}^{\dagger}\)** & **37.6\({}^{\dagger,\ddagger}\)** & 22.6 & 27.0\({}^{\dagger}\) & 18.2 & 33.9 & 31.9 & **28.2** \\ \hline \hline
**Avg.** & 30.4 & 25.1 & 31.5\({}^{\dagger,\ddagger}\) & **25.0** & 30.4\({}^{\dagger}\) & **26.5\({}^{\dagger}\)** & 31.2 & 30.9 & 26.1 \\ \hline \multicolumn{10}{l}{**GPT2**} \\ \hline GPT2-M & 24.7 & 23.1 & 18.2 & 28.7 & 32.7\({}^{\dagger}\) & 35.2\({}^{\dagger,\ddagger}\) & 18.7 & 34.8\({}^{\dagger}\) & 33.6\({}^{\dagger}\) \\ GPT2-L & 19.6 & 28.1\({}^{\dagger}\) & 20.2\({}^{\dagger}\) & 31.2 & 32.4\({}^{\dagger}\) & 37.8\({}^{\dagger,\ddagger}\) & 18.6 & 33.1\({}^{\dagger}\) & 35.9\({}^{\dagger,\ddagger}\) \\ GPT2-XL & 22.0 & 23.6\({}^{\dagger}\) & 23.8\({}^{\dagger}\) & 29.7 & 29.1 & 38.0\({}^{\dagger,\ddagger}\) & 18.2 & 29.8\({}^{\dagger}\) & 37.1\({}^{\dagger,\ddagger}\) \\ GPTJ-fB & 23.9 & 25.6\({}^{\dagger}\) & 19.6 & 34.3 & 33.3 & 36.8\({}^{\dagger,\ddagger}\) & 24.4 & 34.5\({}^{\dagger}\) & 38.4\({}^{\dagger,\ddagger}\) \\ \hline \hline
**Avg.** & 22.5 & 25.1\({}^{\dagger}\) & 20.5 & 31.0 & 31.9\({}^{\dagger}\) & 37.0\({}^{\dagger,\ddagger}\) & 20.0 & 33.1\({}^{\dagger}\) & 36.2\({}^{\dagger,\ddagger}\) \\ \hline \multicolumn{10}{l}{**OPT**} \\ \hline OPT-350M & 26.1 & 28.7\({}^{\dagger}\) & 25.4 & 27.0 & 29.5\({}^{\dagger}\) & 35.0\({}^{\dagger,\ddagger}\) & 21.7 & 26.6\({}^{\dagger}\) & 27.3\({}^{\dagger,\ddagger}\) \\ OPT-1.3B & 26.1 & 28.3\({}^{\dagger}\) & 23.5 & 26.0 & 30.5\({}^{\dagger}\) & 38.7\({}^{\dagger,\ddagger}\) & 23.0 & 26.9\({}^{\dagger}\) & 29.8\({}^{\dagger,\ddagger}\) \\ OPT-6.7B & 26.2 & 26.0 & 24.2 & 26.7 & 31.0\({}^{\dagger}\) & 36.5\({}^{\dagger,\ddagger}\) & 21.7 & 25.8\({}^{\dagger}\) & 35.9\({}^{\dagger,\ddagger}\) \\ OPT-13B & 27.7 & 26.9 & 26.0 & 24.4 & 30.1\({}^{\dagger}\) & 38.0\({}^{\dagger,\ddagger}\) & 20.2 & 29.6\({}^{\dagger}\) & 34.9\({}^{\dagger,\ddagger}\) \\ OPT-6B & 20.1 & 24.7\({}^{\dagger}\) & 22.4\({}^{\dagger}\) & 26.8 & 29.1\({}^{\dagger}\) & 34.6\({}^{\dagger,\ddagger}\) & 19.8 & 19.1 & 25.3\({}^{\dagger,\ddagger}\) \\ \hline \hline
**Avg.** & **25.2** & **26.9\({}^{\dagger}\)** & 24.3 & 26.2 & 30.0\({}^{\dagger}\) & **36.6\({}^{\dagger,\ddagger}\)** & 21.3 & 25.6\({}^{\dagger}\) & 30.6\({}^{\dagger,\ddagger}\) \\ \hline \multicolumn{10}{l}{**Flan-T5**} \\ \hline FT5-S & 19.7 & 16.9 & 17.0 & 33.6 & 33.1 & 33.0 & 19.4 & 17.2 & 15.9 \\ FT5-B & 24.2 & 23.7 & 20.9 & 31.7 & 32.5\({}^{\dagger}\) & 33.4\({}^{\dagger,\ddagger}\) & 14.2 & 15.5\({}^{\dagger}\) & 16.8\({}^{\dagger,\ddagger}\) \\ FT5-L & 24.9 & 22.3 & 20.6 & 36.2 & 37.1\({}^{\dagger}\) & 38.6\({}^{\dagger,\ddagger}\) & 24.3 & 18.1 & 21.1\({}^{\ddagger}\) \\ FTS-XL & 26.1 & 23.7 & 19.5 & 38.4 & 35.6 & 37.4\({}^{\ddagger}\) & 28.4 & 21.0 & 22.5\({}^{\ddagger}\) \\ FT5-XXL & 24.9 & 22.9 & 20.3 & 31.9 & 34.7\({}^{\dagger}\) & 41.7\({}^{\dagger,\ddagger}\) & 23.8 & 16.9 & 22.2\({}^{\ddagger}\) \\ \hline \hline
**Avg.** & **24.0** & **21.9** & **19.7** & **34.3** & **34.6\({}^{\dagger}\)** & 36.8\({}^{\dagger,\ddagger}\)** & 22.0 & 17.8 & **19.7\({}^{\ddagger}\)** \\ \hline \hline
**Overall