Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Authors: Chenhui Shen
Liying Cheng
Xuan-Phi Nguyen
Yang You
Lidong Bing
Venue: EMNLP 2023 Findings (2023)
Keywords: LLM evaluation
ChatGPT
abstractive summarization evaluation
GPT-4

Abstract: With the recent undeniable advancement in reasoning abilities in large language models (LLMs) like ChatGPT and GPT-4, there is a growing trend for using LLMs on various tasks. One area where LLMs can be employed is as an alternative evaluation metric for complex generative tasks, which generally demands expensive human judges to complement the traditional automatic metrics for various evaluation dimensions such as fluency and consistency. In this work, we conduct extensive analysis to investigate the stability and reliability of LLMs as automatic evaluators for abstractive summarization. We found that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements due to significant limitations. That is, LLM evaluators rate each candidate system inconsistently and are dimension-dependent. They also struggle to compare candidates with close performance and become more unreliable with higher-quality summaries by obtaining a lower correlation with humans. In other words, with better abstractive summarization systems being introduced at a fast pace, LLMs may result in misleading and unreliable evaluations.

Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Chenhui Shen" 1.2 Liying Cheng 1.3 Xuan-Phi Nguyen Yang You2 Lidong Bing
'DAMO Academy, Alibaba Group, Singapore Nattional University of Singapore *Hupan Lab, 310023, Hangzhou, China
{chenhui. liying. cheng, x.nguy,n, 1 jeliibaaa-iin.comm youyêcomp. mus.edu. sg
Abstract
"gold" summaries become less important. Although many neural-based metries such as BERTScore and BARTScore (Zhang et al., 2020b; Yuan et al., 2021), are advocated as more aalman-lligndd, the evaluation criteria are also becoming increasingly complex. As a result, abstractive summarization may not be sufficiently evaluated with automatic metrics (Owezarzak al., 2012; Nenkova, 2006), and often require extensive human evaluations as complements Yang al., 2023; Welbl al., 2021). However, human evaluations often come with hefty costs and slow iteration cycles, while also being difficult to reproduce and standardize due to small sample sizes and potential human biases (Shen et al., 2022b; Liu et al., 2022).
With the recent undeniable advancement in reasoning abilities in large language models (LLMs) like ChatGPT and GPT-4, there is growing trend for using LLMs various tasks. One area where LLMs can be employed is as alternative evaluation metric for complex generative tasks, which generally demands ex- pensive human judges complement the tradi- tional automatic metries for various evaluation dimensions such as fluency and consistency. In this work, we conduct extensive analysis in- vestigate the stability and reliability of LLMs as automatic evaluators for abstractive summariza- tion. We found that while ChatGPT and GPT-4 outperform the commonly used automatic met- rics, they are not ready as human replacements due to significant limitations. That is, LLM evaluators rate each candidate system incon- sistently and are dimension dependddt......n They also struggle compare candidates with close performance and become more unreliable with higher-quality summaries by obtaining a lower correlation with humans. In other words, with better abstractive summarization systems being introduced fast pace, LLMs may result in misleading and unreliable evaluations."
Recent large language models (LLMs) like Chat- GPT and GPT-4 (OpenAI, 2023) have demon- strated outstanding capabilities in language com- prehension and reasoning. This leads growing trend of employing LLMs as evaluators for com- plex language generation tasks by prompting them with carefully and claborately crafted instructions (Chiang and Lee, 2023; Gao et al., 2023; Wang et al., 2023a; Wu et al., 2023; Luo et al., 2023; Liu et al., 2023). Despite the preliminary success sug- gested by such works, it is still inconclusive as what degree of confidence we can trust the evalu- ation results produced by LLMs across different dimensions, despite their supposedly high average correlation with humans. also unclear certain LLM- based metrics are more reliable than others, or if their reliability and fairness vary for different candidate systems.
1 Introduction
The desire for inexpensive and fast automatic met- rics has never stopped growing. In certain tasks like extractive summarization, where full source sentences are selected to appear in the summaries, simple n-gram overlap metrics against the "gold" summaries like ROUGE (Lin, 2004) BLEU (Pa- pineni et al., 2002) may work well because the correct answer space is narrow, However, for more open tasks like abstractive summarization, there are countless equally good summaries and
In this work, we conduct extensive analysis to assess whether LLM evaluators can reliably re- place human judges. Specifically, incorporate two common human evaluation approaches with LLM evaluators, namely Likert-scale scoring (He et al., 2022; Shen et al., 2022b; Zhang al., 2020a) and head-to-head (H2H) comparisons (Shen et al., 2022a; Li et al., 2020; Liu and Lapata, 2019). For
Chenhui is under the Joint PhD Program between A1- ibaba and National University of Singapore.
Correspouding author.
OOur code and data are fully released at https:://g/t //githubb com/DAMO- DDM///AMOOMR MOM0-------- NLP-SG/LLB SG/LLM_summsee summeval.

---

Likert-scale scoring, we explore direct reason-then- score (RTS) generation and multiple-choice ques- tion (MCQ) method. The former instructs the LLM to provide reasoning before giving a score, while the latter simply prompts it to choose a specific score with pre-determined description as the rea- son. For the HleadtooHHead (H2H) comparison, we prompt LLM for a preference over the summaries from two compared candidate systems.
or paraphrasing techniques (Shen et al., 2023; Liu et al., 2022; Xiao et al., 2022: Lewis al., 2020; Zhang et al., 2020a; Ziegler et al., 2019; Bing et al., 2015; Xu and Durrett, 2021). As such, ab- stractive summarization poses significantly more challenges for automatic and human evaluation pipelines (Saha et al., 2022: Pagnoni et al., 2021), because it is increasingly insufficient to use the provided "gold" summary as ground truth.
Our experiments show that LLM evaluators, with RTS and MCQ, outperform existing auto- matic metrics (Lin, 2004; Yuan et al., 2021). How- ever, they are not ready to be reliable alternatives for human evaluation yet. Specifically, (i) LLM evaluators struggle distinguish candidates with close performances ($ 4.2.1). (ii) LLM evalua- tors are candidate-dependent, meaning they do not exhibit highly consistent degrees of human alignment for different candidates (S 4.2.3). Thus, they may unfairly favor or disfavor an evaluated candidate. (iii) LLM evaluators are dimension- dependent, meaning they have varying degrees of evaluation capabilities for different dimensions like coherence and fluency ($ 4.2.3). (iv) Lastly, as the quality of summaries improves with better candi- dates, LLM evaluators become unreliably less cor- related with human judgments, according to our newly proposed meta-correlation metric ($ 4.2.4). While we still call for better automatic metric, in the meantime, we suggest temporary solution in $ for abstractive summarization practitioners to use LLMs more reliably. Specifically, we advocate calculating the correlation between RTS and MCQ as a preliminary indicator of the reliability of the LLM for certain dimensions. If RTS and MCQ do not generally agree with each other, then further human evaluations are required.
Human Evaluation Human evaluation can be conducted with different approaches. Some work (He et al., 2022; Shen et al., 2022b; Zhang et al., 2020a; Cheng et al., 2020; Gao et al., 2019; Liu et al., 2018; Li al., 2017; Kryšcińskii et al., 2018) employ a Likert scale evaluate the summaries on discrete ranges, such as from 1 to 5. Meanwhile, many others suggest comparison approaches by asking human annotators to select the best sum- mary out of 2 or more generated summaries from different systems (Shen et al., 2022a; Li et al., 2020; Liu and Lapata, 2019; Fan et al., 2018; Fabbri al., 2019). Following this, we test LLM-based evalu- ators using both approaches with human- human-friendly instruction prompts.
Automatic Evaluation ROUGE (Lin, 2004) has been a common lexical overlap metric evalu- ate summarization systems. Apparently, ROUGE is not sufficient for abstractive summarization, be- cause the "gold" labels it relies on cannot com- prehensively account for the complexity and vari- ability of this task. In addition, the common us- age of sentence fusion techniques and novel words for abstractive summarization may make ROUGE even less reliable. Zhang et al. (2020b) propose the neural-based BERTScore, which leverages the BERT word embeddings to compute the semantic similarity among tokens. Yuan et al. (2021) 1) later introduce BARTScore, which uses BART (Lewis et al., 2020) to compute the probability of a sum- mary given its input article. Nonetheless, these metrics may not reflect all of the complicated eval- uation dimensions required for abstractive summa- rization mentioned earlier, nor do they have suffi- ciently high correlations with humans.
2 Related Work
Summarization The summarization task in- volves generating a summary that contains concise and important (i.e., salient) contents of the origi- nal input article (Nenkova and McKeown, 2012).
This task has been handled with 2 different ap- proaches: extractive and abstractive. Unlike extrac- tive summarization systems that directly extract salient phrases or sentences from the input article (Ernst al., 2022; Chen al., 2021; Zhou al., 2018; Dong et al., 2018), abstractive summariza- tion systems are expected to generate summarics using their own words and apply sentence fusion
LLM- Evaluation There are many concur- rent works that demonstrate the potential of LLMs to conduct complex human tasks (Chiang and Lee, 2023; Gao al., 2023; Wang et al., 2023a; Wu et al., 2023; Luo et al., 2023; Liu et al., 2023; Cheng et al., 2023). The key advantage of instruction-

---

tuned LLMs, like ChatGPT or GPT-4 (Ouyang al., 2022: OpenAI, 2023), is that we can ex- plicitly describe in natural language what our eval- uation criteria and dimensions are and how to score the summaries, similar to how we would explain such tasks to a human expert. Chiang and Lee (2023) use LLMs for open-ended story evaluations, while Luo et al. (2023) apply ChatGPT specifi- cally for evaluating the consistency of summaries. Wu et al. (2023) formulate LLMs as diverse role- players to evaluate summaries from the perspec- tives of different personas. Wang et al. (2023a) and Liu et al. (2023) also explore the LLM's evalua- tion potential in various dimensions for the natural language generation task. Our work differs from the above works in that besides investigating the LLMs' capability using different approaches actoss various dimensions for abstractive summarization, we further focus on the reliability of LLM across evaluated systems and dimensions.
Score the following Summary given the corresponding Arti- cle with respect to relevance from one five, where in dicates "irrelevance" and five indicates "perfect relevance", Note that relevance measures the Summary's 's selection of important content from the Article, whether the Summary grasps the main message of Article without being over- whelmed by unnecessary or less significant details.
Article: [article]
Summary: (summary]
Provide your reason in one sentence, then give final score:
Table I: Example prompt for the RTS method on the relevance dimension. Texts in (blue) represent the arti- cle and the corresponding summary to be evaluated.
Choose an option from A to E in order to score the follow- ing Summary given the corresponding Article with respect relevance from one five, where one indicates "imel- evance", and five indicates "perfect relevance". Note that relevance measures the Summary's 's selection of important content from the Article, whether the Summary grasps the main message of the Article without being overwhelmed by unnecessary or less significant details.
3 LLM as a Zero-Shot Evaluator
We investigate an LLM's evaluation capabilities in the dimensions of coherence, consistency, fluency, and relevance respectively, as defined by Fabbri et al. (2021) (see Appendix A). Following com- mon human evaluation approaches, we propose two methods for Likert-scale scoring, namely the reason-then-score method and the multiple-choice question method, well as method for head- to-head comparisons. We describe each method in $ 3.1 using the relevance dimension as an example (see more prompts and details in Appendix B). We further experiment with alternative phrasings for different methods in Appendix C.
Article: (article]
Summary: (summary)
A: The Summary is totally irrelevant to the Article. Score:
One.
B: The majority of the Summary is irrelevant to the Article.
Score: Two.
C: Some information the Summary relevant to the
Article whereas some are not. Score: Three
D: The majority of the Summary is relevant to the Article. Score: Four.
E: All information included in the Summary is relevant to the Article. Score: Five.
Your Answer (enter letter from A to E):
Besides exploring different evaluation methods, the stability of LLM- based evaluations across dif- ferent summarization systems is equally important. Ideally, a stable LLM evaluator should perform equally well regardless of the evaluated systems, with a close (if not identical) degree of alignment with human judgments. In $ 3.2, we propose a meta-correlation metric and explain how it can gauge the extent to which LLM evaluators' per- formances depend on the evaluated systems, which indicates how stable and reliable they may be with evaluating any future candidate systems.
Table 2: Example prompt for the MCQ method on the relevance dimension. Texts (blue) represent the arti- cle and the corresponding summary to be evaluated.
Wei al., 2022), an intuitive method is to ask the LLM to evaluate a specific dimension by first generating the reasoning and then a corresponding score. Since the SummEval dataset (Fabbri et al., 2021) contains human scores on a Likert scale of to 5, we also ask the LLM to score the summaries in the same range, as shown in Table 1.
3.1 Summary Evaluation Methods
MCQ Scoring (MCQ) Nevertheless, previous works find that the reasoning generated by the LLM does not always make sense (Lyu et al., 2023;
Reason-then-Score (RTS) Given the success of chain-of- thought prompting (Kojima et al., 2022;

---

Choose a more relevant summary from Summary #1 and Summary #2 with respect to the corresponding Article by choosing an option from A, B, or Note that relevance measures the summary's s selection of important content from the Article, whether the summary grasps the main message the Article without being overwhelmed by unnecessary or less significant details.
examine the correlation between the LLM and human evaluations for each system. Idcally, if an LLM stable across systems, should pro- duce evaluations that are similarly correlated to human evaluations. Otherwise, if the correlations differ significantly across different candidates, then we may conclude that the LLM's 's evaluations are system-dependent.
Article: (article]
Second, we define a meta-correlation metric to quantify the extent which the LLM's perfor- mance affected by the quality of the evaluated systems. Specifically, we use the average human score for each candidate as an indicator of its sum- marization quality (Qi), shown in Equation (1);
Summary #1:
[summary from model A)
Summary #2: (summary from model B)
A: Summary #1 is more relevant.
B: Summary #2 is more relevant.
Both Summary #1 and Summary #2 equally relevant.
Your choice (enter letter from A C):
N
1
Q: = N
fuman (Di.j)
(1)
Table 3: Example prompt for the H2H method on the relevance dimension. Text in (blue}: the specific article, and the corresponding summaries generated by pair of compared models.
where fnuman (-) indicates the human evaluation, Ji.j represents the jth summary generated by the jth candidate system. Each candidate's s quality is calculated as an average of N generated summaries (NV = 100 for all systems). Next, we use the corre- lation P. between LLM scores and human scores an indicator of the LLM's 's evaluation performance for the 5th candidate, follows:
Wang et al., 2023b; Gao et al., 2022). To avoid the misguidance of wrongly generated reasoning, we explore a more constrained MCQ method for the Likert-scale scoring. As shown in Table 2, instead of allowing the LLM to freely generate its thoughts, we dictate specific reasoning for each score.
P. - fL=
(2)
Head-to-Head Comparison (H2H) Lastly, some concurrent works also observe that ChatGPT can act as an effective ranking model (Ma al., 2023a,b). We thus explore the hcad-to-head comparison approach for LLM-based evaluations. As shown in Table 3, we present summaries (Summary #1 and #2) generated by different summarization systems on the same input article, then prompt the LLM to select the better summary, to indicate a tie. Moreover, to avoid potential biases that arise from the summary IDs, we conduct each evaluation twice, presenting the same summary as either #1 or #2 respectively.
[fnuman (0i,1) (0.1)... I.....::nuuu Inuman (
where p denotes the correlation metric (i.e., Spear- man correlation, Pearson correlation, or Kendall's ss Tau?), and (LLM(-) indicates the LLM's evaluation for each summary Pij- Finally, we calculate the meta- meta-correlltton' M on a total of ls candidates as:
M = =[[p(((),.. Qn). [P,, Pk))
(3)
Idcally, an LLM should work well regardless of the quality of the evaluated systems, which means that M should be close to zero. On the other hand, significant M would indicate undesirable rela- tionship between the LLM's evaluation capability and the quality of the evaluated systems, suggest- ing that the LLM evaluation is not stable, such that may not evaluate each candidate system fairly using the same standards.
3.2 Stability of LLM Evaluators
To ensure fairness across all evaluated systems, we argue that it is crucial for LLMs to produce stable evaluations. That is, regardless of evaluated systems, the LLMs should maintain a consistent degree of alignment with human judgments. We investigate such stability in two ways.
"We use the scipy.ttats.kendallaa package, which imple- ments tau-b variant that accounts for ties.
"W use the same correlation metric for both Equation2 and 3. For instance, if P. is obtained using Spearman correlation, then M is also calculated using Spearman correlation.
First, We categorize the summarics based on their originating summarization systems, and then

---

4 Experiments
the factual alignment between the summary and the summarized source, (iii) Fluency the quality of individual sentences, and (iv) Relevance the selection of important content from the source.
4.1 Setups
We use the ChatGPT snapshot ($ 4.2) for all three methods. By using a fixed snapshot, we ensure all evaluations are con- ducted with the same LLM model. In addition, we evaluate with the GPT-4 "gpt-4-0314" snap- shot ( 4.3) using the best evaluation method de- termined by ChatGPT check for any potential improvement. Given that ChatGPT and GPT-4 are amongst the top performing LLMs, we use their performance to estimate the potential of LLMs as reliable evaluators. Additional results using three different-sized Llama 2 models (Touvron et al., 2023) are reported in Appendix D, which all per- forms worse. Similar to Luo et al. (2023) and Wu et al. (2023), we set the temperature to 0 and reset the dialogue history for each evaluation instance.
5-turboo-- 00311"
To compare all evaluation meth- ods on equal ground with human evaluation, we use four different measurements. First, we count the number of correct preferences (#CP), which is the number of times each automatic metric has the same preference as the average human scores do over a set of compared system pairs ($ 4.2.1). This can help measure the alignment of evaluation meth- ods with humans at a granular level. To determine the preferred system by particular metric, we as- sign a system point if its generated summary is evaluated as better than that of the other system according the metric, or assign both systems 0.5 for tie. Then, we aggregate the different scores for the compared systems for all 100 test inputs, and the system with higher score is considered the preferred system by that metric (see Appendix Table 22 for details).
Measurements
Dataset We use the SummEval benchmark dataset (Fabbri et al., 2021). This dataset contains expert human annotations for coherence, consis- tency, fluency, and relevance on the generation re- sults from 12 abstractive systems (see details in Appendix table 21) on the CNN/DM dataset (Her- mann et al., 2015). Each evaluated system gen- erates summaries for the same 100 news articles, and each summary is scored by expert annotators from to 5. The annotations achieve with high kappa coefficient of 0.713 (Fabbri et al., 2021). We further calculate the annotations' standard devia- tions across each evaluated system in Appendix Table 20. Given a step size of 1, the standard devi- ations are considered very small, thus suggesting that this dataset has high level of human agree- ment. Following Chiang and Lee (2023), Chhun et al. (2022), and Guan and Huang (2020), we use the average human scores the reference scores.
Next, we also use the Pearson correlation (Co- hen al., 2009), Spearman correlation (Spearman, 1987), and Kendall's 's Tau (Kendall, 1938) mea- sure the relationship between the scores of auto- matic evaluators and humans ($ 4.2.2, 4.2.3, 4.2.4). While the Pearson score mcasures linear relation- ships, the other two measure the ordinal relation- ship that may be non-linear. Moreover, Kendall's ss Tau is less sensitive than Spearman correlation to outliers due to its paired counting of concordant and discordant pairs.
4.2 ChatGPT Evaluator
In this section, we examine the ChatGPT evaluator across many aspects, ranging from human correla- tion and stability across different systems.
Baselines We use ROUGE (Lin, 2004) F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L, BERTScore (Zhang et al., 2020b), BARTScore, BARTScore-CNN, and BARTScooe-CCNN-PA (Yuan et al., 2021) baseline metrics. The last two metrics use BART models fine-tuned on CNN/DM's training data, and are especially strong.
4.2.1 Correct Preferences
The ultimate goal of evaluation is to determine if one candidate system is better than the other in a compared pair. The number of correct preferences (#CP) metric normalizes all evaluation methods into determining whether an evaluator can, as a human expert would, pick the same better system or determine a tie. We conduct such analysis with different pairs of summarization systems on the same input articles. Due the limited budget for API calls, we only evaluate H2H on a challenge set, consisting of candidate pairs with the closest
Prompts We conduct evaluation following our prompt formats given in Table 1, 2. and 3. Follow- ing Fabbri et al. (2021), we re-use the definitions of the evaluation dimensions: (i) Coherence - the collective quality of all sentences, (ii) Consistency

---

Coherence
Consistency
Fluency
Relevance
Metrics
Spear.
Pear.
Kend.
Spear.
Pear.
Kead.
Spear.
Pear:
Kend.
Spear.
Pear.
Kead.
ROUGE-1*
0.193
0.202
0.136
0.155
0.186
0.121
0.075
0.153
0.058
0.323
0.361
0.231
ROUGF-22
0.145
0.146
0.101
0.137
0.156
0.107
0.053
0.095
0.041
0.255
0.262
0.181
ROUGE-L*
0.148
0.158
0.105
0.133
0.167
0.103
0.078
0.146
0.060
0.306
0,340
0.219
BERTScore*
0.375
0,383
0,265
0.163
0.182
0.127
0.167
0.229
0.130
0.396
0.414
0.285
BARTScore"
0,381
0.391
0.275
0.271
0.265
0:212
0.168
0.187
0.131
0.381
0.391
0.276
BARTScore-CNN
0.461
0.480
0.332
0.389
0.413
0.305
0.310
0.378
0.241
0,425
0.309
BARTScore- PARA"
PARA
0.455
0.455
0.328
0.413
0.459
0.324
0.368
0.417
0.286
0.414
0.440
0.299
ChaiGPT-RTS
0.388
0.399
0.312
0.423
0.532
0.378
0.285
0.302
0.240
0.448
0.463
0.357
ChatGPT-MCQ
0.424
0.416
0.350
0.343
0.487
0.320
0.343
0.431
0.305
0.384
0.395
0.329
GPT-4-RTS
0.427
0.461
0.361
0.556
0.618
0.522
0.498
0.600
0.452
0.448
0.428
0.373
Table 4: Spearman (Spear.) correlations, Pearson (Pear.) correlations, and Kendall's Tau (Kend.) between various metrics and human scores for total of 1,200 summaries. results derived from Wang et al. (2023a). Bolded: best results. Underlined: second best results. Values in light gray color are insignificant (p-value > 0.05).
Metrics
Colt
Con
Flu
Rel
Avg
drops significantly to 63.6% (7 out of 11), which indicates LLM evaluators struggle to differentiate the closely matched candidate systems. In other
Random
3,67/22
3,67/22
3,67/22
3,67/22
3,67/22
ROUCE-1
3/40
5/46
3/46
4147
3,75/44,75
ROUGE-2
2/38
7/48
3/45
4.00/44.25
ROUGE-L
2/31
5/37
4/39
6641
4.25/37.00
words, LLM evaluators may only reliably compare candidates with relatively large performance gap.
BERTSeere
4/18
5/14
4F14
6M16
4,75/15.50
BARTScone
8/46
7448
5/45
6A16
6.50/46.25
BARISoorc-CN
9/53
5153
5154
5.75/53.25
4.2.2 Correlations with Human
9/49
8/54
6/52
5/51
7.00/51.50
ChaGPT-RTS
6/54
6/56
9/62
7/62
7,00/58.50
Table 4 reports that Spearman, Pearson correla- tions, and Kendall's Tau between scores of multiple automatic evaluators and humans with a total of 1200 summaries from all systems, across the four evaluation dimensions. As shown, ChatGPT RTS and MCQ demonstrate stronger correlations with humans than many automatic evaluators, such as ROUGE and BARTScore, with up to 0.2 gains in fluency. While RTS achieves higher correlations in the dimensions of consistency and relevance, MCQ has relatively strong correlations in the dimensions of coherence and fluency. Meanwhile, the special- ized BARTScore-ChN family also shows competi- tive performance in coherence, most likely due to the fine-tuning process with CNN/DM.
ChaGPTTMMQQ
5/54
7/56
8/60
7/58
6,75/57.00
ChaGGPT-HH
8/-
7/-
7/-
6.50/
GPT-4-RTS
5/55
7/53
8/60
7/56
6,75/56.00
Table 5: Number of correct preferences (#CP) on the 11- 11-pair challenge set (in black) and the 66-pair full set (in brown). Random: for each pair, there are three pos- sibilities (two possibilitics for one model being better, one possibility for tie) the random #CP is one-third of the total compared pairs.
average performances according to human scores. However, for RTS, MCQ, and other baselines, we can easily calculate the #CP for all 66 possible pairs (see Appendix E).
Table 5 reports the #CP for both the standard 66- pair full set (in brown) and the 1 1-pair challenge set (in black). As shown for the larger standard set, RTS unanimously obtains the largest #CP across all dimensions, with an average of 58.5 out of 66 candidate pairs (i.e. 88.6% accuracy).
4.2.3 Per-candidate Correlations
Next, break down the human correlation of ChatGPT-RTS for each candidate system and mea- sure the statistical spread for the correlations across all systems (see raw results in Appendix table 23), Idcally, a stable evaluator should exhibit the same human correlation across candidates and dimen- sions, and display flattened boxes in a line.
Despite the high overall accuracy, weaknesses of such evaluators are revealed as we dive into their performances in the 11 -pair challenge set (black scores of Table 5), where the evaluated candidates are close matches. Specifically, BARTScore-CNN Para performs better than RTS in coherence and consistency, possibly because it is fine-tuned with same-domain summarization data. For fluency and relevance, ChatGPT-RTS still performs best among all evaluators. Nonetheless, its average accuracy
However, as illustrated in Figure 1, the spread of correlations for different candidates is particu- larly wide, with up to 0.5 correlation difference in consistency. This means that the RTS evaluator exhibits significantly varying degree of alignment with human judgment for different candidates. other words, ChatGPT-RTS is candidate deppedent

---

Coherence
Consssency
Fluency
Relevance
Spear.
Pear.
Kead.
Spear:
Pear
Kend.
Spear.
Pear.
Kend.
Spear.
Pear:
Kead.
ROUGE:1 1
0.0
000022
0.091
0.490
00.527
0.303
00.40
00.118
00.273
0.420
0.387
00.273
ROUGE-2
0.559
-0,259
-0.480
-0.152
-0.217
-0.438
-0.152
-0.084
-0.120
-0.121
ROUGE-L
0.231
0.251
0.121
-0.510
-0:522
-0.303
-0.168
-0.412
-0.152
-0.234
-0.266
-0.121
BFRTScore
-0.413
-0.403
-0.212
-0,580
-0.800
-0.424
-0.455
-0.663
-0.303
-0.685
-0.756
-0.515
BARTScore
-0.916
-0.747
-0.788
-0,266
-0.504
-0.121
0.154
0.123
0.182
-0.769
-0.837
-0,006
BARTScore-CNN
-0.913
MARTSSore-CNN-PA
.0.720
-0.858
006066
-0.685
.0.888
-0.576
40.294
00.522
/0.212
.0.853
40,800
0.727
ChaGGT-RTS
-0.04
-0.072
00.11
.0.811
-0.751
-0.636
40.748
+0.728
+0,606
-0.559
-0,433
-0.94
ChatGPT-MQQ
-0.175
-0.11
-0.182
-0.818
-0411
-0.636
-0.622
-0.484
-0. 00.994
-0.350
-0.622
-0.212
GPT-4-RTS
-0.531
40.678
-0.424
-0.600
-0.103
-0.236
-0.839
-0.520
-0.515
-0.958
40.880
-0.48
Table 6: Meta Meta-correlation for various evaluation methods. Bolded: most negative meta- cora-correlation. Underlined: second most negative meta-correaation, Values in light gray color are insignificant (p-value 2 0.05).
Spearman Pesnson Kendall
Concelations (P.)
0.6
0.4
MCQ
0.4
A
0.2
3.6
4
4.5
0.2
Average Human Scores (Q:)
Spread of per-candidate correlations with human scores for ChatGPT-RTS evaluations.
Figure/ 1:
Figure 3: Relationship between per-model correlations (Kendall's Tau) and human scores on consistency.
rTS
MCQ
0
Kendall's Tau for both RTS and MCQ in Figure 2. As shown, both RTS and MCQ exhibit strong neg- ative meta-correlation for consistency and fluency. This suggests that ChatGPT becomes less human- aligned with improving qualities of the evaluated systems.
-0.5
-1
To illustrate this phenomenon further, we scatter the paired coordinates of the summarization system quality (Qi, Equation (1)) and ChatGPT's evalua- tion performance (P,, Equation (2)) in Figure 3. As shown, while the LLM evaluator is better human- correlated with lower-quality candidates (< 3.5), it less reliable when dealing with high- can- didates (> 4.7) with much lower and inconsistent correlations.
Coh
Con
Flu
Rel
Figure 2: Meta-correlation (Kendall's Tau) for RTS and
MCQ. Shaded: statistically significant with p < 0.05.
and one should not expect such LLM evaluators to have the same level of human alignment summarization system. Similar trends can also be observed for MCQ (see Appendix table 24).
We compare the meta-correlation for all evalu- ation metrics in Table 6. We can see that while the ROUGE metrics exhibit no significantly nega- tive meta-correlation, the neural metrics all display significant meta-correlatio in certain dimensions. One highly likely reason for this behavior is due to the varying biases inherent the neural mod- els, which would explain why ROUGE simple n-gram overlap metric doesn't it exhibit significant negative meta-correlations. Interestingly, ROUGE- even shows a strong positive meta-correlation on coherence (which is plausible, because bi bi-gram
In addition, the medians across the four dimen- sions are also different. This indicates that the ChatGPT is also dimeension-deeedeer and unsta- ble. Given such varying performances across dif- ferent dimensions, ChatGPT may not behave well with newly introduced evaluation criterion.
4.2.4 Summary Quality vs Human Alignment
Using our proposed meta-correlation measurement in 3.2, we analyze the relationship between sum- mary quality and human correlation of LLM evalu- ators. We illustrate the cereelionol in terms of

---

MCQ
[ Speceman I Pessson I Kandall
0.8
4.59g
4.61
0.6
Average Scores
0.4
3.98
3.68
3.72
0.2
3
2,59
Coh
Con
Flu
Rel
Coh
Con
Flu
Rel
Figure 5: GPT-4's (RTS) spread of per-candidate corre- lations.
Figure 4: The average ChatGPT RT'S and MCQ scores and human scores across dimensions.
maries according to false premises (more examples in Appendix G). For instance, RTS may penalize the summary's 's repetitiveness in the consistency di- mension or suppress fluency ratings for missing important details. On the other hand, the MCQ counterpart gives higher overall scores, most likely because the confined set of pre-defined reasons prevents such unrelated penalization, though not leading better human alignment.
overlap performance may be more accurate as can- didates produce more coherent texts). Both the BARTScore variants and LLMs demonstrate the most negative meta-correlations. ChatGPT-RTS has the most negative meta- correlation in the dimensions of consistency and fluency, indicating that it may be the least reliable to evaluate high-quality systems on these dimen- sions. On the other hand, the BARTScore family may be unreliable in comparing systems with high qualities of coherence, consistency, and relevance. So far, the observations discussed 4.2.3 and $ 4.2.4 collectively suggest that LLM evaluators may not be a reliable standalone metric for chal- lenging scenarios, and further human evaluation is required for conclusive decisions.
4.3 GPT-4 Evaluator
A natural question ask is whether such afore- mentioned limitations are resolved with an stronger LLM. In this section, we conduct similar analyses on GPT-4 (OpenAI, 2023) with the RTS method. We present the GPT-4 results in the last rows of Table 4 and 5. The results suggest that stronger LLM does not necessarily translate to a stronger LLM evaluator, although Table 4 does show that GPT-4 outperforms ChatGPT in terms of human correlation consistently across most dimensions.
4.2.5 RTS and MCQ Scores
Lastly, delve into the detailed scores generated by ChatGPT with either the RTS MCQ method. Since both methods score the summaries in the same range of human scores of 1 to (Fabbri et al., 2021), we can show a direct comparison of the aver- age RTS and MCQ scores with human scores in Fig- ure 4 (see more details in Appendix F). As shown, the RTS scores are much lower than the human scores across all dimensions, while MCQ scores are consistently higher and better match the hu- man scores (except for relevance). In other words, while RTS is best aligned with humans according $ 4.2.1 and $ 4.2.2, we cannot replace the human scores with RTS scores in absolute terms.
Unfortunately, GPT-4 still suffers from the same limitations as ChatGPT. It appears to be both candidate-dependent and dimension-depenndnnt, demonstrated by the large spreads with varying median values across dimensions Figure and the significantly negative mneaa-correattonn. out dimensions (Table 6). However, GPT-4 is less dimension-dependdan as compared to ChatGPT, the medians in the box plots in Figure 5 are more aligned than those in Figure 1.
In addition, there notable enhancement the meta-correlation for consistency, which we at- tribute to a significant reduction in reported halluci- nations with GPT-4 (OpenAl, 2023). It is possible that with much more instruction training to avoid hallucinations, GPT-4 is much better aligned with humans detect inconsistencies (i.e. hallucina-
The discrepancy may be attributed to the un- faithful reasoning generated by LLMs (Lyu et al., 2023; Wang al., 2023b; Gao et al., 2022). Our further investigation suggests that ChatGPT-RTS generates false or snvelated-to-dimenne ing. Thus, it is possible that the much lower scores could be caused by ChatGPT penalizing the sum-
reason-
Fabbri et al. (2021) observe similar issues with crowd- sourced non-expert annotators.

---

Coherence
Consistency
Fluency
Relevance
Spear.
Pear.
Kend.
Spear
Pear.
Kead.
Spear.
Pear.
Keod.
Spear.
Pear.
Kend.
plRs, parn))
0.343
0. 1.198
0.091
0.685
0,506
0.576
0,727
0.797
0,545
0.168
0,128
0.000
plRs, P.
0.657
0.625
0.394
0.685
0.616
0.515
0.322
0.573
0,212
0.091
-0.106
0.000
Table 7: Correlations between RTS-MCQ R2 and RTS-Human (PRIX) and MCQ-Human (P/OO)) High values suggest Rg can be reliability indicator for RTS and MCQ. Light gray values are insignificant (p 2 0.05).
P. ItrS/MCQ is the performance of either method as defined in Equation (2). Given significantly large positive values of either p(Ri, PRTS) or p R,, PMCC), we then conclude that R; be used as reliable indicator for the performance of the corresponding method.
tions) in summaries.
Nevertheless, GPT-4 exhibits a much worse neg- ative meta-correlation the relevance dimension, which, interestingly, scems to reflect the challenges of maintaining both "truthfulness" and "informa- tiveness" (Ouyang et al., 2022). This is because a model could be easily made more truthful if al- lowed to provide less relevant information (for in- stance, by refusing to answer the users' questions). It is possible that with reduced capability in the informativeness dimension, the model is less capa- ble of differentiating the nuances less relevant summaries when the summary quality is generally high. Nevertheless, we leave to future work to determine whether GPT-4's more negative meta- correlation in the relevance dimension could be related to its stronger performance in consistency. We provide more details the GPT-4 evaluator in Appendix H.
As shown in Table 7, Rg demonstrates a signif- icant correlation with P.RSS on both the consis- tency and fluency dimensions, and with P:McQ on the coherence and consistency dimensions. This means that if RTS and MCQ generally agree with each other on the candidate's performance on a particular dimension with high pl R4, PRTS) (or p R,, PMCC)), RTS (or MCQ) is more likely be human-aligned. Meanwhile, if RTS disagrees with MCQ (R, r), further human evaluators are required to provide a conclusive evaluation. We provide R3 values for ChatGPT on each evaluated system in Appendix Table 29.
5 A Temporary Efficient Framework
6 Conclusion
Despite the aforementioned limitations, it may be hard to resist the temptation of using LLM evalu- ators given their superiority over other automatic metrics. In such a case, one should be able to tell when LLM evaluators are more likely to be unre- liable and employ further human evaluation when necessary. To this end, suggest combining the RTS and MCQ scores cost-efficient framework. Specifically, we calculate the correlation between RTS and MCQ scores for the ,th candidate system as reliability indicator:
We explore the potential of using LLMs with dif- ferent prompting techniques as metrics for abstrac- tive summarization systems. Our extensive anal- ysis suggests that while LLMMs like ChatGPT per- form better than commonly used automatic met- rics across different summarization systems and dimensions. they are still ready to replace hu- man evaluators because they are candidate- and inmension-dependeen and they do not align well with human when comparing high-quality candi- dates. Nonetheless, if an LLM evaluator is to be used, we suggest combining multiple evaluation methods preliminary indicator to determine whether the metric is likely to be unreliable and whether further human evaluation is required.
R2 (([RRT(())))) SRTS JRTS (91,)))
(4) [fMco(gg.1), (M,N()
Then, we can loosely infer that up to a reliability tolerance E (0, 1), the LLM evaluators (either RTS or MCQ) are reliable if R; > r. In other words, given candidate i, if RTS and MCQ agree with each other up certain degree of tolerance r, we may assume the evaluator is reliable enough to avoid invoking further human evaluation.
Limitations
Potential Human Bias. We benchmark the LLM evaluation results against the average of three hu- man expert scores. Naturally, it is possible that these scores may exhibit potential biases of the human experts. Nevertheless, we wish to explore
To validate this theory, we measure the cor- relations pf R,, PRTS) or p(R; PMCC), where

---

whether LLM evaluators are aligned with human experts, and may naturally exhibit the same bias as a human would. In other words, we examine whether we can reliably replace human annotators with LLMs, instead of seeking a "perfect" solution that has absolutely zero bias.
maries for each system relatively low (around USD per system using ChatGPT). Since LLMs also conduct evaluations much faster than humans (around 2 minutes for LLMs versus 10 hours for human for 100 summaries); it may not pose signifi- cant barriers if one was to re-evaluate all compared systems single LLM.
Dataset Size. Given the constraints of the small size of the human-annotated SummEval dataset, we could only evaluate 100 summaries generated for each summarization system, with a total of 12 abstractive summarization systems, Since we have observed significant correlation of LLM evalua- tions with humans for the consolidated 1200 sum- maries across all systems, it is possible that with a larger evaluation number, the per-system corre- lation could also be improved. In addition, given only 12 evaluated systems, our meta-correlation may still be subject to sample biases. We leave more investigations for the future once there are larger annotated datasets.
Limited Use of the Temporary Solution Unfor- tunately, our temporary efficient framework doesn't apply to the relevance dimension, where the R₃ has no significant correlation with the performances of either RTS or MCQ. Morcover, the value may be dataset-dependent, and is hard to decide where to draw this line. We leave for future work of de- veloping better methods to gauge the reliability of LLM evaluations.
Acknowledgements
Yang You is being sponsored by NUS startup grant (Presidential Young Professorship). Singa- pore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant.
Prompt tuning. Designing better prompt for LLMs are also ongoing research. Although it is possible that LLMs may act better evaluators with better prompts, prompt tuning is not our fo- cus. We seek highlight the limitations of the investigated LLMs and have demonstrated that lim- itations such as negative meta-correlation are also found with a few other alternative prompts (see Appendix C).
References
Lidong Bing. Piji Li, Yi Liao, Wai Lam, Weiwei Guo, and Rebecca Passonneau. 2015, Abstractive multi- document summarization via phrase selection and merging. In Proceedings of ACL-IJCNLP.
Moye Chen, Wei Li, Jiachen Liu, Xinyan Xiao, Hua Wu, and Haifeng Wang. 2021. Sgsum: Transform- ing multi-document summarization into sub-graph selection. Proceedings of EMNLP.
We note that the "gpt-3. 5 5-turbo-0301" snapshot cur- rently taken down' by OpenAl and replaced with a This is also one disadvantage of using out-of-the-box commer- cialized LLM for summarization evaluations, as the exact checkpoints may not be stably available. As result, future models may not be fairly com- pared against previously evaluated models using a different LLM checkpoint. Nevertheless, our paper only seeks to investigate the potential of LLM as an out-of-the-Boe evaluator, and the Ap models are currently one of the strongest. Eventually, we wish to raise awareness of some of the significant limitations found with these LLMs, which need to be resolved before L.MMs can be used direct re- placements for human evaluations. In addition, we also note that the cost of evaluating only 100 sum-
Availability of Commercialized LLM
newer snapshot, "gpt-3.5-turbo-66"
Liying Cheng, Xingxuan Li, and Lidong Bing. 2023. Is gpt-4 good data analyst? arXiv preprint 23Xiv:2305. 1505.
Liying Cheng, Dekun Wu. Lidong Bing. Yan Zhang Zhanming Jie, Wei Lu, and Luo Si. 2020. Ent-desc: Entity description generation by exploring knowl- edge graph. In Proceedings of EMNLP.
Cyril Chhun, Pierre Colombo. Fabian M. Suchanek. and Chloé Clavel. 2022. Of human criteria and au- tomatic metrics: A benchmark of the evaluation of story generation. In Proceedings of Coling.
Cheng- Han Chiang and Hung-yi yi Lee. 2023, Can large language models be alternative to human evalua- tions? In Proceedings of ACL.
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacoh Ben- esty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coeffi- cient. Noise reduction in speech processing.
We release all LLM generations involved in our experi- ments in https:/ thtbb thub. com/DAMO-No SG/LLM_summev summev al as JSON files.

---

Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization contextual bandit. In Proceedings of EMNLP.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising www.rreeeueseeeeeeeeeeeeeeeeeeee
sequence-to-sequenc pre-training for nat- ural language generation, translation, and compre- hension. In Proceedings of ACL.
Ori Ersst, Avi Caciularu, Ori Shapira, Ramakanth Pa- sunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition-levt clustering for multi- document summarization. In Proceedings of NAACL.
Piji Li, Wai Lam. Lidong Bing, and Zihao Wang. 2017. Deep recurrent generative decoder for abstractive text summarization. In Proceedings of EMNLP.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of ACL.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In Proceedings of ACL.
Alexander R Fabbri, Wojciech Kryáciński, Bryan Me- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. TACL.
Chin- Yen-Yewwlan Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branckes Out.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In ICLR.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of ACI.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment. arXiv 2303.16634.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. In Proceedings of ICML.
Yang Liu and Mirella Lapata. 2019. Hierarchical trans- formers for molti-docuumnt summarization. In Pro- ceedings of ACL.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang. and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. arXiv preprint arXiv:230400255.
Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed H Awadallah, and Dragomir Radev. 2022. Leveraging locality in abstractive text summarization. In Proceedings of EMNLP.
Shen Gao, Xiuying Chen, Piji Li, Zhaochun Ren, Li- dong Bing, Dongyan Zhao, and Rui Yan. 2019. stractive text summarization by incorporating reader comments. In Proceedings of AAAI.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.1500
Jian Guan and Minlie Huang. 2020. UNION: An Un- referenced Metric for Evaluating Open-ended Story Generation. In Proceedings of EMNLP.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning. arXiv preprim arXiv:2301. 133799379.
Junxian He. Wojeiech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. CTRL- Towards generic controllable text summariza- tion. In Proceedings of EMNLP.
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023a. Zero-shot listwise document reranking with a large language model. arXiv preprint 200.02506
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines read comprehend. NeurIPS.
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023b. Large language model not good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXhv:2303.00 00850959.
MG Kendall. 1938. new measure of rank correlation. Biometrika.
Ani Nenkova. 2006. Summarization evaluation for text and speech: issues and approaches. In Ninth Interna- tional Conference on Spoken Language Processing.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models zero-shot reasoners. NeurIPS.
Ani Nenkova and Kathleen McKeown. 2012. A survey of text summarization techniques. Mining text data.
Wojeiech Kryšciniski, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of EMNLP.
OpenAL. 2023. Gpt-4 technical report. arXiv preprint.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. NearlPS.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Cop- pin, and Po-Sen Huang. 2021. Challenges in detoxi- fying language models. In Findings of EMNLP.
Karolina Owezarzak, John Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accu- racy of automatic evaluation in summarization. In Proceedings of workshop evaluation metrics and system comparison for automatic summarization.
Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. 2023. Large language models are diverse role-pli playeps for summarization evaluation. arXiv preprint 2rXiv22333.15777
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. Primera: Pyramid-base basedd masked sen- tence pre-training for multi-document summarization. In Proceedings of ACI.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for factuality metrics. In Proceedings of NAACL-HLT.
Jiacheng Xu and Greg Durrett. 2021. Dissecting gen- eration modes for abstractive summarization models via ablation and attribution. In Proceedings of ACL- IJCNLP.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of ACL.
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang. Mingfeng Xue. Boxing Chen, and Jun Xie. 2023. Tailor: soft-prompt-based approach to attribute-based controlled text generation. In Pro- ceedings of ACL.
Swarnadeep Saha, Shiyue Zhang, Peter Hase, and Mo- hit Bansal. 2022. Summarization programs: Inter- pretable abstractive summarization with neural mod- ular trees. In ICLRR
Chenhui Shen, Liying Cheng, Lidong Bing, Yang You, and Luo Si. 2022a. Senths: Sentence-level beam search for controllable summarization. EMNLP.
Weizhe Yuan, Graham Neubig, and Pengfei Liu, 2021. Bartscore: Evaluating generated text as text genera- tion. NewrlPS.
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Li- dong Bing, and Yang You. 2023. A hierarchi- cal encoding-decooding scheme for abstractive multi- document summarization. In Findings of EMNLP.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- Liu. 2020a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of ICML.
Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022b. Mred: A meta- review dataset for strutture-comtroolaalle text genera- tion. Findings of ACI.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with bert. In ICLR.
Qingyu Zhou, Nan Yang. Furu Wei, Shaohan Huang. Ming Zhou, and Tiejun Zhao. 2018. Neural docu- ment summarization by jointly learning score and select sentences. In Proceedings of ACL.
C Spearman. 1987, The proof and measurement of association between two things. American Journal of Psychology.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv: wwXiv:999.08599)
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt good nig evaluator? a preliminary study. arXiv preprint 23xv:2303..0000
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Fdd Chi, and Denny Zhou. 2023h. Self-consistency improves chain of thought reasoning in language models. In ICLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le, Denny Zhou, al. 2022. Chain-of-thie of-hogh prompting elicits rea- soning in large language models. NewrIPS.

---

Score the following Summary given the corresponding Ar- ticle with respect to consistency from one to five, where one indicates "inconsistency" and five indicates "perfect consistency". Note that consistency measures the factual aligmment between the Summary and the Article, whether the Summary is faithful to the Article without introducing contradictions or misleading representations.
Choose an option from A to E in order to score the follow- ing Summary given the corresponding Article with respect to consistency from one to five, where one indicates "in- consistency" and five indicates "perfect consistency". Note that consistency measures the factual alignment between the Summary and the Article, whether the Summary is faithful to the Article without introducing contradictions or mislead- ing representations.
Artiele: (article|
Article: (article]
Summary: (summary)
Summary: (summary)
Provide your reason in one sentence, then give final score:
A: The Summary is totally inconsistent with the Article.
Table 8: Example prompt for the RTS method on the consistency dimension. Text in (blue): the specific article and the corresponding summary to be evaluated.
Score: One.
B: The majority of the Summaay is inconsistent with the
Article. Score: Two.
C: Some information in the Summaay is consistent with the
Article whereas some are not. Score: Three
Score the following Summary given the corresponding Arti- ele with respect to fluency from one to five, where one indi- cates "disfluency" and five indicates "perfect fluency". Note that fluency measures the quality of individual sentences in the Summary, whether the Summary is well-written, gram- matically correct, and readable the sentence level.
D: The majority of the Summary consistent with the
Article, Score: Four.
E: All information included in the Summary is consistent with the Article. Score: Five.
Your Answer (enter letter from A to E):
Article: (article]
Table 11: Example prompt for the MCQ method on
the consistency dimension. Text in [cyan]: ; the specific article and the corresponding summary to be evaluated.
Summary:
(summary)
Provide your reason in sentence, then give final score:
structured and well-organized. The summary should not just be a heap of related informa- tion but should build from sentence to sen- tence to a coherent body of information about a topic.
Table 9: Example prompt for the RTS method on the fluency dimension. Text in (cyan): the specific article and the corresponding summary to be evaluated.
Score the following Summary given the corresponding Ar- ticle with respect to coherence from to five, where indicates "incoherence" and five indicates "perfect coher-
2. Consistency: The factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source doc- ument.
ence", Note that coherence measures the collective quality of the Summary, whether the Summary presents informa-
tion that flows smoothly and avoids abrupt transitions or disjoint statements.
Article: (article]
3. Fluency: The quality of individual sentences. Sentences in the summary should have no for- matting problems, capitalization errors or ob- viously ungrammatical sentences (e.g., frag- ments, missing components) that make the text difficult to read.
Summary: (summary]
Provide your reason in one sentence, then give final score:
Table 10: Example prompt for the RTS method on the coherence dimension. Text in [cyan]: ): the specific article and the corresponding summary to be evaluated.
4. Relevance:
Selection of important content from the source. The summary should include only important information from the source document.
A Evaluation Dimensions
Fabbri et al. (2021) has defined 4 evaluation dimen- sions as follows:
We follow the above definitions for designing ChatGPT's evaluation prompts.
1. Coherence: The collective quality of all sentences. The summary should be well-

---

Choose an option from A to E in order to score the follow- ing Summary given the corresponding Article with respect to fluency from one to five, where one indicates "disflu- ency" and five indicates 'perfect fluency". Note that fluency measures the quality of individual sentences in the Sum- mary, whether the Summary is well-written, grammatically correct, and readable on the sentence level.
Choose a more consistent summary from Summary #1 and Summary #2 with respect the corresponding Article by choosing an option from A, B, or C. Note that consistency measures the factual alignment hetween the summary and the Article, whether the summary is faithful to the Article without introducing contradictions or misleading represen- tations.
Article: (article]
Article: [article]
Summary:
(summary)
Summary #1:
[summary from model A}
A: The Summary is totally disfluent. Score: One.
Summary #2: (summary from model B)
The majority of the Summary is disfluent. Score: Two. C: Some sentences in the Summary are fluent whereas some
A: Summary #1 is more consistent.
are not. Score: Three
B: Summary #2 is more consistent.
The majority the Summary is fluent. Score: Four
C: Both Summary #1 and Summary are equally consis- tent.
E: All sentences in the Summary are fluent. Score: Five.
Your Answer (enter letter from A to E):
Your choice (enter letter from to C):
Table 12: Example prompt for the MCQ method on the fluency dimension. Text in (cyan]: the specific article and the corresponding summary to be evaluated.
Table 14: Example prompt for the H2H method on the consistency dimension. Text in [cyan): the specific article, and the corresponding summaries generated by pair of compared models.
Choose option from to in order score the following Summary given the corresponding Article with respect co- herence from one to five, where one indicates "incoherence" and five indicates "perfect coherence". Note that coherence measures the collective quality of the Summary, whether the Summary presents information that flows smoothly and avoids abrupt transitions disjoint statements.
fluent summary from Summary #1 and Sum- mary #2 with respect the corresponding Article by choos- ing an option from A, B, or C. Note that fluency measures the quality of individual sentences the summary, whether the summary is well-written, grammatically correct, and readable on the sentence level.
Choose
Article: [article]
Article: [article]
Summary:
(summaay)]
Summary #1: (summary from model A}
A: The Summary completely incoherent. Score: One.
Summary #2: (summary from model B)
The Summary is mostly incoherent. Score: Two.
C: The Summary is somewhat coherent. Score: Three
A: Summary #1 is more fluent.
D: The Summary mostly coherent. Score: Four.
B: Summary #2 is more fluent.
E: The Summary is completely coherent. Score: Five.
C: Both Summary #1 and Summary #2 are equally fluent.
Your Answer (enter letter from A E):
Your choice (enter 1 letter from A to C):
Table 13: Example prompt for the MCQ method on the coherence dimension. Text in [cyan]: specific article and the corresponding summary to be evaluated.
Table 15: Example prompt for the H2H method on the fluency dimension. Text in [cyam): ;: the specific article, and the corresponding summaries generated by pair of compared models.
B Prompt Details and Design
We show the RTS prompts for relevance, consis- tency, fluency, and coherence in Table 1, Table 8, Table 9, and Table 10 respectively.
To determine the exact definitions used in our prompts for each dimension, we re-use the first sentence from Fabbri et al. (2021)'s definition. We then prompt the LLM to provide definition for the evaluated dimension, such "define the word relevance in the context of summarization" then extract the key phrases generated that we believe to fit the definitions of Fabbri et al. (2021) to make up the full definition. We believe this approach may
We show the MCQ prompts for relevance, con- sistency, fluency, and coherence in Table 2, Table 11, Table 12, and Table 13 respectively.
We show the H2H prompts for relevance, consis- tency, fluency, and coherence in Table 3, Table 14, Table 15, and Table 16 respectively.

---

D Llama 2 Results
Choose more coherent summary from Summary #1 and Summary #2 with respect the corresponding Article by choosing an option from A, B, or C. Note that coherence measures the collective quality of the summary, whether the summary presents information that flows smoothly and avoids abrupt transitions or disjoint statements.
We report the results of using three different sizes of Llama 2 models as LLM evaluators in Table 19, As shown, while the smallest model (7B) exhibits very low correlations with human scores (and only significant the consistency and relevance dimen- sions); the larger models (13B and 70B) demon- strate significant correlations with human scores on the full dataset level. However, even the best- performing 70B model fails to outperform the hu- man correlation of BARTScore, and is completely overwhelmed by the results of ChatGPT and GPT- 4. This suggests that the open-sourced Llama 2 models are not suitable to be used zero-shot evaluators. Moreover, all Llama 2 models exhibit significant meta-correlations for at least one dimen- sion.
Article: (article]
Summary #1: (summary from model A]
Summary #2: (summary from model B]
Summary #1 is coherent.
B: Summary #2 is more coherent.
C: Both Summary #1 and Summary #2 are equally coherent.
Your choice (enter letter from A to C):
Table 16: Example prompt for the H2H method the coherence dimension. Text in [cyan]: the specific article, and the corresponding summaries generated by pair of compared models.
Coh
Con
Flu
Rel
Avg
ChaGGPT-RTS2 ChaGPT-MMQQ
6
7
6
6,00
5.25
ChaiGPT-StarEwe
6
6.75
Table 17: Total correct pairs for alternative prompts. Coh: coherence; Con: consistency; Flu: fluency; Rel: relevance.
help the LLM to better evaluate the summaries according to definitions partially generated in its own language. Nevertheless, we didn't 't invest ex- tensive efforts in prompt designs as this is not key focus. We also demonstrate that our prompts have better evaluation results than two alternative prompts in Appendix
Alternative prompts
We also ChatGPT evaluate with the exact prompts from Wang al. (2023a). We name these prompts "StarEval" since they prompt the LLM to give one to five stars for the summary. In ad- dition, we use ChatGPT to evaluate with alterna- tive prompts for RTS and MCQ by using the full definition as shown in Appendix A instead of sup- plementing the definition with ChatGPT-gennrated phrases. We name these two prompts RTS2 and MCQ2 respectively.
We show the results of these alternative prompts in Table 17 and Table 18.

---

Coherence
Consistency
Fluency
Relevance
Spear.
Pear.
Kead.
Spear.
Pear.
Kend.
Spear.
Pear.
Kend.
Spear.
Pear.
Kend.
Human-Corr
0.339
0.338
0.285
0,393
0.497
0.350
0.290
0.280
0.252
0.440
0.441
0.351
RTS2
Meta-Corr
-0.559
-0.476
-0.424
-0.74
-0.843
-0.576
-0.825
-0.823
-0.636
-0.385
-0.566
-0.212
Human-Corr
0.430
0.423
0.355
0.327
0.258
0.396
0.229
0.206
MCQ2
Meta Cotr
00,308
00,275
-0.212
00.45
40.446
.0.64
00811
-0615
-0.336
00.17
-0.007
-0.182
StarEval
Human Conr
0.418
0.417
0.341
0,297
0.421
0.264
0.246
0.323
0,217
0.393
0.405
0.323
Meta-Cou Conr
-0.084
-0.001
0.000
-0.664
-0.684
-0,545
00.44
-04460
-0.212
-0.997
-0.579
+0.305
Table I8: Results of using alternative prompt with ChatGPT. Light gray values are insignificant (p-value 0.05).
Human-Cove reports the overall correlation of ChatGPT scores with human scores, Meta-corr shows the meta- correlation.
Coherence
Consistency
Fluency
Relevance
Spear:
Pear.
Kend.
Spear:
Pear.
Kend.
Spear.
Pear.
Kend.
Spear.
Pear:
Kend.
Hurman-Can Corr
00.001
-0.000
-0.001
0.114
0.130
0.104
0.043
0.038
0,039
0.067
0.064
0.057
7B
Meta-Conr
0.245
-0,258
0.182
-0.524
-0.720
-0.424
-0.042
0.463
-0.03
0.238
0.169
0.212
Haman-Corr
0.153
0.165
0,123
0.180
0,209
0.162
0.187
0.179
0,167
0,234
0.266
0.192
13B
Meta-Com
0.2877
00425
-0.182
0,580
00,45
00.424
(0.455
00.233
-0.303
-0.049
-0,00
0.000
Human-Cor
0.234
0.254
0,186
0.357
0.395
0.319
0.155
0.161
0.134
0.248
0.285
0,200
70B
Mcta-Cosr
-0.420
-0.407
-0.273
-0.811
-0.690
-0.667
-0.322
-0.319
-0.182
-0.238
-0,123
-0.182
Table 19: Results of Llama 2 models of 7B, 13B, and 70B RTS correlations. Light gray values are insignificant (p-value 2 0.05). Human-Cor reports the overall correlation of LLM scores with human scores. Meta-cort shows the meta-correlation.
ID
Coh
Con
Flu
Rel
TD
Model Name
Coh
Con
Flu
Rel
Avg
M8
0.58
0.09
0.13
0.51
M22
BART
4.18
4.94
4.90
4.25
4.57
M9
0.65
0.15
0.34
0.54
M23 Pegasus (C4)
4.16 4.91
4.88
4.26
4.55
M10
0.58
0.22
0.20
0.54
M17 T5
4.00
4.93
4.93
4.23
4.52
MI1
0.59
0.34
0.40
0.49
M12 Unified-ext-abs
3.60
4,96
4.85
3.85
4.32
M12
0.65
0.03
0.14
0.54
M13 ROUGESal
3.44
4.82
4.86
3.83
4.24
M13
0.62
0.07
0.14
0.51
MIS Closed book decoder
3.35
4.95
4.80
3.67
4.19
M14
0.65
0.09
0.18
0.54
M14 Multi-task (Ent QG)
3.20
4.90
4.74
3.63
4.12
M15
0.60
0.04
0.13
M8 Pointer Generator
3.29
4.65
4.79
3.55
4.07
M17
0.58
0.05
0.08
0.52
M9 Past-abb-rl
2.38
4.67
4.50
3.52
3.77
M20
0.48
0.29
0.24
0.49
M10 Bottom-Up
2.73
4.25
4.42
3.38
3.70
M22
0,58
0.02
0.14
0.54
M20 GPT-2 (zero-shol)
3.63
3.40
3.97
3.30
3.58
M23
0.58
0.05
0.12
0.54
M1l Improve-ubs
2.28
3.27
3.65
3.15
3.09
Table 20: The standard deviation of human annotations across different summarization systems and evaluation dimensions.
Table 21: The average human evaluation scores of vari-
ous abstractive summarization models reported by Fab- bri et al. (2021). We calculate the average (Avg) score of the reported coherence (Coh), consistency (Con), flu- ency (Flu), and relevance (Rel) scores. Rows are sorted according to the Avg column values in descending or- der.
E Challenging Pairs
To count the total correct pairs, we only evaluate the challenging pairs, which consist of summariza- tion systems of consecutive performances accord- ing to average human scores across all dimensions. Thus, each pair contains 2 summarization systems with the smallest difference in terms of average performance.
we simply need to compare the evaluated values across all systems, and each metric only needs to evaluate a total of 1200 summaries. However, for H2H, we need to evaluate total of 6,600 summary pairs for the full standard set, and each pair needs to be evaluated twice with different summary posi- tions (see 3.1), resulting in a total of 13,200 LLM evaluations. Due to a limited budget, we thus only compare a challenge set of 11 pairs, reducing the total required LLM evaluations to 2,200.
For instance, as shown in Table 21, M22 has the best average human score of 4.57, followed by M23 of 4.55, then M17 of 4.52. We thus compare model pairs of "M22-M23" and "M23-M17". The full challenge set is shown in Table 22. For RTS, MCQ, and all other baseline metrics,

---

Coherence
Consistency
Fluency
Relevance
Model A
Model B
LLM
Human
LLM
Human
LLM
Human
LLM
Human
M22
M23
65.5
53.5
55.75
52.5
61
49.5
x
58.75
49.5
x
48.25
45.5
52
MI7
66.5
43.25
48.5
54.5
49.25
72.5
M12
58
56.5
54.5
56.75
50
58
45
MI3
M15
45.5
49.5
52
48.75
52
51.25
60.5
MI5
M14
57
53.5
56.5
52
V
54
57
M14
M8
49.25
54.5
47
46.5
V
49.25
53.5
M8
M9
77.5
78.5
53
80.5
63.5
V
76
54
M9
M10
45
36
41.5
58
46
44.5
V
41.5
56
X
M10
M20
58.25
24
61,75
64
V
63.75
61.5
V
61.5
54.5
M20
MI1
56.5
82
50
53
51
58.5
V
50
53
X
#CP
Table 22: #CP calculation for the ChatGPT-HHH metric of Model A over Model B. The numerical values in the middle section columns are aggregated scores for Model A. We omit the value for Model B, which is simply "100 aggregated scores for Model A". We use "v" to indicate both LLM and humans prefer the same model, and "x" otherwise. The model pairs are sorted in descending order according to the average human scores for each model.
Coherence
Consistency
Fluency
Relevance
ID
Spear.
Pear.
Kead.
Spear:
Pear.
Kend.
Spear.
Pear.
Kend.
Spear:
Pear.
Kead.
M8
0,420
0.383
0.323
0.229
0.273
0.209
0,274
0.245
0.236
0.519
0.509
0.438
M9
0.174
0.243
0.142
0.119
0.265
0.103
0,256
0.258
0.219
0.254
0.307
0.200
MLO
0.365
0.415
0.292
0.305
0,452
0,251
0.258
0.288
0.223
0.367
0.378
0.284
MI1
0.378
0.420
0,320
0.404
0.488
0.335
0,227
0.288
0.182
0.501
0.511
0.394
M12
0.208
0.237
0,160
0.087
0.020
0.082
0.086
0.107
0.071
0.438
0.451
0.354
M13
0.455
0.473
0,359
0.178
0.037
0.167
0.063
0.151
0.055
0.403
0.403
0.329
M14
0.433
0.467
0,355
0.114
0.187
0,105
-0.007
0.055
-0,005
0.421
0.435
0.336
ML5
0.372
0.385
0,279
0.189
0,316
0.177
0,100
0.087
0.085
0.252
0.288
0.193
M17
0.291
0.320
0.233
-0.086
-0.061
-0.084
0.017
0.046
0.015
0.204
0.199
0.175
M20
0.382
0.394
0.310
0.528
0.501
0.430
0.367
0.349
0.307
0.292
0.278
0,237
M22
0.215
0.293
0.162
-0.072
-0.052
-0.070
-0.114
-0.131
-0.101
0.201
0.300
0.172
M23
0.392
0.427
0,320
0.003
0.305
0.002
-0.078
-0.022
-0.069
0.223
0.324
0.184
Table 23: Spearman (Spear.) correlations, Pearson (Pear.) correlations, and Kendall's ' Tau (Kend.) between ChatGPT-RTS and human scores on the 100 summaries for each model. Values in light gray color are insignificant (p-value 2 0.05).
Coherence
Consistency
Fluency
Relevance
ID
Spear:
Pear.
Kend.
Spear.
Pear:
Kead.
Spear:
Pear.
Kend.
Spear:
Pear.
Kend.
M8
0.289
0.310
0.236
0.235
0.362
0.226
0.348
0.348
0.321
0.349
0.419
0.302
M9
0.170
0.170
0.148
0,211
0.321
0,200
0.198
0.299
0.174
0.318
0.356
0.276
M10
0.352
0.314
0.293
0.138
0,273
0.125
0.155
0.210
0.138
0.420
0.427
0.362
MI1
0.285
0.312
0.250
0,380
0,370
0.317
0,200
0,218
0.163
0.397
0.428
0.334
M12
0.306
0.304
0.258
-0.025
-0.059
-0.024
0.256
0.306
0.239
0.283
0.287
0.246
M13
0.425
0.425
0.351
0.471
0,312
0.457
0.199
0.214
0.186
0.435
0.402
0.375
M14
0.490
0.472
0.422
0.1
0.157
0.110
0.084
0.143
0.078
0.326
0.329
0.284
M15
0.317
0.298
0.250
0.061
0,513
0.060
0.055
0.025
0.050
0.433
0.420
0.378
M17
0.250
0.255
0.215
-0.106
-0.081
-0.105
-0.011
0.024
-0.011
0.293
0.285
0.260
M20
0.463
0.450
0.381
0.455
0.442
0.371
0.494
0.450
0.404
0.326
0.334
0.264
M22
0.211
0.173
082
-0.096
-0.080
-0.095
-0.092
-0.056
-0.087
0.352
0.371
0.313
M23
0.218
0.209
0. 189
0.107
0.448
0.104
.0,275
-0.269
-0.261
0.147
0.187
0.129
Table 24: Spearman (Spear.)
correlations. Pearson (Pear.) ChatGPT-MCQ and human scores on the 100 summaries for each model. Values in light gray color are insignificant (p-value 2 0.05).
correlations, and Kendall's Tau (Kend.) between
F Average ChatGPT scores
conservatively with RTS, and becomes more opti- mistic with MCQ.
We present the average ChatGPT evaluation scores for each model across all dimensions in Table 25. Generally, the same trend holds for the individual systems, that ChatGPT score systems much more

---

Coherenee
Consistency
Fluency
Relevance
Char-RTS
Char-MQQ GPT-4
Chai-RTS
Chat-MQQ GPT-
Chat-MCQ
Chur-RS
Chai-MQQ
M8
2.43
3.70
4.56
3.29
4.21
4,70
4.77
4,65
2,51
3.76
4,61
4.79
3.56
4.33
4.71
3.55
1.80
3.51
4.41
2.38
3.93
4.69
4.94
4.67
2.16
3.45
4.15
4.50
3.51
4.33
4.82
3.52
M10
2,05
3.49
4.05
2.73
3.77
4.52
4.60
4.25
2.23
3.43
3.42
4.42
3.45
4.25
4.62
3.36
M1l
1.70
2.63
2.93
2.28
2.36
4,11
3.72
3.27
1.71
2.66
2.77
3.65
2.92
4.06
3.87
3.15
M12
2.75
3,92
4.75
3.60
4.46
4.75
5,00
4,96
2.59
3.84
4,72
4,85
3.89
4,40
4.95
3.85
M13
2.91
3,99
4.76
3,44
4,40
4,73
4.89
4,82
2.69
3.93
4.69
4,86
3.90
4.41
4.85
3.83
3.63
M15
2,65
3.81
1.58
3.35
4.33
4.75
4,90
4.95
2.60
3.84
4,61
4.50
3.78
4.32
4,82
3.67
4.89
M20
2,00
2,95
2.99
3,63
2.66
3.57
3.51
3.40
2.22
3.02
3.05
3.97
2.75
3.78
3.47
3.30
3.54
4.09
4.92
4.18
4.83
4.97
4.94
3.57
4.06
4.94
4,90
4.41
4,43
4.25
M23
3.00
4.88
4.74
4.84
4.94
4,91
3.08
4.05
4,76
4.88
4.10
4.45
4.26
2,52
3,68
4.35
3.35
4.06
4,59
4.68
4,55
2.59
3.66
4,29
4.61
3.68
4.29
4.65
3.72
Table 25: Average scores for ChatGPT using RTS (Chat-RTS) and MCQ (Chat-MCQ). GPT-4 using RTS (GPT-4).
well as the human scores.
Dimension Generation
Dimension Generation
Consistency The summary inconsistent with the article as omits important details and repeats a phrase multiple times, resulting in score of
Coherence
The summary somewhat coherent but in- cludes incorrect information, such as stat- ing that Merson restarted his row with Burnley instead Andros Townsend, and does not flow smoothly; therefore, would give it Consistency The summary is consistent with the article as it mentions the helicopter airlifting patients the hospital, but leaves out important de- tails such as the number of people involved and their injuries. Final score:
Consistency
The summary includes some relevant informa- tion but lacks coherence and omits important details, resulting in a score of 2.5 out of for consistency.
of 3.
Fluency
The summary is well-written and grammat- ically correct, but lacks important details from the article such the potential link between antibiotics and weight gain, and the warnings about pre-prescciption of an- tibioties and the development of superbugs. Score: 3/5.
Consistency The summary accurately reflects the main points of the article, although it could have mentioned Falcao's unhappiness in France. Final score: 4.
Fluency
The summary lacks coherence and does not provide clear understanding the article, resulting in a score of for disfluency.
Fluency
The summary is coherent and grammatically correct, does not provide enough infor- mation about the incident. Final score: 3. The summary is only sentence and do not provide enough information about the article's content. Final score: 1.
Fluency
Table 26: Examples of wrong reasons generated during RTS by ChatGPT that do not correspond to the evalu- ated dimension. Bolded: reasons that don't match the evaluated dimension.
Table 27: Examples wrong reasoms generated during RTS by GPT-4 that don't correspond the evaluated dimension. Bolded: reasons that don't match the evalu- ated dimension.
Incorrect Reasons in RTS
We illustrate some of the observed incorrect rea- sons generated with the RTS method by ChatGPT that do not correspond to the evaluated dimension in Table 26. We further provide incorrect reasons generated by GPT-4 in Table 27.
H GPT-4 Evaluator
We also look into the reasoning of GPT-4 and dis- cover that makes the same mistakes as ChatGPT by penalizing the summary for reasons unrelated to the evaluated dimension (see Table 27).
We haven't 't compiled the exact number of mis- matched reasons for either LLM, but we do ob- serve non-trivial fraction of the evaluations con- taining murelated-to-dimensii reasoning during preliminary verification. For instance, the con- sistency dimension, ChaiGPT-RTS has 42% eval- reasoning
Another major difference is that GPT-4 tends to give overly generous scores. In one exception- ally extreme case, GPT-4 gives full scores for all generations by M12 in terms of consistency. Ta- ble 25 also shows the much higher average scores given by GPT-4 across all dimensions than those of ChaiGPT-RTS.
uations containing for the MI1 (avg. score 2.36) model, and 10% for the M17 model (avg. score of 4.84).

---

Cohereace
Consistency
Fluency
Relevance
ID
Spear.
Pear.
Kend.
Spear:
Peur:
Kend.
Spear.
Pear.
Kend.
Spear.
Pear:
Kend.
M8
0,429
0.446
0.362
0.449
0.597
0.433
0.412
0.409
0.385
0.425
0.489
0.365
0.288
0.346
0.243
0.349
0.350
0.331
0.407
0.413
0.490
0.358
0.495
0.403
0.579
0.360
0.510
0.537
0.429
MI1
0.584
0.572
0.474
0.529
0.536
0.448
0.436
0.358
0,500
0.519
0.405
M12
0.245
0.360
0.209
0.387
0.506
0.372
0.169
0.106
0.149
M13
0.271
0.271
0.230
0.535
0.444
0.518
0.093
0.337
0.089
0.237
0.337
0.206
M14
0.263
0.312
0.222
-0.040
-0.026
-0.040
0.311
0.432
0.291
0.277
0.315
0.241
M15
0.403
0.418
0.342
0.158
0.504
0.155
0.240
0.242
0.225
0.307
0,402
0.267
M17
0.285
0.240
0.246
0.390
0.748
0.385
0.091
0.131
0,088
0,210
0.253
0,185
M20
0.427
0.418
0.334
0.378
0.362
0.313
0.482
0.468
0.402
0.479
0.491
0.391
M22
0.026
0.004
0.023
0.346
0.608
0.343
0.248
0.141
0.242
0.1 0.143
0.084
0.126
M23
0.320
0.330
0.283
0.360
0. 168
0.354
0.267
0.136
0.251
0.005
0.003
0.005
Table 28: Spearman (Spear.) correlations, Pearson (Pear.) correlations, and Kendall's Tau (Kend.) between GPT-4
RTS and human scores on the 100 summaries for each model. Values in light gray color are insignificant (p-value 2 0.05). Note that for the consistency of M12, correlations cannot be calculated because GPT-4 gives scores to all examples.
Coherence
Consistency
Fluency
Relevance
ID
Spear.
Pear.
Kend.
Spcar.
Pear.
Kend.
Spear,
Pear:
Kend.
Spear.
Pear.
Kend.
M8
0.464
0.455
0.407
0.557
0.613
0.526
0.391
0.367
0.341
0,517
0.514
0.471
M9
0.436
0.415
0.416
0.215
0,342
0.195
0.578
0.571
0,517
0.450
0.485
0.412
M10
0.374
0.345
0.340
0,300
0,297
0,270
0.631
0.585
0,551
0,506
0.507
0.459
MI1
0.326
0.428
0.307
0.437
0,421
0,393
0.354
0.509
0.303
0.467
0.486
0.419
M12
0.500
0.469
0.444
0.446
0,486
0.424
0.273
0.320
0.241
0.588
0.591
0.542
M13
0.462
0.450
0.408
0.470
0,510
0.445
0.325
0.345
0,282
0,515
0.498
0.475
M14
0.524
0.460
0.463
0.325
0,293
0.307
0.216
0.257
0.194
0,501
0.476
0.455
M15
0.534
0.499
0.470
0.266
0,326
0.251
0.383
0.369
0.337
0.584
0.550
0.535
M17
0.312
0.294
0.274
-0.016
-0.008
-0.015
0.251
0.261
0.225
0.530
0.479
0.504
M20
0.636
0.623
0.566
0.733
0.664
0.644
0.566
0.556
0.492
0.570
0.565
0.496
M22
0.314
0.338
0.276
0.245
0.151
0.241
0.371
0.341
0.330
0.471
0.412
0.451
M23
0.380
0.386
0.337
0.158
0.496
0.153
0.296
0.275
0.259
0.450
0.505
0.421
Table 29; R, the reliability indicator calculated by the Spearman (Spear.) correlations, Pearson (Pear.) correlations. and Kendall's Tau (Kend.) between ChatGPT-RTS and ChaiGPT-MCQ. Values in light gray color are insignificant (p-value 2 0.05).


Reviews:
[{"review_id": "TTpRoaqamt", "reviewer": "Reviewer_LdaQ", "summary": "", "questions": "* Q1) (Line 328) If there were 3 expert scores per item, did you just use simple average of 3? \n  Any per-annotator normalization (such as z-score)?\n\n* Q2) In table 8, is it meaningful to add \\rho(P^{MCQ}, P^{RTS}) for a comparison?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and detailed review, with many constructive suggestions. Let us clarify and address the raised concerns as follows:\n\n* _We only use the __expert annotation portions__ of SummEval with __high__ inter-annotator agreement:_\n\nAlthough the __crowd-sourced annotator__ agreement is low, we use the **expert annotations** instead (as stated in line 328) with a kappa coefficient of 0.713 (discussed in the SummEval paper). The coefficient is super strong given the context of human evaluations. For instance, the highest kappa by [Chiang and Lee, 2023] was only 0.33. We thus use SummEval for its good annotation quality.\n\n* _We did not assume that human annotations are perfect and unbiased:_\n\nWe believe that **no** human or LLM alike is perfect or unbiased. Instead, we explore whether LLM evaluators are __aligned with human experts__, and may naturally exhibit the same bias as a human would. In other words, we examine whether we can reliably replace human annotators with LLMs.\n\n* _How to combine the 3 scores_\n\n‒ We used an average of the 3 expert scores, following several published works such as [Guan and Huang, 2020], [ChHun et al., 2022], and [Chiang and Lee, 2023]. \n\n‒ Following the reviewer's suggestion, we re-calculate the correlations using per-rate z-normalized human scores and observe the correlation for each model at each dimension exhibits __very little change__ (avg. &pm;0.005) from the results using non-normalized scores.  Furthermore, the meta-correlation for RTS with ChatGPT is __still significantly negative__ for consistency, fluency, and relevance at -0.797, -0.741, and -0.580. We thus suggest that z-normalization has __very little impact__ on SummEval human evaluation scores.\n\n* _Kendall Tau implementation_\n\nwe used the \"scipy.stats.kendalltau\" functions' default settings, which uses the tau-b variant that __accounts for ties__. We will add this detail to the main paper.\n\n* _Whether it is meaningful to add \\rho(P^{MCQ}, P^{RTS}) in Tab 8_\n\n‒ We believe it is not necessary to include  \\rho(P^{MCQ}, P^{RTS}), denoted as R_i.  We only intend to investigate __whether R_i is a good indicator of the performance__ of either RTS or MCQ, and we demonstrate so __by R_i's strong positive correlation__ with P^{MCQ} or P^{RTS} in certain dimensions. \n\n‒ However, one may be interested in finding a threshold value of R_i for relatively good reliability of either P^{MCQ} and P^{RTS}. We observe that generally 0.5 is a good threshold for SummEval, and will include a detailed table for paired R_i values with P^{MCQ} and P^{RTS} in the paper.\n\n* _Regarding suggestions for references:_\n\nWe acknowledge that some references are from arXiv, despite our best efforts to find their peer-reviewed versions. This is because the field of \"LLM-Evaluators\" is __relatively recent and actively growing__. Most of the works are in 2023 or the end of 2022 and do __not yet__ get enough chance to be published. We will carefully go through each reference again for updates and inclusion of URLs.\n\n* We would like to provide additional results using Llama-2 models, which are much weaker than ChatGPT and GPT-4, but also demonstrate dimensional and candidate dependences in terms of meta-correlation (__bolded__: p-value&leq;0.05)\n\n|   metrics   | Coherence | Consistency | Fluency | Relevance |\n| ----------- | ----------- | ----------- | ----------- | ----------- | \n| correlation (7B)  | 0.016  |  __0.203__ | __0.074__ | __0.080__ |\n| correlation  (13B)  | __0.223__  |  __0.202__ | __0.117__ | __0.246__ |\n| correlation  (70B)  | __0.186__  |  __0.319__ | __0.136__ | __0.200__ |\n| meta-correlation  (7B)    | 0.112 | __-0.678__| -0.371 | -0.245| \n| meta-correlation   (13B)   | -0.392 | -0.497 | -0.049 | -0.420 | \n| meta-correlation   (70B)   | -0.378 | __-0.804__ | -0.315 | -0.238 | \n\nReferences:\n\n[Chiang and Lee, 2023] Can Large Language Models Be an Alternative to Human Evaluation? (ACL 2023)\n\n[Guan and Huang, 2020] UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (EMNLP 2020)\n\n[ChHun et al., 2022] Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation (COLING 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Important problem/research question in the post-ChatGPT era.\nNice formulation of the problem and many contributions (See contributions). \nStraightforward process to tackle the research problem. \nSome useful analysis about when LLM as metric are reliable and when they are not (see Contributions).", "reasons_to_reject": "#R1) A hypothesis in this work is that “LLM should work well regardless of the quality of the evaluated systems” (Line 299) needs careful consideration. This assumes that human annotations are perfect. But in practice human annotations are also biased (SummEval dataset is know to have poor inter-annotator agreement, as per Fabri et al 2021). In similar tasks such as machine translation (see WMT metrics shared task), there has been normalizations to account for the variance in human scores, which attempt to fix it. This paper doesn't not provide details on the topic, but instead assumes human scores are perfect. \n\n#R2) This paper shows that LLM based metric becomes unreliable when the summarizer model quality becomes near human. When quality becomes near human, number of ties increase, and an importantly ranking correlation coefficients (such as the one used in this paper), does not properly reward ties.  There are many variants of Kendall Tau, attempting to handle ties in annotation, which would be worthwhile to explore (Authors have not disclosed what variant of Kendall's Tau formulation they used).", "questions_to_authors": "* Q1) (Line 328) If there were 3 expert scores per item, did you just use simple average of 3? \n  Any per-annotator normalization (such as z-score)?\n\n* Q2) In table 8, is it meaningful to add \\rho(P^{MCQ}, P^{RTS}) for a comparison?", "paper_topic_and_main_contributions": "Topic: Meta evaluation for (abstractive) summarization task. This paper tries to find the effectiveness of recent LLMs as a summarization metric (taking ChatGPT and GPT-4 as examples).\nPaper describes how to use ChatGPT as a metric (in 3 variants), and tries to find strengths and weaknesses of them in comparison to alternative metrics: ROUGE (3 variants), BERTScore (1variant), BARTScore(3 variants). Findings are based on SummEval benchmark dataset containing human annotations for a bunch of abstractive summarization models. They show that ChatGPT outperforms all others. Then they ask “can we replace human annotators with ChatGPT”. Answer: No (or not yet). Then, what about GPT-4? Answer: Not yet. \n\n*Contributions:*\n1. Describes in detail how to use instruction finetuned LLMs such as ChatGPT as a metric. They formulate 3 flavors: reason-then-score (RTS), multiple-choice-question(MCQ), head-to-head(H2H) \n1. Meta evaluation of metrics: authors show the strengths and weaknesses of ChatGPT variants and how they stack up against Rouge, BERTScore, BartScore. They show which flavor is the winner (spoiler: ChatGPT-RTS).\n1. Then stability of LLM Evaluators: They define/propose a meta-correlation metric. Finding: they show ChatGPT become unreliable (i.e, hard time ranking summaries) \n   1. when summarizatiion system qualities are too close (as per human scores). \n   1. when summary qualities reach highest (correlation approach zero, indicating randomness)\n1. They propose a method to detect when ChatGPT metric becomes unreliable so as to invoke human annotators as a fall back", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Presentation improvements: Tables appear to have too many numbers than needed to make a point. Given Kendall’s Tau is capturing ranking correlation, maybe we can drop Spearman coefficient? You may present all those numbers on the web. Also, some of the tables in Appendix maybe presented in website/gitrepo instead of appendix. \n\n* Bibliography: A lot of references are from arXiv. Please reference peer-reviewed versions of the papers instead of drafts. Also, note that a lot of references are missing URLs (which can be easily spot blue color of paper title).\n* Appendix: a lot of white space. Consider reordering tables and sections to minimize whitespace. \n* Appendix: Tables 22+ should have same order of rows as in Table21 which ordered them by decreasing order of human performance."}}, {"review_id": "3nmYC7kErz", "reviewer": "Reviewer_iAX6", "summary": "", "questions": "a. Have you tried other datasets, and are the results on other datasets such as *newsroom* consistent with the conclusions of this paper?\n\nb. Have you tried other LLMs, and are the results on other LLMs such as *LLaMA* consistent with the conclusions of this paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review. Let us clarify and address the raised concerns as follows:\n\n* _Experiments on more datasets_\n\nWe believe there is no other sufficiently large dataset that __as reliable and representative__ for abstractive summarization as SummEval. We feel NewsRoom has significant limitations:\n\n‒ *Not representative*: Annotations of only 4 abstractive systems (3 are the same model with different training data), only 60 examples, making it much more suspectable to out-liers\n\n‒ *Outdated*: All 4 systems used are __before 2018__ with mediocre performance (human scores range from 2.40 - 3.73, on par with the worst-performing candidates in SummEval), thus not representative of the stronger recent summarization models. \n\n‒ *Low quality*: The human evaluation of NewsRoom is done by __crowed-sourced annotators__, which are discovered to be questionable by SummEval as well as [Karpinska et al., 2021]\n\nWe have run ChatGPT (gpt-3.5-turbo-0613) with NewsRoom, but observe very low correlations (Kendall tau) as detailed below ( __bold__ for p < 0.05):\n|   metrics                   | Coherence | Consistency | Fluency | Relevance |\n| -----------                 | -----------    | ----------- | ----------- | ----------- | \n| BARTScore              | __0.324__   | __0.431__  | __0.248__   | __0.422__ |     \n|  ChatGPT  |      0.012     | __0.296__ | __0.122__ | __0.208__ |\n\nBroken down to individual systems, there is almost no correlation except for pointer n, which is trained on the full NewsRoom dataset.\n|   metrics                   | Coherence | Consistency | Fluency | Relevance |\n| -----------                 | -----------    | ----------- | ----------- | ----------- | \n| abstractive               |     0.029      |    -0.106    |    0.032   |   -0.019    |     \n|  pointer c                  |      -0.101     |   0.020     |    0.109   |    0.019    |\n|  pointer n                  |      -0.047     | __0.346__ | __0.263__ |   0.134  |\n|  pointer s                  |      -0.041     |  -0.070      |   -0.013   |  -0.016   |\n\nUpon further analysis, the majority (83% on coherence, 70% on consistency, 75% on fluency, and 67% on relevance) of the summaries receive a score of 1.  We thus believe the overly mediocre candidates have led to low correlation scores by ChatGPT.\n\n* _Experiments on open-sourced LLMs_\n\nAs discussed in our limitations (lines 618-620), our paper only seeks to explore __the best potential__ of LLMs as human-replacements, and thus select __the strongest models__ (i.e. ChatGPT and GPT-4 [OpenAI, 2023]). \nNevertheless, we show results on all three sizes of \"Llama-2-chat\" models below in terms of Kendall tau (__bold__ indicates p < 0.05). \n\n|   metrics   | Coherence | Consistency | Fluency | Relevance |\n| ----------- | ----------- | ----------- | ----------- | ----------- | \n| correlation (7B)  | 0.016  |  __0.203__ | __0.074__ | __0.080__ |\n| correlation  (13B)  | __0.223__  |  __0.202__ | __0.117__ | __0.246__ |\n| correlation  (70B)  | __0.186__  |  __0.319__ | __0.136__ | __0.200__ |\n| meta-correlation  (7B)    | 0.112 | __-0.678__| -0.371 | -0.245| \n| meta-correlation   (13B)   | -0.392 | -0.497 | -0.049 | -0.420 | \n| meta-correlation   (70B)   | -0.378 | __-0.804__ | -0.315 | -0.238 | \n\nAs shown, the Llama-2 cannot even outperform BARTScores in terms of dataset-level correlations. However, they also exhibit dimensional and candidate dependencies, which is __consistent__ with the conclusions of this paper.\n\nReferences: \n\n[Karpinska et al., 2021] The perils of using Mechanical Turk to evaluate open-ended text generation. (EMNLP 2021)\n\n[OpenAI, 2023] GPT-4 Technical Report", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments in this paper were conducted in great detail, accompanied by experimental results, case studies, and analysis of each viewpoint.\n\n2. The experimental results and conclusions of the paper will be of help for researchers/users to choose proper and efficient evaluators for abstractive summarization.\n\n3. The paper provides constructive opinions about building a temporary efficient framework as an evaluation pipeline to tell when LLM evaluators have become unreliable and employ further human evaluation when necessary.", "reasons_to_reject": "1. The experimental results and conclusions are all based on one dataset *SummEval*. Therefore, these conclusions are not convincing and cannot be generalized to all abstractive summarization tasks. Of course, *SummEval* is basically the most challenging dataset for summary tasks.\n\n2. The paper only selects a commercialized LLM chatGPT to conduct experiments, and it is necessary to explore the performance of open source LLMs.", "questions_to_authors": "a. Have you tried other datasets, and are the results on other datasets such as *newsroom* consistent with the conclusions of this paper?\n\nb. Have you tried other LLMs, and are the results on other LLMs such as *LLaMA* consistent with the conclusions of this paper?", "paper_topic_and_main_contributions": "The paper conducts a comprehensive analysis to investigate the stability and reliability of LLMs as automatic evaluators for abstractive summarization and it has drawn many interesting conclusions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jYPQAYfqwe", "reviewer": "Reviewer_i7an", "summary": "", "questions": "- Question A: Why the extractive systems in the SummEval dataset are excluded? I do not think it is necessary.\n\n- Question B: What are the distributions of human scores for the examples corresponding to different systems and dimensions? For example, I want to see the statistics of the human scores for each system. This will affect my understanding of the LLMs' dependence on candidates and dimensions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review. Let us clarify and address the raised concerns as follows:\n\n* _Request for experiments on more datasets and LLMs_\n\nKindly refer to our response for Reviewer iAX6. We will include these results in the Appendix as they are __consistent__ with the conclusions of our paper.\n\n* _Concerns on the \"conclusion\" that RTS penalizes summaries according to false premises_\n\nThis is actually not a conclusion but __one plausible reason__ why RTS is giving consistently lower scores than MCQ (lines 486-487, Appendix Table 23). We haven't tabulated the results, but we do observe a __non-trivial fraction__ of the annotations with unrelated-to-dimension reasoning. For instance, on the consistency dimension, M11 (scored 2.36) has __42%__ reasonings containing dimension-irrelevant reasonings, and even M17 with a top score of 4.84 has __10%__ such reasonings.\n\n* _Whether previous metrics and human annotations are candidate and dimension dependent_\n\nAs we define candidate-dependence and dimension-dependence as whether the metric exhibits consistent degrees of __human alignment__, it doesn't apply to human annotations which are __used as the reference__. We provide the meta-correlations (Kendall tau) for all evaluated metrics in the table below (__bolded__: p < 0.05). \n\n|   metric    | Coherence | Consistency | Fluency | Relevance |\n| ----------- | ----------- | ----------- | ----------- | ----------- | \n| Rouge 1                        | 0.014  | -0.455 | -0.420 | -0.329 |\n| Rouge 2                        | __0.615__ | -0.217 | -0.203 | -0.084 |\n| Rouge L                        | 0.259  | -0.510 | -0.203 | -0.182 |\n| BERTScore.                  | -0.315 | __-0.580__| -0.455 | __-0.643__|\n| BARTScore                   |__-0.916__| -0.161 | 0.154  | __-0.769__|\n| BARTScore-CNN          |__-0.797__| -0.678 | -0.510 | __-0.825__|\n| BARTScore-CNN-PARA|__-0.769__| -0.692 | -0.231| __-0.853__ |\n| ChatGPT-RTS                | -0.189 | __-0.797__ | __-0.748__ | -0.559 |\n| GPT-4-RTS                    | -0.497 |  -0.364 | __-0.692__  | __-0.958__  |\n \n\nRouge doesn't exhibit significant negative meta-correlation, whereas the neural metrics show significant meta-correlations for some dimensions, although varying from metric to metric. Nevertheless, although the previous neural metrics may equally suffer from dimensional and candidate dependencies, they are __frequently complemented with human evaluation__; however, one may be tempted to replace human evaluation with LLM Evaluators given the various recent strong results. Thus, our goal is to *raise awareness of the potential unreliability* of LLM evaluations out-of-the-box.\n\n* _Concerns on the reliability score, r_\n\n‒ The score, r, is the correlation of P^{RTS} and P^{MCQ}. We believe the reviewer is asking if it is hard to know to what extent r indicates the LLM scores are reliable? For SummEval our observed threshold is around 0.5. Admittedly, the threshold is likely to be dataset-dependent, so we acknowledge the reviewer's concern about the limitation of this solution. \n\n‒ Nevertheless, our main focus is to demonstrate that ChatGPT and GPT-4 are not yet human-level evaluators, and this mitigation method is only a __temporary solution__ that we explore. We will leave designing better metrics to future work (such as how to effectively utilize LLM evaluators).\n\n* _Concerns on the exclusion of extractive systems_\n\nOur focus is on abstractive summarization because it poses stronger challenges for evaluation due to its much wider answer space and open nature (lines 41-56). For instance, the extractive systems achieve close to full marks on consistency and fluency [Fabbri et al., 2021]. Thus, we only test the LLM evaluators on the relevant systems.\n\n* _Distribution of human scores for different systems and dimensions_\n\nWe show the average scores for each system and dimension in Appendix Tab 23 (briefly referred to in lines 476-477). The SummEval paper presents the standard deviation for each dimension in Fig. 2 (rightmost figure), and we also provide a detailed per-model breakdown in the table below. Given the minimum step size for annotation is 1, these values are considered very small, indicating substantial annotator agreements.\n\n|   metric    | Coherence | Consistency | Fluency | Relevance |\n| ----------- | ----------- | ----------- | ----------- | ----------- | \n|M8 | 0.58 | 0.09 | 0.13 | 0.51|\n|M9 | 0.65 | 0.15 | 0.34 | 0.54|\n|M10 | 0.58 | 0.22 | 0.20 | 0.54|\n|M11 | 0.59 | 0.34 | 0.40 | 0.49|\n|M12 | 0.65 | 0.03 | 0.14 | 0.54|\n|M13 | 0.62 | 0.07 | 0.14 | 0.51|\n|M14 | 0.65 | 0.09 | 0.18 | 0.54|\n|M15 | 0.60 | 0.04 | 0.13 | 0.53|\n|M17 | 0.58 | 0.05 | 0.08 | 0.52|\n|M20 | 0.48 | 0.29 | 0.24 | 0.49|\n|M22 | 0.58 | 0.02 | 0.14 | 0.54|\n|M23 | 0.58 | 0.05 | 0.12 | 0.54| \n\nReferences:\n\n[Fabbri et al., 2021] SummEval: Re-evaluating Summarization Evaluation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper keeps pace with the newest advances in NLG evaluation, which leverages LLMs for human-like evaluation. Different from the studies that test LLMs on common datasets and claim their superiority, a more in-depth analysis is presented, and some limitations of LLMs-based evaluation are revealed.\n\n- A meta-correlation metric is defined to measure the degree to which the performance of LLMs-based evaluators is affected by the quality of the evaluated systems, which can be used to further investigate the bias when using LLMs-based evaluators.", "reasons_to_reject": "- This paper only uses one dataset for the experiment. More experiments on other datasets and LLMs are needed to support the title of this paper.\n\n- The authors argue that ChatGPT-RTS generates false or unrelated-to-dimension reasoning and thereby penalizes the summaries according to such false premise. Though several cases are shown in the appendix, it is insufficient to draw such a conclusion. I would like to see a more detailed analysis of this problem and more human examinations of the correctness of LLMs' reasoning may be required.\n\n- The paper does not tell us the extent to which the commonly used evaluation metrics and human annotators are candidate-dependant and dimension-dependant. Without these results, it is hard to comprehensively understand the limitations of LLM evaluators from a relative perspective. In other words, LLMs have these limitations, what about the previous evaluation metrics and human evaluators?\n\n- The proposed efficient framework lacks experimental results and is hard to use in practice. It will be very hard for researchers to determine the reliability score $r$. This framework also seems to have the limitations of LLM evaluators stated in this paper.", "questions_to_authors": "- Question A: Why the extractive systems in the SummEval dataset are excluded? I do not think it is necessary.\n\n- Question B: What are the distributions of human scores for the examples corresponding to different systems and dimensions? For example, I want to see the statistics of the human scores for each system. This will affect my understanding of the LLMs' dependence on candidates and dimensions.", "paper_topic_and_main_contributions": "This paper tests the evaluation ability of ChatGPT and GPT-4 on the SummEval dataset with different assessment methods and prompt designs (RTS, MCQ, and H2H). The analysis shows ChatGPT correlates with human judgments better than commonly used evaluation metrics but they may not be reliable because there are some drawbacks, such as that they are candidate-dependant and dimension-dependant. Finally, a temporary efficient framework that uses the correlation between the results of RTS and MCQ as an indicator is proposed.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n\n- line 114, line 457: \"human judgment\" $\\rightarrow$ \"human judgments\"\n\n- line 602: \"are\" $\\rightarrow$ \"is\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper evaluates recent LLMs (Language Model Metrics) as summarization metrics, using ChatGPT and GPT-4. It provides an extensive exploration of ChatGPT's capabilities as a metric and juxtaposes its strengths and weaknesses against alternative metrics such as ROUGE, BERTScore, and BARTScore. These evaluations are grounded in the SummEval benchmark dataset, which includes human annotations for various summarization models.

While some reviewers expressed concerns about the use of a single dataset in the experiments, I believe it's important to note that this limitation is not a valid reason for rejection. This limitation arises due to the lack of alternative high-quality datasets for meta-evaluation in abstraction summarization.

However, reviewers did raise legitimate concerns regarding missing experiments involving other (commercial) LLMs, the need for further human examination of LLM reasoning correctness, and practical guidance on utilizing the efficient framework. The authors have provided additional results in their rebuttals, which should be incorporated into the paper along with a comprehensive discussion to address these concerns.
Decision: Accept-Findings