LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores (2024)

Abstract: Automatic evaluation of generated textual content presents an ongoing challenge within the field of NLP. Given the impressive capabilities of modern language models (LMs) across diverse NLP tasks, there is a growing trend to employ these models in creating innovative evaluation metrics for automated assessment of generation tasks. This paper investigates a pivotal question: Do language model-driven evaluation metrics inherently exhibit bias favoring texts generated by the same underlying language model? Specifically, we assess whether prominent LM-based evaluation metrics (e.g. BARTScore, T5Score, and GPTScore) demonstrate a favorable bias toward their respective underlying LMs in the context of summarization tasks. Our findings unveil a latent bias, particularly pronounced when such evaluation metrics are used in a reference-free manner without leveraging gold summaries. These results underscore that assessments provided by generative evaluation models can be influenced by factors beyond the inherent text quality, highlighting the necessity of developing more reliable evaluation protocols in the future.

heading: 1 Introduction
text: Evaluation is a fundamental element in both tracking progress and ensuring meaningful advancements across various dimensions within the field of Natural Language Processing. Therefore, the reliability of evaluation metrics plays a critical role in this process. Evaluating generated texts is one of the challenging and open problems in NLP given that different forms can convey the same meaning. This challenge has led to the development of various evaluation metrics for tasks involving Natural Language Generation (NLG). While human evaluation by experts stands as the most reliable approach for assessing generated outputs, it is costly and time-consuming, limiting its broader use. As a result, automatic evaluation metrics have emerged
as practical alternatives to keep pace with the rapid progress in NLP (van der Lee et al., 2019). Recent evaluation metrics for generation tasks, such as BERTScore (Zhang et al., 2020), BARTScore (Yuan et al., 2021), T5Score (Qin et al., 2022), GPTScore (Fu et al., 2023), and G-Eval (Liu et al., 2023), increasingly rely on pretrained language models. However, this trend poses a paradox, as the very outputs being evaluated are generated by these pretrained language models, raising concerns about inherent biases. For instance, an evaluation metric based on the BART model might yield inflated scores for outputs produced by a BART-based language model.
In this paper, we systematically investigate this potential bias, utilizing six prominent language models, namely BART (Lewis et al., 2020), T5 (Raffel et al., 2020), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), FLAN-T5 (Chung et al., 2022), and Cohere along with their corresponding evaluation metrics (e.g. BARTScore, T5Score, and GPTScore) or conditional generative probability, for the task of summarization, which is a typical task in natural language generations and frequently employed in automatic text evaluation. Our analysis involved examining numerous variations of these six families of generative mod-
12688
els, considering their varying sizes and finetuning settings both as generators and evaluators.
We conducted our analysis using the CNN/Daily Mail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. The assessment covers two settings: reference-based, using gold summaries for evaluation (a common approach in supervised summarization), and reference-free, comparing generated summaries against source documents (a common approach in both unsupervised summarization and factuality assessment).
Based on our analysis, we have derived the following findings: (1) Generative evaluators tend to assign higher scores to the content generated by the same underlying model. This bias becomes more pronounced when the fine-tuning configuration and model size match for both the generator and evaluator. (2) Inflated scores are particularly noticeable in the reference-free setting, which is concerning due to the popularity of this evaluation approach for assessing the factual correctness of generated texts (Koh et al., 2022). (3) Apart from self-bias, inflated scores are also influenced by the preference for longer summaries by certain evaluators.
Our work has implications for model selection, evaluation strategies, and the development of more reliable and unbiased evaluation metrics in the field of natural language generation.

heading: 2 Related Work

heading: Reference-based Evaluation Metrics
text: Reference-based metrics are commonly used to evaluate text generation tasks, including summarization, by measuring the similarity between generated and reference texts. Traditionally, metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) were employed to assess a generated text based on surface-level similarities, measured through the n-gram overlap between the generated and reference texts.
Recent trends in summarization evaluation lean towards semantic-level assessments, moving beyond direct word overlap comparisons. Notable metrics embracing this approach include BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), BARTScore (Yuan et al., 2021), BLEURT (Sellam et al., 2020), and variations thereof. By leveraging pretrained language models, these metrics focus on capturing semantic content, providing a more nuanced and accurate evaluation
of summarization system outputs.
Reference-free Evaluation Metrics With the widespread use of generation models across diverse domains, the need for reference-free evaluation metrics has surged. In response to this challenge, recent attention has been directed towards metrics that enable the evaluation of generated texts solely based on source documents, especially when annotated reference texts may not be available for new domains (Böhm et al., 2019; Gao et al., 2020; Wu et al., 2020; Chen et al., 2021; Scialom et al., 2021; Honovich et al., 2021; Zhong et al., 2022; Liu et al., 2023).
Representative reference-free metrics in recent years include generative evaluation models, exemplified by BARTScore (Yuan et al., 2021) and GPTScore (Fu et al., 2023), which are also used for reference-based evaluation. These metrics frame text evaluation as a natural language generation task, intuitively assigning higher probabilities to higher-quality generated texts. For instance, a recent study by Koh et al. (2022) has acknowledged BARTScore in reference-free mode as the factual consistency metric with the highest overall correlation to human factual consistency scores, particularly in the context of long document abstractive summarization. Therefore, the reliability of these metrics is important given their use for evaluating sensitive aspects such as factuality correctness.
Automatic Evaluation Metrics Pitfalls Despite their widespread use, automatic evaluation metrics have notable shortcomings. These metrics may not be robust when faced with challenges such as spurious correlations, noise, or out-of-domain texts (Sai et al., 2021; Vu et al., 2022; Durmus et al., 2022; Zhao et al., 2023; He et al., 2023). Furthermore, their effectiveness diminishes when evaluating very long documents (Amplayo et al., 2022). There is also evidence suggesting a potential bias towards ranking extractive summaries higher than abstractive ones (Amplayo et al., 2022).
Traditional reference-based evaluation metrics such as ROUGE or BLEU have been criticized for their inability to measure content quality or capture syntactic errors (Reiter and Belz, 2009). Consequently, these traditional metrics often exhibit weak correlations with human judgements, demonstrating that they cannot accurately reflect the realworld performance of generation systems (Peyrard, 2019; Mathur et al., 2020). For example, they might assign high scores to outputs that are flu-
ent but meaningless and unfaithful, as long as many of the same words are used (Gehrmann et al., 2021). Although embedding-based metrics (e.g., BERTScore) show improved performance in similarity measurement, they are still inadequate for assessing the extent of shared information between two summaries, a crucial indicator of summary information quality (Deutsch and Roth, 2021).
Reference-free metrics, on the other hand, exhibit a bias towards outputs generated by models that are more similar to their own (Deutsch et al., 2022). To the best of our knowledge, this study represents the initial attempt to perform an exploration, which has not yet been undertaken systematically. Additionally, question-answering-based referencefree metrics for summarization evaluation are prone to inheriting errors within summaries (Kamoi et al., 2023).
Metrics based on Large Language Models, which are capable of conducting both referencebased and reference-free evaluations, typically demonstrate superior correlations with human quality judgements across diverse NLG tasks and evaluation dimensions (Deutsch et al., 2022). While prior work has reported that LLM-based metrics prefer LLM-generated text, raising a concern about the shortcomings of LLMs as evaluators (Liu et al., 2023), our work conducts a systematic evaluation to address a fundamental question: Do language model-driven evaluation metrics inherently display bias favouring texts generated by the same underlying language model? We explore this question across both reference-based and reference-free evaluations and for a range of different large language models.

heading: 3 Methodology
text: To investigate the impact of the model’s self-bias— determining whether a language model-based evaluator favours outputs generated by a similar language model—we conduct a comprehensive series of experiments involving both quantitative comparisons and qualitative analysis. Our quantitative comparisons involve using language models of varying sizes and finetuning configurations as both the evaluator and generator models. This structured approach enables us to systematically examine the potential bias across different LM configurations. Subsequently, we verify the results through qualitative analysis using a subset of models’ summaries that are accompanied by human evaluation to fur-
ther demonstrate that higher scores produced by evaluators as a result of self-bias do not necessarily correlate with higher quality generated outputs.

heading: 3.1 Evaluators
text: We describe the evaluation process as follows: given a source text s, a human written reference r, generate a hypothesis h, which can be represented as: y = f(h, a,S) (1) where h denotes hypothesis, a refers to the aspect to evaluate, and S denotes supplementary text (i.e., s or r) that is employed alongside evaluations in various settings (Fu et al., 2023). For instance, it could be the source text s in a reference-free scenario which assesses the summary based on the source article directly (Fabbri et al., 2021). Whereas in the reference-based paradigm, the evaluation considers semantic overlap between the generated hypothesis h (e.g. model generated summaries) and reference summaries r (Bhandari et al., 2020).
The evaluators (i.e. based on BART, T5, GPT model variants as well as Cohere) utilised in our study all share a conditional probability paradigm, which can generally be formulated as
Score(h|d, a,S) = m∑
t=1
wt log p(ht|h<t,S, θ). (2)
Here θ is the model parameter, d refers to the task description and wt denotes the weight of the token at position t, where previous works normally treat each token equally (Yuan et al., 2021; Fu et al., 2023). We provide further descriptions of each type of evaluator below.
BARTScore BARTScore (Yuan et al., 2021) introduced the generative evaluation approach treating text assessment as a generation task, employing probability of the text being generated by BARTbased models (Lewis et al., 2020) to assess the quality of text generated across various tasks such as machine translation, summarization, and datato-text.
T5Score T5Score (Qin et al., 2022) was proposed providing both generative and discriminative training strategies for assessing T5-variant models as the core of this generative evaluation paradigm1.
1In our work, the training process of T5Score models only involves generative training due to the unavailability of publicly accessible checkpoints trained in a discriminative manner.
The integration of dual training strategies enables more types of data to be incorporated into the metric. T5Score closely aligns with BARTScore in terms of evaluation framework. Thus, when only considering the generative training strategy, T5Score is analogous to BARTScore, but for the T5 model series.
GPTScore Leveraging generative models to conduct evaluation has been further advanced with various of more recent large language models (Fu et al., 2023), showing a great performance and covering a rich variety of aspects for comprehensive evaluations. With an understanding of natural language instructions, GPTScore (including GPT-X and FLAN-T5 models) can perform intricate and personalized assessments without additional training.
Cohere We additionally include Cohere, the more recent language model to enrich our assessments. The evaluation scores assigned by the model is calculated according to Eq. 2, aligned with BARTScore, T5Score, and GPTScore.

heading: 3.2 Generation Models
text: We analyze different variants of the BART, T5, GPT-2, GPT-3, FLAN-T5 and Cohere models, taking into account two different variables: the model size and the finetuning dataset. Regarding size, we consider small, base, medium, and large variations of each model, when available. For the finetuning dataset, we examine three distinct settings: (1) using the pretrained language model without finetuning on a summarization dataset, (2) finetuning on CNN, and (3) finetuning on XSUM. For instance, BART-Base-CNN represents a BART-base model that is finetuned on the CNN dataset. For each of the model types, we have used their corresponding standard prompts for the task of summarization.2
To ensure the reproducibility of our analysis, we exclusively employ publicly available checkpoints for the utilized models. Apart from the GPT3Curie model that is taken from the OpenAI API and generation model obtained from Cohere, the rest of the models are taken from the Hugging Face model hub3.
We use each of these generation models both
2More details about the corresponding summarization prompts are included in Appendix A.2.
3https://huggingface.co/models.
for generating the summaries4 as well as the underlying model for the LM-based evaluator. All the checkpoints used for generators and evaluators in our experiments can be found in the Appendix A (Table 4 and Table 5).

heading: 3.3 Datasets
text: We use documents from two well-established summarization datasets including CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016) and the extreme summarization (XSUM) dataset (Narayan et al., 2018).
For quantitative comparisons, we randomly selected 500 documents from each of these datasets. We provide these documents to each of the generation models to obtain their corresponding generated summaries. For qualitative analysis, we use the SummEval benchmark (Fabbri et al., 2021) and the RoSE benchmark (Liu et al., 2022). These benchmarks include summaries from various generation models, as well as human evaluations, enabling us to assess the quality of these summaries.
The SummEval benchmark contains summaries generated by various summarization models (i.e. BART, T5 and GPT2) for 100 articles from the CNN/DM test set, with each summary supplemented by human annotations. More specifically, SummEval incorporates human annotations by both expert and crowd-sourced human annotators, targeting dimensions of coherence, consistency, fluency, and relevance. Ratings are on a scale of 0 to 5, with higher values indicating better performance.
Similarly, RoSE contains summaries generated by recent generative models based on CNN/DM documents, accompanied by their corresponding human evaluations. We use 100 summaries from each of the BART and GPT-3 models from the ROSE benchmark. The RoSE benchmark proposed an assessment protocol termed “Atomic Content Units” (ACUs) (Liu et al., 2022). ACU score gauges quality of evaluated summaries based on whether the presence of single facts (i.e., atomic facts) from reference are included in the evaluated summaries. ACU score is calculated by ACU matching:
f(s,A) = |As||A| (3)
where A is a set of ACUs from gold summaries and As denotes the ACUs of candidate summary s.
4We use the zero-shot setting for the models that are not finetuned on summarization datasets.
The distribution of human scores in RoSE and SummEval are given in Table 1.

heading: 3.4 Quantitative Comparisons
text: We employ 20 language model-based evaluators for our experiments including six BARTSCORE evaluators (Yuan et al., 2021), seven T5SCORE evaluators (Qin et al., 2022), six GPTScore evaluators, and the Cohere evaluator.5
We assess the evaluators in two settings: (a) reference-free, where the metric evaluates the likelihood of the summary being generated from the source text, and (b) reference-based, where the generated summary is evaluated based on the reference summary.
Due to the nature of log probabilities, original scores from each evaluator is be negative, and a higher score indicates better quality according to the evaluator. When weights wt in Eq. 2 are treated equally, the evaluation protocols of BARTScore, T5Score, and GPTScore are all conditional probability paradigms. To ensure comparability among the scores provided by 20 distinguished evaluators, a uniform normalization process is applied to the scores generated by each evaluator. The normalization procedure standardizes the scores across a scale ranging from 0 to α 6 as formulated in Eq. 4, where Xi,j indicates scores evaluated by the j-th evaluator on summaries generated by the i-th generator.
Xnormi,j = α(Xi,j −miniXi,j) maxiXi,j −miniXi,j
(4)
In this context, a normalized score of α signifies the highest quality attributed by the evaluator, while a score of 0 indicates the lowest quality.
5Appendix A.2 contains more details about the evaluators. 6In our work, we set parameter α to 1.
As the length of the generated summary is a key factor influencing the evaluation results, we further analyse the impact of lengths for the content generated by the models along with the experiments. In this regard, we also compute the correlations between the length of the text and the scores assigned by evaluators to identify trends in evaluators’ preferences.

heading: 3.5 Qualitative Analysis
text: For qualitative analysis, we employ Spearman Correlations (Zar, 2014) and Kendall Correlations (Freedman et al., 2007), which respectively assess monotonic relationships and order associations between human evaluations and LM evaluator scores. They are common metrics for assessing correlations with human judgements.
For the SummEval dataset, we calculate the correlations for four aspects (i.e. Coherence, Consistency, Fluency and Relevance ), aligned with the reference-free input setting in the evaluation protocol as specified by Yuan et al. (2021). For the evaluations based on the RoSE benchmark, we use ACU annotations that are suited for reference-based summary salience evaluation. Therefore, we employ the correlation values obtained from the SummEval dataset for the reference-free setting and those from the RoSE benchmark for the reference-based setting.

heading: 4 Experimental Results

heading: 4.1 Quantitative Comparisons: Assessing Self-Bias in LM-Evaluators Towards Their Own Output
text: Figures 2 and 3 display heatmaps presenting evaluator scores for various summaries generated by different generators from CNN/DM documents in reference-free and reference-base settings, respectively. These scores are computed by averaging the individual scores of the selected 500 documents. In both heatmaps, we observe darker cells along the diagonal line, running from the top left to the bottom right. This indicates the potential evaluator bias towards their corresponding generator models i.e., self-bias. However, this bias is notably more pronounced in the reference-free setting, commonly used for factuality evaluation (Koh et al., 2022).
Furthermore, as shown in Figure 2, we note a distinct trend: T5-based generators, whether finetuned or not, tend to receive higher scores when assessed using different T5Score variations com-
pared to evaluations using BARTScore, GPTScore, or Cohere. This results in a concentrated dark rectangle at the heatmap’s centre. Similarly, we observe a parallel trend for BART-based generators, whether fine-tuned or not.
Meanwhile, evaluators tend to assign higher ranks to generators trained on the same dataset as themselves, rather than to those fine-tuned on different datasets (see Figure 6 in Appendix B.1). For example, when using T5 models fine-tuned on the XSUM dataset as evaluators, there is a noticeable preference for BART-XSUM generators over T5-vanilla models, even though the evaluations are performed for the CNN Daily dataset. We observe the same pattern on summaries generated based XSUM documents.

heading: 4.2 Bias towards Longer Summaries
text: Another notable pattern in Figure 2 is the high scores for the BART-based generators, indicated by both BARTScore variants and different GPTScores. To further investigate this phenomenon, we calculate the average length of summaries generated by each generator for each of the datasets. Notably, BART models and Cohere that have not been fine-tuned for summarization tend to produce the longest summaries on average. This is followed by the fine-tuned BART models on the CNN dataset.
Conversely, T5-based models score the summaries generated by Cohere low, as they tend to favour shorter summaries. A similar preference for short summaries can also be observed for evaluators finetuned on XSUM, which one-sentence summaries.
Subsequently, we computed the Spearman correlation between the scores under the reference-free setting given by each of our examined evaluators and the length of the corresponding summary. The results are presented in Figure 5. Based on these results, with the exception of evaluators fine-tuned on XSUM, BARTScore and GPTScore variants tend to assign higher scores to longer summaries. This observation explains the darker squares positioned in the top-right corner of Figure 2 for high values of GPTScore variants, highlighting their inclination to assign higher scores to BART and BART-CNN generators that produce longer summaries. It is worth noting that this correlation with summary length is prominent within the reference-free setting. We observe a similar but less obvious pattern in the reference-based evaluations, as shown in Figure 3.

heading: 4.3 Qualitative Analysis: Correlation of Self-Bias with Human Evaluation
text: To further verify the evaluators’ self-bias, we repeat the experiments from § 4.1 on summarization benchmarks that are accompanied by human eval-
uations. While the number of summaries in these benchmarks is limited compared to those in § 4.1, we can use the human annotations to verify that the inflated scores are not correlated with human evaluations.
Figure 4 shows the evaluation results for the SummEval and RoSE benchmarks for the
reference-free and reference-based setting, respectively. As mentioned, we use SummEval for the reference-free setting and RoSE for the referencebased setting with regard to the specific aspects of each of these benchmarks (Yuan et al., 2021). Overall, we observe a trend similar to that shown in Figure 2. For instance, the T5-base generator
receives higher scores from T5-based evaluators.7 Meanwhile, BART-based models receive higher scores from both BARTScore and GPTScore evaluators, instead of T5 evaluator.
Table 3 presents the Spearman and Kendall correlation values of SummEval in the reference-free setting, whereas the Spearman and Kendall correlation values of RoSE in the reference-based setting are given in Table 2.
Overall, we observe that none of the evaluators have a strong correlation with the human annotations on either of these benchmarks. Due to the limited size of the samples (i.e., 100 summaries from SummEval and 100 summaries from ROSE with human annotations, as described in §3.3) and the absence of many of our investigated generators in § 4.1, we cannot draw a conclusive conclusion from the correlation values. Nevertheless, these results demonstrate that none of these evaluators highly correlate with human annotations, and as observed in § 4.1, their inflated scores for their own underlying generator may contribute to this low correlation.

heading: 5 Conclusions
text: Based on experiments, we make the following conclusions: First, the popularity of generative evaluation metrics, such as BARTScore, is on the rise for evaluating the factual accuracy of generated content—a critical concern in modern generator models. However, our results reveal that this evaluation
7In SummEval, the T5 model is only ranked higher when evaluated with certain variants of the T5Score in the referencebased setting.
approach is susceptible to the self-bias, highlighting the need for more robust metrics to assess factual correctness reliably. Second, our analysis indicates that models fine-tuned on the XSUM dataset are not suitable for direct integration into evaluators due to their bias towards shorter summaries. The exception is their use for evaluating summaries aligned with XSUM-style content. Third, notably, similar to traditional evaluation metrics (Sun et al., 2019), contemporary evaluation metrics might also lean towards favoring longer summaries. This bias should be considered when interpreting and applying these metrics. Finally, our study uncovers the presence of the self-bias across all assessed evaluators. Consequently, we recommend avoiding the use of the same underlying model as the generator for assessment. Although the limited human evaluations for our examined models prevent definitive conclusions on selecting the best generative evaluator, our research charts a promising direction for designing more resilient and unbiased evaluation metrics.
In summary, our study identifies a new type of bias in generative evaluators encouraging future research in this direction for designing fairer evaluation metrics.

heading: Limitations
text: We note that our work has the following limitations. Firstly, our experiment has been focused on the summarization task. Expanding the evaluation to encompass a broader range of generation tasks would be highly beneficial. Secondly, conducting a larger-scale human evaluation would be advantageous, as our current experiments are constrained by the limited sample sizes from SummEval and
RoSE. Finally, incorporating additional generation models and evaluators in future work would further enrich the experiment.

heading: Ethics Statement
text: This paper raises no ethical concerns. The data and supplementary materials used in this study are open-sourced and widely employed in existing works.

heading: A Evaluation Setting

heading: A.1 Generator
text: Full details of the models (e.g. checkpoint, prompt setting) that we employed as generators are given in Table 4.

heading: A.2 Evaluator
text: Full details of the models that we employed as our evaluators are given in Table 5 (reference-free settings) and Table 6 (reference-based settings).

heading: B Evaluation Results

heading: B.1 Reference-free Setting
text: Results of XSUM Dataset in Reference-free setting are presented in Figure 6. Evaluation scores for RoSE and SummEval benchmarks under the reference-free setting are shown in Figure 4.
For the meta evaluation, Spearman and Kendall correlation values in the reference-free setting for SummEval benchmark are shown in Table 3.

heading: B.2 Reference-based Setting
text: Heatmap of evaluation result on CNN/DM dataset under reference-based setting is given by Figure 3
Evaluation scores for RoSE and SummEval benchmarks under the reference-based setting are illustrated by Figure 4.
For the meta evaluation, Spearman and Kendall correlation values in the reference-based setting for
RoSE benchmark are shown in Table 2.

heading: Future Work
text: Finally, incorporating additional generation models and evaluators in future work would further enrich the experiment