Venue: NAACL 2025

# Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation

Mingqi Gao , Xinyu Hu1 , Li Lin, Xiaojun Wan

Wangxuan Institute of Computer Technology, Peking University

{gaomingqi,huxinyu,wanxiaojun}@pku.edu.cn

efsort_lestu.pku.edu.cn

Equal contribution.

Footnote 1: footnotemark:

###### Abstract

The correlation between NLG automatic evaluation metrics and human evaluation is often regarded as a critical criterion for assessing the capability of an evaluation metric. However, different grouping methods and correlation coefficients result in various types of correlation measures used in meta-evaluation. In specific evaluation scenarios, prior work often directly follows conventional measure settings, but the characteristics and differences between these measures have not gotten sufficient attention. Therefore, this paper analyzes 12 common correlation measures using a large amount of real-world data from six widely-used NLG evaluation datasets and 32 evaluation metrics, revealing that different measures indeed impact the meta-evaluation results. Furthermore, we propose three perspectives that reflect the capability of meta-evaluation: **discriminative power**, **ranking consistency**, and **sensitivity to score granularity**. We find that the measure using global grouping and Pearson correlation coefficient exhibits the best performance in both discriminative power and ranking consistency. Besides, the measures using system-level grouping or Kendall correlation are the least sensitive to score granularity.

## 1 Introduction

Automatic evaluation metrics (e.g., BLEU Papineni et al., 2002) and BERTScore Zhang et al. (2020)) are widely used in natural language generation (NLG) evaluation to assess the quality of content generated by a system or model for a specific task. Furthermore, the evaluation of the performance of these evaluation metrics is referred to as meta-evaluation, which typically uses the correlation between the metrics and human evaluation as a crucial criterion, as human evaluation is generally considered the gold standard. However, the implementation of the correlation measure is not uniform because it involves two elements that have different possible selections: the grouping method of the evaluation scores (e.g., system level Bhandari et al. (2020) and input level Deutsch et al. (2021)) and the correlation coefficient (e.g., Pearson's \(r\) and Spearman's \(\rho\)).

Our relevant preliminary experiments show that different correlation measures can indeed lead to different meta-evaluation results. However, prior studies have rarely paid attention to the relationships and differences between different measures; instead, they often simply follow the conventional practices of related work or authoritative competitions on evaluation. Some studies even do not clearly describe the correlation measure they used, particularly in terms of grouping methods, let alone explain the reasons for selecting that measure. Moreover, the correlation measures used in some authoritative competitions are constantly changing. For instance, WMT22 Freitag et al. (2022) used segment-level correlations that include three different grouping methods, whereas WMT21 Freitag et al. (2021b) and WMT23 Freitag et al. (2023) only used one. These issues of non-transparency and inconsistency indicate that the correlation measure and meta-evaluation in NLG require more in-depth understanding.

On the other hand, large language models (LLMs) have been increasingly used in automatic evaluation, including both prompting proprietary LLMs for NLG evaluation Liu et al. (2023); Chiang and Lee (2023); Kocmi and Federmann (2023) and fine-tuned LLM evaluators Wang et al. (2023); Xu et al. (2023); Jiang et al. (2023); Li et al. (2023). Unlike traditional continuous evaluation metrics, LLM-based evaluators typically output discrete scores and can assess on different evaluation scales based on flexible requirements (e.g., 1-5, 0-100). This introduces more evaluation ties and varying degrees of the granularity or discretization of evaluation scores, which may affect the fairness of com

parisons in certain correlation measures (Deutsch et al., 2023), making the already confusing selection of correlation measures even more complex.

However, it is by no means easy to strictly determine whether a particular correlation measure is reasonable, as it depends on the specific scenario and research objectives. For example, Deutsch et al. (2023) believe that when using multidimensional quality metric (MQM) (Freitag et al., 2021a) for machine translation evaluation, fine-grained tied scores should be trusted and used for meta-evaluation, while traditional Kendall correlation coefficients cannot handle these situations and are therefore unsuitable. However, in coarser-grained evaluations (such as Likert scale ratings from 1 to 5), ties may not be as reliable, rendering the above conclusion invalid. Therefore, this paper focuses on the comparison and analysis of different correlation measures and their meta-evaluation capabilities, primarily around the following three and characteristic perspectives and research questions:

* **RQ1** (SS4.1): For **discriminative power**, which correlation measures can more effectively distinguish between pairs of automatic evaluation metrics?
* **RQ2** (SS4.2): For **ranking consistency**, which correlation measures can provide more stable rankings for a set of evaluation metrics?
* **RQ3** (SS4.3): For **granularity sensitivity**, which correlation measures can better handle different evaluation score granularity?

To achieve more comprehensive and realistic analyses, we collect six commonly-used NLG evaluation datasets, including 30 different subsets, and calculate and annotate the results of 32 different automatic evaluation metrics (including evaluators using LLMs such as GPT-4). Based on such large amounts of real data, we design specific testing algorithms to analyze the above three questions and summarize the corresponding conclusions. Our contributions are summed up as follows:

1. We point out the necessity for a more thorough understanding of correlation measures in NLG meta-evaluation and demonstrate that the different selections of measures do affect meta-evaluation results.
2. We propose three characteristics reflecting meta-evaluation capabilities as well as corresponding methods to analyze different correlation measures and experiment on a large amount of real-world data from six evaluation datasets and 32 evaluation metrics.
3. Our experimental results show that the measure using global grouping and Pearson correlation exhibits the best meta-evaluation capabilities. We hope our work can deepen the understanding of correlation measures, thereby clarifying their usages in future research1. Footnote 1: Our code and data will be available on [https://github.com/Kite99520/NLGCorrEval](https://github.com/Kite99520/NLGCorrEval).

## 2 Background

In NLG evaluation, we usually focus on the quality of the output generated by a system or model for a given task and input. For example, in automatic news summarization, the input is a news article, and the output is a summary. There are two ways to evaluate the output: human evaluation and automatic evaluation, both typically expressed as scores. Human evaluation scores are considered the gold standard, and their consistency with automatic evaluation metrics is used to assess and compare the performance of different metrics, which is calculated by the specific correlation measure. This process is referred to as meta-evaluation and can be formalized as follows:

Assume that there are \(N\) systems, \(\{s_{i}\}^{N}_{i=1}\) and \(M\) inputs, \(\{d_{j}\}^{M}_{j=1}\). Each system \(s_{i}\) generates an output \(h_{ij}\) for each input \(d_{j}\) and the human evaluation score for each output \(h_{ij}\) is denoted as \(z_{ij}\). These form a meta-evaluation dataset \(D=\{\{d_{j}\}^{M}_{j=1},\{(h_{ij},z_{ij})\}^{N,M}_{i=1,j=1}\}\). In most meta-evaluation datasets, \(N\ll M\), and generally, the range of \(N\) is a few to dozens, while the range of \(M\) is tens to thousands.

On the other hand, an automatic evaluation metric \(m\) typically requires the input, output, and other related optional content, such as the reference for evaluation. For each output \(h_{ij}\), the score given by the automatic evaluation metric is denoted as \(x_{ij}\). If there are \(K\) metrics to be evaluated in meta-evaluation, they are denoted as \(\{m_{k}\}^{K}_{\overline{k}=1}\), and their output scores are denoted as \(\{x^{k}_{ij}\}^{\overline{K}}_{k=1}\) (also denoted as matrices \(\{X_{k}\}^{K}_{k=1}\)).

The correlation \(C(X,Z)\) between \(\{x_{ij}\}^{N,M}_{i=1,j=1}\) (i.e., \(X\)) and \(\{z_{ij}\}^{N,M}_{i=1,j=1}\) (i.e., \(Z\)) is used to evaluate the quality of a certain evaluation metric \(m\). And there are multiple ways to measure this correlation, which can be divided into four categories 

based on the grouping method, where \(c\) denotes a specific correlation coefficient, commonly Pearson's \(r\), Spearman's \(\rho\), and Kendall's \(\tau\):

* **Global Level**: Flatten matrices of evaluation scores into vectors and calculate the correlation coefficient between two \(N\times M\)-dimensional vectors, \(C_{global}(X,Z)=c((x_{ij})^{N,M}_{i=1,j=1},(z_{ij})^{N,M}_{i=1,j=1})\).
* **Input Level**: For each input, calculate the correlation coefficient between two \(N\)-dimensional vectors, and then average the \(M\) correlation coefficients, \(C_{input}(X,Z)=\frac{1}{M}\sum_{j=1}^{M}c((x_{ij})^{N}_{i=1},(z_{ij})^{N}_{i=1})\).
* **Item Level2**: For each system, calculate the correlation coefficient between two \(M\)-dimensional vectors, and then average the \(N\) correlation coefficients, \(C_{item}(X,Z)=\frac{1}{N}\sum_{i=1}^{N}c((x_{ij})^{M}_{j=1},(z_{ij})^{M}_{j=1})\). Footnote 2: WMT22 (Freitag et al., 2022) used this grouping as one of segment-level grouping methods. And we rename it “Item level” to avoid confusion.
* **System Level3**: First average the scores of each system across \(M\) inputs, and then calculate the correlation coefficient between the two \(N\)-dimensional vectors, \(C_{system}(X,Z)=c((\frac{1}{M}\sum_{j=1}^{M}x_{ij})^{N}_{i=1},(\frac{1}{M} \sum_{j=1}^{M}z_{ij})^{N}_{i=1})\).

Footnote 3: From a completeness perspective, there is another grouping method similar to system level, which first averages the scores of each input across \(N\) systems, and then calculates the correlation coefficient between the two \(M\)-dimensional vectors. However, it may reflect the difficulty of inputs and has no significance in evaluation.

It can be seen that the correlation measure includes two parts: the grouping method and the correlation coefficient. Considering the common four grouping methods and three correlation coefficients4 mentioned above, there are \(4\times 3=12\) different correlation measures in total. For two evaluation metrics \(m_{1}\) and \(m_{2}\) and a specific correlation measure \(C\), it is generally considered that \(m_{1}\) outperforms \(m_{2}\) if \(C(X_{1},Z)>C(X_{2},Z)\). Furthermore, the ranking of \(K\) metrics \(\{m_{k}\}_{k=1}^{K}\) can also be determined by comparisons within \(C(X_{k},Z)^{K}_{k=1}\).

Footnote 4: All Kendall’s \(\tau\) in this paper refers to \(\tau_{b}\). See Appendix A for the full definitions of three correlation coefficients.

To verify whether different correlation measures would affect the results of meta-evaluation, we conduct preliminary experiments of 12 measures on common evaluation datasets. Specifically, for two correlation measures \(C_{1}\) and \(C_{2}\), as well as the evaluation metrics to be meta-evaluated \(\{m_{k}\}_{k=1}^{K}\), we calculate the consistency of metric rankings under the two measures using \(\tau((C_{1}(X_{k},Z))^{K}_{k=1},(C_{2}(X_{k},Z))^{K}_{k=1})\). Since we are concerned with rankings here, we choose Kendall's \(\tau\) as the correlation coefficient. The results on SummEval Fabri et al. (2021) are presented in Figure 1, with the details of data introduced in Section 3. Any two different correlation measures lead to different meta-evaluations of metric rankings, with greater inconsistency when different grouping methods are used. Similar results are observed on other datasets, which are presented in Figures 6-10 in the appendix, indicating that the selection of correlation measures is indeed important and requires further analysis.

## 3 Meta-Evaluation Data

For more comprehensive and realistic analyses on correlation measures, we collect six widely used evaluation datasets, as well as the results of 32 common automatic evaluation metrics on them. Our subsequent experiments on three research questions are based on this large-scale real evaluation data.

### Datasets

As shown in Table 1, we select and preprocess six common evaluation datasets from five typical NLG tasks: summarization, story generation, dialogue,

Figure 1: The consistency of evaluation metric rankings using different correlation measures on SummEval, calculated through Kendall’s correlation coefficient.

 

data-to-text, and translation. Due to the large volume of WMT23 data, we only use news domain data from ZH2EN. Following convention Guan et al. (2021); Mehri and Eskenazi (2020), we split the original datasets according to subsets and aspects, resulting in a total of 30 meta-evaluation datasets. Since our primary focus is on analyzing the general characteristics of correlation measures across different scenarios rather than on specific datasets, we label these datasets as D1-D30 for brevity and will summarize their overall performance in subsequent experiments. More details and the correspondence are shown in Table 4.

### Automatic Evaluation Metrics

We select 14 common non-LLM evaluation metrics, including string-based BLEU Papineni et al. (2002), ROUGE-(1,2,L)5Lin (2004), CHRF Popovic (2015), and model-based BERTScore-(p,r,f1) Zhang et al. (2020), MoverScore Zhao et al. (2019), BARTscore-(s-h, r-h, h-r) Yuan et al. (2021), BLEURT Sellam et al. (2020), and COMET Rei et al. (2020). For LLM-based evaluators, we employ 18 experimental settings to prompt proprietary LLMs to score outputs based on task descriptions and aspect definitions, resulting in 18 evaluation metrics: three different proprietary LLMs from OpenAI6 (gpt-3.5-turbo, gpt-4-turbo, gpt-4o); different prompting strategies of three evaluation scales (1-5, 1-10, 0-100); and two sampling settings (temperature T=0 and sampling once, temperature T=1 and sampling ten times with results averaged). In total, there are \(K=32\) automatic evaluation metrics, and more information including the detailed prompts and implementations are shown in Appendix B.

Footnote 5: ROUGE–(1,2,L) refers to three different variants of ROUGE. Similarly, BARTSCore-(s-h-r-h,h-r) refers to three different ways of using BARTScore, typically regarded as different automatic evaluation metrics.

Footnote 6: [https://openai.com/api/](https://openai.com/api/)

## 4 Analyses of Correlation Measures

To analyze and compare the meta-evaluation capabilities of different correlation measures, we consider three important perspectives: discriminative power, ranking consistency, and sensitivity to score granularity. In practice, correlation measures are mainly used to assess the performance of automatic evaluation metrics, with two core applications: comparing two metrics and ranking a set of metrics. For the former, the key lies in the discriminative power of the correlation measure, i.e., whether it can distinguish various metric pairs as effectively as possible. For the latter, the consistency of the correlation measure in ranking the same set of metrics is crucial, i.e., whether the ranking remains stable. In addition, recently emerging LLM-based evaluators feature a flexible and discrete scoring pattern similar to human evaluation, so the sensibility of correlation measures to varying levels of score granularity of the same metric should meet certain expectations. In the following subsections, we will design specific tests and conduct experimental analyses based on these three capability aspects to identify the characteristics of different correlation measures.

### Discriminative Power

In the fields of information retrieval Sakai (2013) and recommendation systems Anelli et al. (2019); Ashkan and Metzler (2019); Valcarce et al. (2020), discriminative power is widely used to evaluate evaluation measures. Inspired by this, we adapted this method to evaluate correlation measures in NLG meta-evaluation.

Specifically, for a given correlation measure, a meta-evaluation dataset (including human scores \(Z\)), and the scores of \(K\) automatic evaluation metrics on it \(\{X_{k}\}^{K}_{k=1}\), we obtain the two-sided p-value for each pair of automatic evaluation metrics through hypothesis testing. The smaller the p-value, the more likely we are to reject the null hypothesis

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Task** & **Name** & **\#Subsets** & **\#Aspects** & **\#Systems** & **\#Inputs** \\ \hline Summarization & SummEval Fabbri et al. (2021) & 1 & 4 & 16 & 100 \\ Translation & WMT23-ZH2EN-NEWS Freitag et al. (2023) & 1 & 1 & 16 & 376 \\ Story Generation & HANNA Chihun et al. (2022) & 1 & 6 & 6 & 60 \\ Story Generation & MANSGuan et al. (2021) & 2 & 1 & 5 & 200 \\ Dialogue & USR Mehiri and Eskénazi (2020) & 2 & 6 & 5 & 60 \\ Data-to-text & WebNLG2020 Castro Ferreira et al. (2020) & 1 & 5 & 16 & 178 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Information and statistics of different evaluation datasets.

 

that there is no difference in the correlation values between the two metric scores with human scores. Thus, a highly discriminative correlation measure should yield many very small p-values. For convenience of comparison, similar to Valcarce et al. (2020), we define the DP value as the average of p-values of all metric pairs, ranging from 0 to 1, with smaller values indicating stronger discriminative power of a correlation measure. Algorithm 1 shows the pseudocode for calculating the DP value. Regarding the hypothesis testing methods used here, we refer to previous work and employ the PermBoth algorithm proposed by Deutsch et al. (2021)7.

Footnote 7: We used the code from [https://nlpstats.readthedocs.io/en/latest/](https://nlpstats.readthedocs.io/en/latest/), with 1000 permutation samples.

```
Input:\(X_{1},\dots,X_{K},Z\in\mathbb{R}^{N\times M}\), \(T\in\mathbb{N}\), \(C\) Output: DP value \(v\gets 0\) \(n=K\times(K-1)/2\) for\(i\in\{1,\dots,K-1\}\)do for\(j\in\{i+1,\dots,K\}\)do \(p_{ij}\leftarrow\texttt{PermutationTest}(X_{i},X_{j},Z,T,C)\) \(v\gets v+p_{ij}\) endfor endfor return\(v/n\) functionPermutationTest\((X,Y,Z,T,C)\) \(q\gets 0\) \(\delta\gets C(X,Z)-C(Y,Z)\) for\(T\) iterations do \(X_{s},Y_{s}\leftarrow\text{empty $N\times M$ matrices}\) for\((i,j)\in\{1,\dots,N\}\times\{1,\dots,M\}\)do ifrandom Boolean is then \(X_{s}[i,j]\gets Y[i,j]\) \(Y_{s}[i,j]\gets X[i,j]\) else \(X_{s}[i,j]\gets X[i,j]\) \(Y_{s}[i,j]\gets Y[i,j]\) endif endfor \(\delta_{s}\gets C(X_{s},Z)-C(Y_{s},Z)\) if\(|\delta_{s}|>|\delta|\)then \(q\gets q+1\) endif endfor return\(q/T\) endfunction
```

**Algorithm 1** Discriminative Power

In addition to giving an overall value, we can also look more closely at the discriminative power of different correlation measures by p-values curves, presented in Figures 11-40 in the appendix. After obtaining the p-values for each pair of evaluation metrics, we sort them in descending order. With the number of evaluation metric pairs on the x-axis and the p-values on the y-axis, we can plot the p-value curves of different correlation measures on a meta-evaluation dataset. The closer the curve is to the coordinate axis, the stronger the discriminative power of the corresponding correlation measure. The DP value numerically equals the area enclosed by the p-value curve and the coordinate axis divided by the number of metric pairs.

Figure 2: DP values of different correlation measures on all meta-evaluation datasets using the permutation test, **the lower the better**. Each column “Dn” shows the result on one dataset, which corresponds to the original dataset as shown in Table 4. The first column presents the overall performance with the averaged results of all datasets.

 

TakeawaysThe overall discriminative power of different correlation measures can be summarized and ranked based on different grouping methods and correlation coefficients as follows:

* Grouping: Global > Input > Item > System
* Correlation Coefficient: Pearson's \(r\) > Spearman's \(\rho\) > Kendall's \(\tau\)

```
0:\(X_{1},\dots,X_{K},Z\in\mathbb{R}^{N\times M}\), \(T\in\mathbb{N}\) , \(C\).
0:RC value \(x_{i}^{*},z_{i}^{*}\gets\text{empty}~{}N\times M_{1}\) matrices \(X_{2}^{*},Z_{2}^{*}\leftarrow\text{empty}~{}N\times M_{2}\) matrices for\(i\in\{1,\dots,N\}\)do for\(j\in\{1,\dots,M_{1}\}\)do \(X_{1}^{i}[i,j]\gets X_{k}[i,D_{1}[j]]\) \(Z_{2}^{i}[i,j]\gets Z[i,D_{1}[j]]\) endfor for\(j\in\{1,\dots,M_{2}\}\)do \(X_{2}^{i}[i,j]\gets X_{k}[i,D_{2}[j]]\) \(Z_{2}^{i}[i,j]\gets Z[i,D_{2}[j]]\) endfor \(R_{1}[k]\gets C(X_{1}^{i},Z_{1}^{i})\) \(R_{2}[k]\gets C(X_{2}^{i},Z_{2}^{i})\) endfor \(\tau^{*}\leftarrow\tau(R_{1},R_{2})\) \(v\gets v+\tau^{*}\) endfor return\(v/T\)
```

**Algorithm 2** Ranking Consistency

### Ranking Consistency

Inspired by the evaluation of different evaluation measures of ordinal classification Sakai (2021a) and information retrieval Sakai (2021b), for a given correlation measure, we randomly split the human scores and evaluation metric outputs in half, derive the rankings of the evaluation metrics on the two halves, and calculate the similarity of the two rankings using Kendall's \(\tau\) as a measure of ranking consistency. We define the RC value as the mean results obtained from repeating this process \(T=1000\) times. Algorithm 2 presents the pseudocode for the calculation. Figure 3 and Table 2 shows the RC values of correlation measures across all meta-evaluation datasets, with the complete results presented in Table 7 in the appendix. According to Table 2, ranking consistency and discriminative power exhibit similar trends for correlation measures, meaning that correlation measures with high discriminative power generally also have high ranking consistency.

TakeawaysThe overall ranking consistency of different correlation measures can be summarized and ranked based on different grouping methods and correlation coefficients as follows:

* Grouping: Global > Input > Item > System
* Correlation Coefficient: Pearson's \(r\) > Spearman's \(\rho\) > Kendall's \(\tau\)

### Sensitivity to Score Granularity

We first introduce the concepts of **evaluation scale** and **score granularity**, which apply to both human

Figure 3: RC values of different correlation measures on all meta-evaluation datasets, **the higher the better**, with the representation of columns similar to Figure 2.

 

evaluation and emerging LLM-based evaluators. In practice, human evaluation scores are not continuous values like traditional metrics (e.g., BLEU, BERTScore). Evaluators typically select a discrete value as the evaluation score based on the scale (also called range) required by the evaluation guidelines (e.g., 1-5, 0-100). This scale is referred to as the **evaluation scale**. On the other hand, multiple evaluators are often involved to enhance the reliability of the evaluation, and the final human score is the average of the scores from multiple evaluators, leading to more diverse values. The combination of the chosen evaluation scale and the number of evaluators determines the number of all possible final scores, which is called **score granularity**. This reflects the degree of discretization in the evaluation and potential ties in the scores. For example, SummEval uses a 5-point Likert scale for human evaluation, and each sample is evaluated by three annotators, resulting in 13 possible averaged scores. Here, the evaluation scale is 1-5, and the score granularity is 13.

Moreover, LLMs with strong instruction-following capabilities like GPT-4 have been increasingly used in automatic evaluation, following the pattern of humans Kocmi and Federmann (2023); Chiang and Lee (2023). Therefore, their output scores are also discrete, and they can simulate multiple evaluators through repeated sampling, which also involves the evaluation scale and score granularity. We have presented statistics for these two quantities in human scores and LLM-based metric scores across different datasets in Tables 4 and 5 in the appendix. The differences in evaluation scales primarily affect the difficulty of evaluation and the ability of the evaluators, which are not the focus of our study. For example, prompting GPT-4-Turbo with the scale of 1-5 or 1-10 is often seen as two distinct evaluation metrics that possess different capabilities. Our main focus, instead, is the impact of score granularity differences on different correlation measures when the evaluation scale and metric remain the same, which can be experimented on real-world data and statistical simulation. We denote the score granularity of humans and metrics as \(G^{h}\) and \(G^{m}\), respectively, with the former depending on the datasets and the latter depending on the evaluation settings.

#### 4.3.1 Real-world Data

We select two typical datasets from Table 1 for experiments: SummEval and WMT23, where their \(G^{h}\) equals to 13 and 251, respectively. In addition,

Figure 4: As the changes of \(G^{m}\), the correlations between the GPT-4-Turbo evaluator and human evaluation using different measures on SummEval (left) and WMT23 (right) with the fixed evaluation scale of 1-5.

\begin{table}
\begin{tabular}{l l c c} \hline Grouping & Correlation & DP value \(\downarrow\) & RC value \(\uparrow\) \\ \hline System & Pearson & 0.136 (5) & 0.698 (7) \\ System & Spearman & 0.339 (11) & 0.485 (12) \\ System & Kendall & 0.381 (12) & 0.486 (11) \\ Input & Pearson & 0.128 (4) & 0.768 (3) \\ Input & Spearman & 0.150 (7) & 0.738 (6) \\ Input & Kendall & 0.149 (6) & 0.739 (5) \\ Global & Pearson & **0.107 (1)** & **0.774 (1)** \\ Global & Spearman & 0.118 (2) & 0.770 (2) \\ Global & Kendall & 0.125 (3) & 0.753 (4) \\ Item & Pearson & 0.168 (8) & 0.653 (8) \\ Item & Spearman & 0.180 (9) & 0.643 (9) \\ Item & Kendall & 0.192 (10) & 0.618 (10) \\ \hline \end{tabular}
\end{table}
Table 2: Overall performance and rankings (in brackets) of different correlation measures with the averaged results of all meta-evaluation datasets in terms of the discriminative power (DP) and ranking consistency (RC).

 

we use the three proprietary LLMs and two evaluation scales (1-5, 1-10) introduced in Section 3.2, with the temperature setting of T=1. The number of multiple samplings varies from 1 to 10 to obtain results for different levels of metric score granularity \(G^{m}\). Figure 4 shows the consistency of GPT-4-Turbo with human evaluations under different correlation measures, with other results included in Figures 41-50 in the appendix. We believe that when the score granularity of the metric \(G^{m}\) is no greater than that of human \(G^{h}\), the evaluation capability of the metric is limited, as fewer possible scores are available. Consequently, as \(G^{m}\) increases until it equals \(G^{h}\), the consistency between the metric and human should increase. However, when \(G^{m}\) exceeds \(G^{h}\), the situation becomes complex, and it is difficult to intuitively estimate the expected trend of the consistency, leaving for future explorations. The results illustrate that the measures using the system-level grouping or Kendall's \(\tau\) are basically not affected by the score granularity, where the consistency almost remains unchanged, not meeting expectations as mentioned above. The situation is generally similar for the two datasets with different LLM-based evaluators and different evaluation scales.

#### 4.3.2 Statistical Simulation

Although real-world datasets and evaluation metrics are more reliable for analysis compared to synthetic data, their scale of data is limited and some variables are less flexible to adjust. Therefore, we also employ the statistical simulation as an auxiliary method to observe whether similar conclusions can be drawn. Based on our collected datasets and certain hypotheses, we model the NLG meta-evaluation data and conduct experiments with parameters estimated from real data. The detailed process, including the sampling algorithm and experimental settings, is described in Appendix C. Figure 5, as well as Figure 51 in the appendix, show the results, which generally align with the observations obtained before.

TakeawaysThe influence of the score granularity on the measures using system-level grouping or Kendall correlation is the least among different grouping methods and correlation coefficients, which is not as expected. And as the score granularity of the metric increases, the consistency between metrics and humans under other correlation measures continuously increases.

## 5 Related Work

In the field of NLP, there is limited research analyzing correlation measures in NLG meta-evaluation. Mathur et al. (2020) found that the introduction of an outlier system can distort the system-level Pearson correlation between machine translation evaluation metrics and human evaluation and quantify the impact. Deutsch et al. (2022) proposed two different ways of calculating system-level correlations by changing the input sets or system pairs and showed they can lead to more precise estimates of metric performance in real-world scenarios. Recently, Deutsch et al. (2023) pointed out that the segment-level Kendall correlation coefficient, widely used in machine translation evaluation, does not handle ties in human scores and metric outputs as expected and thus needs to be calibrated. It is worth mentioning that Perrella et al. (2024) designed several sentinel metrics for machine translation meta-evaluation and showed that global-level grouping and system-level grouping may introduce unfairness.

In the research on automatic evaluation metrics, some works have commented on correlation measures based on experimental results when presenting the performance of different evaluation metrics. Owczarzak et al. (2012) found that, in the domain of summarization evaluation, system-level correlation is more robust to inconsistent human annotations. Freitag et al. (2022) discovered that system-level correlation is hard to distinguish between different machine translation evaluation metrics. Liu et al. (2023); Xu et al. (2023) explained some experimental results as the inappropriate handling of

Figure 5: As the changes of \(G^{m}\), the correlations between metrics and humans using different measures in statistical simulation with \(G^{h}=13\).

 

ties by the Kendall correlation when comparing the performance of different metrics. Additionally, Wei and Jia (2021) demonstrated the output scores of automatic evaluation metrics have less variance than human scores at system level.

In contrast, we focus on the properties and capabilities of typical correlation measures from generic NLG evaluation perspectives, not limited to specific tasks and evaluation metrics.

## 6 Conclusions

We analyze and compare the characteristics and capabilities of 12 typical correlation measures in NLG meta-evaluation through three proposed perspectives. Based on various experiments with large-scale real-word data from six NLG datasets and 32 evaluation metrics, we find measures using global-level grouping and Pearson correlation have better overall meta-evaluation capability, while those using system-level and Kendall correlation show the opposite. And they have all been used on some common evaluation datasets. We hope that our work can deepen the understanding of correlation measures and draw more attention and emphasis on related research in the future.

### Limitations

We mainly analyzed the capabilities of correlation measures through empirical experiments without conducting theoretical analysis. Besides, although our experiments have covered many common and typical evaluation datasets to provide as general analyses as possible, it is impossible to encompass all tasks and evaluation aspects in NLG. Therefore, the conclusions we obtained about the meta-evaluation capability of different correlation measures may be limited to a certain context. Additionally, our work requires quite a few resources, including the API cost of using proprietary LLMs to annotate data and high-performance computation for conducting large-scale empirical evaluations and statistical simulations, which could be improved in the future.

## Acknowledgments

This work was supported by Beijing Science and Technology Program (Z231100007423011) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.

## References

* Anelli et al. (2019) Vito Walter Anelli, Tommaso Di Noia, Eugenio Di Sciascio, Claudio Pomo, and Azzurra Ragone. 2019. On the discriminative power of hyper-parameters in cross-validation and how to choose them. In _Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, September 16-20, 2019_, pages 447-451. ACM. Ashkan and Metzler (2019) Azin Ashkan and Donald Metzler. 2019. Revisiting online personal search metrics with the user in mind. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_, pages 625-634. ACM. Bhandari, Guramatobak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 9347-9359. Association for Computational Linguistics.
* Ferreira et al. (2020) Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina, editors. 2020. _Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)_. Association for Computational Linguistics, Dublin, Ireland (Virtual).
* Chhun et al. (2022) Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloe Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In _Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022_, pages 5794-5836. International Committee on Computational Linguistics.
* Chiang and Lee (2023) David Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 15607-15631. Association for Computational Linguistics.
* Deutsch et al. (2021) Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. _Trans. Assoc. Comput. Linguistics_, 9:1132-1146.
* Deutsch et al. (2022) Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Resamining system-level correlations of automatic summarization evaluation metrics. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, 

WA, United States, July 10-15, 2022_, pages 6038-6052. Association for Computational Linguistics.
* Deutsch et al. (2023) Daniel Deutsch, George F. Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 12914-12929. Association for Computational Linguistics.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021. Summeval: Re-evaluating summarization evaluation. _Trans. Assoc. Comput. Linguistics_, 9:391-409.
* Freitag et al. (2021a) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. _Preprint_, arXiv:2104.14478.
* Freitag et al. (2023) Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George F. Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In _Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6-7, 2023_, pages 578-628. Association for Computational Linguistics.
* neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022_, pages 46-68. Association for Computational Linguistics.
* Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George F. Foster, Alon Lavie, and Ondrej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In _Proceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021_, pages 733-774. Association for Computational Linguistics.
* Guan et al. (2021) Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbaio Ding, Xiaoxai Mao, Changjie Fan, and Minlie Huang. 2021. Openmeva: A benchmark for evaluating open-ended story generation metrics. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 6394-6407. Association for Computational Linguistics.
* Hu et al. (2024) Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. 2024. Themis: Towards flexible and interpretable NLG evaluation. _CoRR_, abs/2406.18365.
* Jiang et al. (2023) Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2023. Tigerscore: Towards building explainable metric for all text generation tasks. _CoRR_, abs/2310.00752.
* KENDALL (1945) M. G. KENDALL. 1945. The treatment of ties in ranking problems. _Biometrika_, 33(3):239-251.
* Kocmi and Federmann (2023) Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, Tampere, Finland, 12-15 June 2023_, pages 193-203. European Association for Machine Translation.
* Li et al. (2023) Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023. Generative judge for evaluating alignment. _CoRR_, abs/2310.05470.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Cheng Guang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 2511-2522. Association for Computational Linguistics.
* Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: reevaluating the evaluation of automatic machine translation evaluation metrics. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 4984-4997. Association for Computational Linguistics.
* Mehri and Eskenazi (2020)Shikib Mehri and Maxine Eskenazi. 2020. USR: an unsupervised and reference free evaluation metric for dialog generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 681-707. Association for Computational Linguistics.
* Noreen (1989) Eric W Noreen. 1989. Computer intensive methods for hypothesis testing: An introduction. _Wiley, New York_, 19:21.
* Onoshima et al. (2019) Takahiro Onoshima, Kenpei Shiina, Takashi Ueda, and Saori Kubo. 2019. Decline of pearson'sr with categorization of variables: a large-scale simulation. _Behaviormetrika_, 46:389-399 

- Volume 2: Short Papers, pp. 359-362. External Links: Link Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. External Links: Link, Document Cited by: SS1.
* G. Perrella, L. Proietti, A. Scire, E. Barba, and R. Navigli (2024)Guardians of the machine translation meta-evaluation: sentinel metrics fall in!. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 16216-16244. External Links: Link, Document Cited by: SS1.
* M. Popovic (2015)chrf: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@EMNLP 2015, 17-18 September 2015, Lisbon, Portugal, pp. 392-395. External Links: Link, Document Cited by: SS1.
* R. Rei, C. Stewart, A. C. Farinha, and A. Lavie (2020)COMET: a neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 2685-2702. External Links: Link, Document Cited by: SS1.
* PROMISE Winter School 2013, Bressanone, Italy, February 4-8, 2013. Revised Tutorial Lectures, Lecture Notes in Computer Science, Vol. 8173, pp. 116-163. External Links: Link, Document Cited by: SS1.
* T. Sakai (2021)Evaluating evaluation measures for ordinal classification and ordinal quantification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 2759-2769. External Links: Link, Document Cited by: SS1.
* 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28
- April 1, 2021, Proceedings, Part I, J. Dy, pp. 12656. External Links: Link, Document Cited by: SS1.
* T. Sellam, D. Das, and A. P. Parikh (2020)BLEURT: learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7881-7892. External Links: Link, Document Cited by: SS1.
* C. Shen, L. Cheng, X. Nguyen, Y. You, and L. Bing (2023)Large language models are not yet human-level evaluators for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 4215-4233. External Links: Link, Document Cited by: SS1.
* C. Spearman (1904)The proof and measurement of association between two things. The American Journal of Psychology15 (1), pp. 72-101. External Links: ISSN 0003-021X, Link, Document Cited by: SS1.
* D. Valcarce, A. Bellogin, J. Parapar, and P. Castells (2020)Assessing ranking metrics in top-n recommendation. Inf. Retr. J.23 (4), pp. 411-448. External Links: Link, Document Cited by: SS1.
* Y. Wang, J. Jiang, M. Zhang, C. Li, Y. Liang, Q. Mei, and M. Bendersky (2023)Automated evaluation of personalized text generation using large language models. CoRRabs/2310.11593. External Links: Link, 2310.11593 Cited by: SS1.
* J. Tian-Zheng Wei and Robin Jia (2021)The statistical advantage of automatic NLG metrics at the system level. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Volsume 1: Long Papers, Virtual Event, August 1-6, 2021, pp. 6840-6854. External Links: Link, Document Cited by: SS1.
* W. Xu, D. Wang, L. Pan, Z. Song, M. Freitag, W. Wang, and L. Li (2023)INSTRUCTSCORE: towards explainable text generation evaluation with automatic feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 5967-5994. External Links: Link, Document Cited by: SS1.
* W. Yuan, G. Neubig, and P. Liu (2021)Bartscore: evaluating generated text as text generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 27263-2727. External Links: Link, Document Cited by: SS1.
* T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi (2020)BERTScore: evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link, Document Cited by: SS1.
* W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger (2019)MoverScore: text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in 

Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 563-578. Association for Computational Linguistics.

## Appendix A Definitions of Pearson's \(r\), Spearman's \(\rho\), and Kendall's \(\tau\)

Pearson correlation coefficient measures the linear relationship between two variables. The formula is:

\[r(x,y)=\frac{\sum_{i=1}^{N}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{N} (x_{i}-\bar{x})^{2}\sum_{i=1}^{N}(y_{i}-\bar{y})^{2}}}\]

Where \(x\) and \(y\) are n-dimensional vectors, and \(x_{i}\) and \(y_{i}\) are the elements of \(x\) and \(y\). \(\bar{x}=\sum_{i=1}^{n}x_{i}\) and \(\bar{y}=\sum_{i=1}^{n}y_{i}\).

Spearman correlation coefficient Spearman (1904) measures the rank-order correlation between two variables. The formula is:

\[\rho(x,y)=1-\frac{6\sum_{i=1}^{n}d_{i}^{2}}{n(n^{2}-1)}\]

Where \(d_{i}=\text{rank}(x_{i})-\text{rank}(y_{i})\) is the difference between the ranks of corresponding values in the two vectors.

Kendall correlation coefficient KENDALL (1945) measures the ordinal association between two variables. It considers concordant and discordant pairs, with \(\tau_{b}\) adjusted for ties. The formula is:

\[\tau_{b}(x,y)=\frac{C-D}{\sqrt{(C+D+T_{x})(C+D+T_{y})}}\]

Where \(C\) is the number of concordant pairs, i.e. pairs of elements \((i,j)\) where \((x_{i}>x_{j}\wedge y_{i}>y_{j})\vee(x_{i}<x_{j}\wedge y_{i}<y_{j})\). \(D\) is the number of discordant pairs, i.e. pairs of elements \((i,j)\) where \((x_{i}>x_{j}\wedge y_{i}<y_{j})\vee(x_{i}<x_{j}\wedge y_{i}>y_{j})\). \(T_{x}\) and \(T_{y}\) are the number of tied values in \(x\) and \(y\), respectively.

## Appendix B Details of Selected Evaluation Metrics

### Non-LLM evaluation metrics

For CHRF and BLEU, we use the implementation of TorchMetrics8. For ROUGE, BERTSCORE, and BLEURT, we use the evaluation package of Huggingface with the default parameters. For MoverScore9, BARTScore10, and COMET11, we use the code from the original GitHub repositories and the default models. We check the licenses of all open source programs to ensure that our use is compliant.

Footnote 8: [https://lightning.ai/docs/torchmetrics/stable/](https://lightning.ai/docs/torchmetrics/stable/)

Footnote 9: [https://github.com/APIHES/emlp19-moversore](https://github.com/APIHES/emlp19-moversore)

Footnote 10: [https://github.com/neulab/BARTscore](https://github.com/neulab/BARTscore)

### Evaluation Prompts for LLMs

We used the same prompts to instruct GPT-3.5, GPT-4, and GPT-4o for NLG evaluation. To save space, we present a template of our prompt in Table 3. We filled the aspect part of the prompt with definitions from original datasets. When the original dataset lacked these definitions, we composed them based on our understanding.

### Reasons not to use non-proprietary LLMs

We originally intended to include some open-source LLMs (e.g., Llama 3 Dubey et al. (2024)). However, we found that their adherence to "evaluation instructions" was not strong enough, often resulting in non-compliant responses. We also considered including some fine-tuned LLM evaluators (e.g., Themis Hu et al. (2024)). However, their limitations due to fine-tuning settings prevented us from requiring them to score with different evaluation scales via prompts. Therefore, we finally only chose proprietary LLMs in our experiments.

## Appendix C Process of Statistical Simulation

Based on the real-world data and some hypotheses, we first establish a probabilistic model for NLG meta-evaluation and then obtain results through the proposed algorithm and repeated sampling.

### Modeling NLG Meta-Evaluation

We posit that the capability of an evaluation metric is primarily reflected in two aspects: the ability to evaluate the overall level of different systems and the ability to evaluate different output texts from a given system. In practice, system-level correlation can estimate the former, while item-level correlation can estimate the latter. Therefore, we treat these two quantities as the control parameters during modeling. We assume that the scores of the evaluation metric and human evaluation here are continuous, and for a system \(s_{i}\), those of outputs it generates for various inputs 

follow a bivariate normal distribution: \(x_{ij},z_{ij}\sim\mathcal{N}(\mu^{m}_{i},\mu^{h}_{i},\sigma^{m}_{i},\sigma^{h}_{ i},\rho_{i})\), where \(\rho_{i}\)12 controls the correlation between metrics and humans within a single system. Based on our observations and those of Shen et al. (2023), this correlation varies across different systems for most evaluation metrics. For simplicity, we assume \(\rho_{i}\) follows a truncated normal distribution: \(\rho_{i}\sim\mathcal{N}(\mu_{\rho_{item}},\sigma_{\rho_{item}})\). Since item-level correlation is defined as the mean correlation coefficient across different systems, its value in existing datasets can be viewed as an estimate of \(\mu_{\rho_{item}}\). Furthermore, assuming \(\mu^{m}_{i}\) and \(\mu^{h}_{i}\) of the above bivariate normal distribution follow another bivariate normal distribution: \(\mu^{m}_{i},\mu^{h}_{i}\sim\mathcal{N}(\mu^{m},\mu^{h},\sigma^{m},\sigma^{h}, \rho_{sys})\), where \(\rho_{sys}\) controls the correlation between \(\mu^{m}_{i}\) and \(\mu^{h}_{i}\). And similarly, system-level correlation in real-world scenario can be seen as an estimate of \(\rho_{sys}\) because \(\frac{1}{M}\sum_{j=1}^{M}x_{ij}\) and \(\frac{1}{M}\sum_{j=1}^{M}z_{ij}\) in the definition of system-level correlation are viewed as estimates of \(\mu^{m}_{i}\) and \(\mu^{h}_{i}\).

Footnote 12: This \(\rho\) does not refer to the Spearman correlation coefficient, the same below.

Although the assumption of continuous scores makes it convenient to model and sample data, we need to analyze the impact of score granularity, which requires further data discretization. We then follow the practice of Onoshima et al. (2019) that has a similar situation to sample \(G^{h}-1\) and \(G^{m}-1\) thresholds from uniform distributions \(U(\mu^{m}-\sigma^{m},\mu^{m}+\sigma^{m})\) and \(U(\mu^{h}-\sigma^{h},\mu^{h}+\sigma^{h})\) to discretize them. The Algorithm 3 shows the pseudocode for the entire sampling process.

## Appendix B

\begin{table}
\begin{tabular}{p{456.7pt}} \hline \hline
**Prompts and Instructions** \\ \hline
**###Instruction\#f Please act as an impartial and helpful evaluator for natural language generation (NLG), and the audience is an expert in the field.** Your task is to evaluate the quality of [task] strictly based on the given evaluation criterion.** Begin the evaluation by providing your analysis concisely and accurately, and then on the next line, start with "Rating:" followed by your rating on a Likert scale from {scale} (higher means better). You MUST keep to the strict boundaries of the evaluation criterion and focus solely on the issues and errors involved; otherwise, you will be penalized.** Make sure you read and understand these instructions, as well as the following evaluation criterion and example content, carefully. \\
**###Evaluation Criterion\#\#f {aspect} & \\
**###Example\#f {source\_des}: & \\
**{target\_des}: & \\
**{target\_} & \\
**\#ffYour Evaluation\#\#f** \\ \hline \hline  \\ \end{tabular}
\end{table}
Table 3: Prompts and instructions used for LLMs to evaluate and annotate NLG tasks.

 



### Experimental Settings

For the data collected in Section 3, all evaluation scores from humans and metrics are normalized to the 0-1 scale for parameter estimation, with results shown in Tables 4 and 5. We used the data where GPT-4-turbo is used as the evaluation metrics to estimate the parameters and the values are averaged across different meta-evaluation sets: \(\mu^{m}=\mu_{1}^{m}=\cdots=\mu_{N}^{m}=0.47,\mu^{h}=\mu_{1}^{h}=\cdots=\mu_{N}^{ h}=0.65\), \(\sigma^{m}=\sigma_{1}^{m}=\cdots=\sigma_{N}^{m}=0.16,\sigma^{n}=\sigma_{1}^{ n}=\cdots=\sigma_{N}^{h}=0.14\), \(\sigma_{\rho_{item}}=0.14\), \(\rho_{sys}=0.92\) and \(\mu_{\rho_{item}}=0.35\). For the number of systems and input documents, we consider \(N=16,M=100\), which is typical and the same as SummEval. The corresponding \(G^{h}\) is set to two common values: 13 and 250, and \(G^{m}\) is selected from 2 to 15 and 5 to 100, respectively. Due to the huge amount of computation, we set \(T_{1}=T_{2}=100\).

## Appendix D Other Figures and Tables

Given the limited space of the main text, we present the complete experimental results here.

Figure 6: The consistency of evaluation metric rankings using different correlation measures on HANNA, calculated through Kendall’s correlation coefficient.

Figure 7: The consistency of evaluation metric rankings using different correlation measures on MANS, calculated through Kendall’s correlation coefficient.



Figure 8: The consistency of evaluation metric rankings using different correlation measures on USR, calculated through Kendall’s correlation coefficient.

Figure 10: The consistency of evaluation metric rankings using different correlation measures on WMT23-ZH2EN-NEWS, calculated through Kendall’s correlation coefficient.

Figure 9: The consistency of evaluation metric rankings using different correlation measures on WebNLG2020, calculated through Kendall’s correlation coefficient.

 



\begin{table}
\begin{tabular}{l l l c r r r r r r} \hline Dataset & Subset & Aspect & Label & \(\mu^{h}\) & \(\sigma^{h}\) & \(G^{h}\) & Scale & Tie Ratio \\ \hline SummEval & CNN/DM & Coherence & D1 & 0.60 & 0.15 & 13 & 1-5 & 0.10 \\ SummEval & CNN/DM & Consistency & D2 & 0.92 & 0.14 & 13 & 1-5 & 0.67 \\ SummEval & CNN/DM & Fluency & D3 & 0.92 & 0.09 & 13 & 1-5 & 0.53 \\ SummEval & CNN/DM & Relevance & D4 & 0.69 & 0.09 & 13 & 1-5 & 0.13 \\ WMT23 & GeneralMT2023_NEWS & Overall Quality & D5 & 0.84 & 0.05 & 251 & 0-250 & 0.11 \\ HANNA & WP & Coherence & D6 & 0.54 & 0.13 & 13 & 1-5 & 0.13 \\ HANNNA & WP & Complexity & D7 & 0.36 & 0.14 & 13 & 1-5 & 0.12 \\ HANNA & WP & Empathy & D8 & 0.32 & 0.09 & 13 & 1-5 & 0.13 \\ HANNA & WP & Engagement & D9 & 0.42 & 0.13 & 13 & 1-5 & 0.12 \\ HANNA & WP & Relevance & D10 & 0.41 & 0.14 & 13 & 1-5 & 0.10 \\ HANNA & WP & Surprise & D11 & 0.28 & 0.10 & 13 & 1-5 & 0.15 \\ MANS & ROC & Overall & D12 & 0.38 & 0.15 & 21 & 1-3 & 0.06 \\ MANS & WP & Overall & D13 & 0.45 & 0.08 & 21 & 1-3 & 0.08 \\ USR & Persona-Chat & Engaging & D14 & 0.76 & 0.13 & 7 & 1-3 & 0.24 \\ USR & Persona-Chat & Maintains Context & D15 & 0.74 & 0.19 & 7 & 1-3 & 0.31 \\ USR & Persona-Chat & Natural & D16 & 0.89 & 0.08 & 7 & 1-3 & 0.48 \\ USR & Persona-Chat & Overall & D17 & 0.69 & 0.19 & 13 & 1-5 & 0.11 \\ USR & Persona-Chat & Understandable & D18 & 0.96 & 0.04 & 4 & 0-1 & 0.84 \\ USR & Persona-Chat & Uses Knowledge & D19 & 0.45 & 0.35 & 4 & 0-1 & 0.38 \\ USR & Topical-Chat & Engaging & D20 & 0.57 & 0.25 & 7 & 1-3 & 0.15 \\ USR & Topical-Chat & Maintains Context & D21 & 0.62 & 0.23 & 7 & 1-3 & 0.17 \\ USR & Topical-Chat & Natural & D22 & 0.64 & 0.21 & 7 & 1-3 & 0.17 \\ USR & Topical-Chat & Overall & D23 & 0.54 & 0.27 & 13 & 1-5 & 0.08 \\ USR & Topical-Chat & Understanding & D24 & 0.67 & 0.23 & 4 & 0-1 & 0.33 \\ USR & Topical-Chat & Uses Knowledge & D25 & 0.55 & 0.24 & 4 & 0-1 & 0.33 \\ WebNLG2020 & WebNLG2020 & Correctness & D26 & 0.88 & 0.07 & 401 & 0-100 & 0.03 \\ WebNLG2020 & WebNLG2020 & Data Coverage & D27 & 0.9 & 0.06 & 401 & 0-100 & 0.04 \\ WebNLG2020 & WebNLG2020 & Fluency & D28 & 0.83 & 0.06 & 401 & 0-100 & 0.01 \\ WebNLG2020 & WebNLG2020 & Relevance & D29 & 0.91 & 0.05 & 401 & 0-100 & 0.05 \\ WebLGL2020 & WebNLG2020 & Text Structure & D30 & 0.87 & 0.05 & 401 & 0-100 & 0.02 \\ \hline \end{tabular}
\end{table}
Table 4: Information of all the meta-evaluation datasets and estimated parameters.

\begin{table}
\begin{tabular}{l l l l l l l l l l l} \hline Metric & \(\widehat{\mu^{m}}\) & \(\widehat{\sigma^{m}}\) & \(\widehat{\rho_{sys}}\) & \(\widehat{\mu_{o,tem}}\) & \(\widehat{\sigma_{obs}}\) & \(r_{S}\) & \(r_{N\times M}\) & \(G^{m}\) & Scale & Tie Ratio \\ \hline GPT3.5\_T=0\_100 & 0.44 & 0.08 & 0.88 & 0.22 & 0.13 & 0.41 & 0.38 & 101 & 0-100 & 0.35 \\ GPT3.5\_T=0\_1\_10 & 0.33 & 0.08 & 0.83 & 0.20 & 0.13 & 0.41 & 0.35 & 10 & 1-10 & 0.46 \\ GPT3.5\_T=0\_1\_5 & 0.32 & 0.10 & 0.86 & 0.21 & 0.12 & 0.40 & 0.35 & 5 & 1-5 & 0.49 \\ GPT3.5\_T=1\_0\_100 & 0.45 & 0.08 & 0.89 & 0.27 & 0.14 & 0.48 & 0.44 & 1001 & 0-100 & 0.03 \\ GPT3.5\_T=1\_1\_10 & 0.33 & 0.08 & 0.88 & 0.26 & 0.13 & 0.46 & 0.41 & 91 & 1-10 & 0.10 \\ GPT3.5\_T=1\_5 & 0.32 & 0.09 & 0.88 & 0.25 & 0.13 & 0.46 & 0.41 & 41 & 1-5 & 0.14 \\ GPT4.T=0\_0\_100 & 0.51 & 0.15 & 0.92 & 0.35 & 0.14 & 0.56 & 0.55 & 101 & 0-100 & 0.22 \\ GPT4.T=0\_1\_10 & 0.45 & 0.15 & 0.92 & 0.34 & 0.13 & 0.55 & 0.53 & 10 & 1-10 & 0.28 \\ GPT4.T=0\_1\_5 & 0.44 & 0.18 & 0.92 & 0.31 & 0.14 & 0.55 & 0.51 & 5 & 1-5 & 0.45 \\ GPT4.T=1\_0\_100 & 0.51 & 0.14 & 0.93 & 0.39 & 0.14 & 0.59 & 0.58 & 1001 & 0-100 & 0.02 \\ GPT4.T=1\_1\_10 & 0.45 & 0.15 & 0.93 & 0.38 & 0.13 & 0.59 & 0.56 & 91 & 1-10 & 0.06 \\ GPT4.T=1\_1\_5 & 0.43 & 0.17 & 0.92 & 0.36 & 0.14 & 0.58 & 0.55 & 41 & 1-5 & 0.18 \\ GPT4.T=0\_0\_100 & 0.48 & 0.14 & 0.92 & 0.37 & 0.13 & 0.57 & 0.55 & 101 & 0-100 & 0.21 \\ GPT4.T=0\_1\_100 & 0.42 & 0.15 & 0.91 & 0.35 & 0.12 & 0.55 & 0.53 & 10 & 1-10 & 0.27 \\ GPT4.T=0\_1\_5 & 0.40 & 0.17 & 0.91 & 0.32 & 0.13 & 0.54 & 0.50 & 5 & 1-5 & 0.43 \\ GPT4.P=1\_0\_100 & 0.48 & 0.14 & 0.92 & 0.39 & 0.13 & 0.59 & 0.58 & 1001 & 0-100 & 0.02 \\ GPT4to\_T=1\_1\_10 & 0.42 & 0.14 & 0.92 & 0.38 & 0.13 & 0.59 & 0.56 & 91 & 1-10 & 0.05 \\ GPT4to\_T=1\_1\_5 & 0.40 & 0.16 & 0.91 & 0.35 & 0.13 & 0.57 & 0.54 & 41 & 1-5 & 0.16 \\ BERTScore-f & 0.87 & 0.04 & 0.60 & -0.02 & 0.10 & 0.26 & 0.22 & / & / & 0.01 \\ BERTScore-p & 0.87 & 0.04 & 0.53 & -0.02 & 0.10 & 0.23 & 0.20 & / &



\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline Grouping & Correlation & D1 & D2 & DS3 & DA4 & DS5 & Do6 & D7 & D8 & DB9 & D10 \\ \hline System & Pearson & 0.844 (4) & 0.878 (2) & 0.673 (12) & 0.637 (10) & 0.853 (8) & 0.777 (7) & 0.928 (1) & 0.825 (1) & 0.876 (1) & 0.643 (10) \\ System & Spearman & 0.695 (8) & 0.756 (10) & 0.724 (7) & 0.622 (11) & 0.657 (12) & 0.581 (11) & 0.680 (8) & 0.542 (11) & 0.669 (8) & 0.553 (11) \\ System & Kendall & 0.666 (11) & 0.766 (8) & 0.724 (12) & 0.603 (12) & 0.681 (11) & 0.533 (12) & 0.680 (9) & 0.532 (12) & 0.637 (12) & 0.534 (12) \\ Input & Pearson & 0.901 (10) & 0.876 (3) & 0.887 (1) & 0.843 (1) & 0.854 (7) & 0.842 (4) & 0.842 (5) & 0.767 (10) & 0.828 (2) & 0.784 (5) \\ Input & Spearman & 0.867 (20) & 0.842 (5) & 0.824 (3) & 0.809 (2) & 0.852 (9) & 0.815 (6) & 0.841 (6) & 0.789 (10) & 0.798 (6) & 0.852 (1) \\ Input & Kendall & 0.862 (30) & 0.873 (4) & 0.815 (4) & 0.799 (3) & 0.897 (3) & 0.836 (34) & 0.833 (7) & 0.791 (3) & 0.792 (7) & 0.833 (2) \\ Global & Pearson & 0.843 (5) & 0.889 (1) & 0.876 (2) & 0.791 (4) & 0.880 (4) & 0.833 (3) & 0.854 (4) & 0.752 (6) & 0.820 (4) & 0.825 (3) \\ Global & Spearman & 0.836 (0.765) & 0.904 (0.872) & 0.572 (5) & 0.925 (1) & 0.839 (3) & 0.877 (2) & 0.819 (2) & 0.810 (5) & 0.805 (4) \\ Global & Kendall & 0.803 (7) & 0.825 (6) & 0.789 (7) & 0.751 (5) & 0.845 (10) & 0.862 (1) & 0.803 (7) & 0.852 (3) & 0.773 (6) \\ Item & Pearson & 0.679 (9) & 0.775 (7) & 0.679 (1) & 0.746 (7) & 0.861 (6) & 0.749 (6) & 0.696 (10) & 0.652 (9) & 0.657 (9) & 0.741 (7) \\ Item & Spearman & 0.673 (10) & 0.704 (11) & 0.671 (11) & 0.730 (8) & 0.734 (8) & 0.683 (11) & 0.669 (8) & 0.654 (11) & 0.731 (8) \\ Item & Kendall & 0.655 (12) & 0.673 (12) & 0.680 (9) & 0.722 (9) & 0.878 (5) & 0.740 (10) & 0.601 (12) & 0.639 (10) & 0.654 (11) & 0.709 (9) \\ \hline Growing & Correlation & D11 & D12 & D13 & D14 & D15 & D16 & D17 & D18 & D19 & D20 \\ \hline System & Pearson & 0.822 (1) & 0.595 (12) & 0.554 (10) & 0.674 (7) & 0.557 (12) & 0.671 (6) & 0.589 (12) & 0.702 (11) & 0.613 (12) & 0.782 (12) \\ System & Spearman & 0.854 (10) & 0.771 (7) & 0.432 (12) & 0.899 (1) & 0.562 (10) & 0.538 (12) & 0.751 (0) & 0.259 (10) & 0.606 (3) & 0.132 (12) \\ Systum & Kendall & 0.569 (12) & 0.771 (8) & 0.487 (11) & 0.886 (1) & 0.558 (11) & 0.592 (11) & 0.555 (11) & 0.611 (13) & 0.148 (11) \\ Input & Pearson & 0.755 (15) & 0.803 (5) & 0.695 (4) & 0.684 (6) & 0.777 (3) & 0.689 (4) & 0.809 (3) & 0.382 (7) & 0.336 (4) & 0.853 (1) \\ Input & Spearman & 0.795 (4) & 0.768 (6) & 0.653 (6) & 0.670 (7) & 0.748 (6) & 0.679 (7) & 0.737 (7) & 0.284 (9) & 0.195 (5) & 0.771 (6) \\ Input & Kendall & 0.733 (6) & 0.805 (4) & 0.671 (6) & 0.651 (9) & 0.723 (8) & 0.659 (7) & 0.737 (8) & 0.318 (8) & 0.192 (9) & 0.673 (7) \\ Global & Pearson & 0.720 (7) & 0.859 (2) & 0.770 (3) & 0.807 (7) & 0.745 (7) & 0.784 (1) & 0.826 (2) & 0.662 (1) & -0.082 (9) & 0.841 (2) \\ Global & Spearman & 0.803 (8) & 0.878 (1) & 0.772 (7) & 0.745 (5) & 0.818 (7) & 0.820 (4) & 0.834 (11) & 0.838 (11) & 0.838 (11) & 0.873 (3) \\ Global & Kendall & 0.798 (3) & 0.834 (3) & 0.782 (10) & 0.747 (4) & 0.792 (7) & 0.777 (5) & 0.759 (4) & 0.630 (4) & 0.653 (11) & 0.816 (4) \\ Item & Pearson & 0.606 (8) & 0.701 (10) & 0.549 (9) & 0.801 (11) & 0.756 (4) & 0.644 (9) & 0.745 (50) & 0.540 (4) & 0.416 (11) & 0.618 (8) \\ Item & Spearman & 0.590 (9) & 0.712 (9) & 0.604 (7) & 0.583 (10) & 0.750 (5) & 0.659 (8) & 0.716 (10) & 0.541 (6) & -0.150 (11) & 0.570 (9) \\ Item & Kendall & 0.573 (11) & 0.648 (11) & 0.574 (5) & 0.562 (12) & 0.721 (9) & 0.623 (6) & 0.663 (11) & 0.550 (5) & -0.155 (12) & 0.552 (10) \\ \hline Grouping & Correlation & D11 & D22 & D23 & D24 & D25 & D26 & D27 & D28 & D29 & D30 \\ \hline System & Pearson & 0.727 (17) & 0.748 (6) & 0.721 (7) & 0.725 (7) & 0.655 (1) & 0.761 (7) & 0.847 (7) & 0.596 (10) & 0.725 (7) & 0.663 (10) \\ System & Spearman & 0.799 (11) & 0.169 (12) & 0.085 (12) & 0.115 (11) & 0.717 (11) & 0.952 (12) & 0.698 (11) & 0.



Figure 12: The p-value curves of correlation measures on meta-evaluation D2.

Figure 16: The p-value curves of correlation measures on meta-evaluation D6.

 

Figure 19: The p-value curves of correlation measures on meta-evaluation D9. Figure 20: The p-value curves of correlation measures on meta-evaluation D10.

Figure 17: The p-value curves of correlation measures on meta-evaluation D7.

 

Figure 23: The p-value curves of correlation measures on meta-evaluation D13.

Figure 24: The p-value curves of correlation measures on meta-evaluation D14.

 

Figure 30: The p-value curves of correlation measures on meta-evaluation D20.

Figure 31: The p-value curves of correlation measures on meta-evaluation D21.

 

Figure 37: The p-value curves of correlation measures on meta-evaluation D27.

Figure 38: The p-value curves of correlation measures on meta-evaluation D28.

 

Figure 43: As the changes of \(G^{m}\), the correlations between the GPT-3.5-Turbo evaluator and human evaluation using different measures on SummEval with the fixed evaluation scale of 1-5.

Figure 42: As the changes of \(G^{m}\), the correlations between the GPT-3.5-Turbo evaluator and human evaluation using different measures on SummEval with the fixed evaluation scale of 1-10.

Figure 46: As the changes of \(G^{m}\), the correlations between the GPT-4o evaluator and human evaluation using different measures on SummEval with the fixed evaluation scale of 1-10.

Figure 41: As the changes of \(G^{m}\), the correlations between the GPT-3.5-Turbo evaluator and human evaluation using different measures on SummEval with the fixed evaluation scale of 1-5.

Figure 43: As the changes of \(G^{m}\), the correlations between the GPT-3.5-Turbo evaluator and human evaluation using different measures on SummEval with the fixed evaluation scale of 1-5.

 

Figure 47: As the changes of \(G^{m}\), the correlations between the GPT-4o evaluator and human evaluation using different measures on WMT23 with the fixed evaluation scale of 1-5.

Figure 48: As the changes of \(G^{m}\), the correlations between the GPT-4o evaluator and human evaluation using different measures on WMT23 with the fixed evaluation scale of 1-10.

Figure 50: As the changes of \(G^{m}\), the correlations between the GPT-4-Turbo evaluator and human evaluation using different measures on WMT23 with the fixed evaluation scale of 1-10.