Venue: NAACL 2024

# Not All Metrics Are Guilty:

Improving NLG Evaluation by Diversifying References

 Tianyi Tang\({}^{1,6}\) * Hongyuan Lu\({}^{4}\), Yuchen Eleanor Jiang\({}^{5}\), Haoyang Huang\({}^{2}\), Dongdong Zhang\({}^{2}\), Wayne Xin Zhao\({}^{1,6}\) * Tom Koemi\({}^{3}\), Furu Wei\({}^{2}\)

\({}^{1}\) Gaoling School of Artificial Intelligence, Renmin University of China

\({}^{2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong  \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflygmail.com

This work was done during internship at MSRA.Corresponding author

###### Abstract

Most research about natural language generation (NLG) relies on evaluation benchmarks with limited references for a sample, which may result in poor correlations with human judgements. The underlying reason is that one semantic meaning can actually be expressed in different forms, and the evaluation with a single or few references may not accurately reflect the quality of the model's hypotheses. To address this issue, this paper presents a simple and effective method, named **Div-Ref**, to enhance existing evaluation benchmarks by enriching the number of references. We leverage large language models (LLMs) to diversify the expression of a single reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. We conduct comprehensive experiments to empirically demonstrate that diversifying the expression of reference can significantly enhance the correlation between automatic evaluation and human evaluation. This idea is compatible with recent LLM-based evaluation which can similarly derive advantages from incorporating multiple references. _We strongly encourage future generation benchmarks to include more references, even if they are generated by LLMs, which is once for all._ We release all the code and data at [https://github.com/RUCAIBox/Div-Ref](https://github.com/RUCAIBox/Div-Ref) to facilitate research.

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong     \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong      \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong       \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{taohua,dozhang,tomkocmi,fuwei}@microsoft.com betmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{kaohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{haohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{thaohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{taohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{taohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{taohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{4}\) The Chinese University of Hong Kong    \({}^{5}\) AIWaves Inc.

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{taohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

+
Footnote †: 2}\) Microsoft Research Asia, China   \({}^{3}\) Microsoft

\({}^{6}\) Beijing Key Laboratory of Big Data Management and Analysis Methods

{steventianyitang,hongyuanlu}@outlook.com eleanor.jiang@aiwaves.cn

{taohua,dozhang,tomkocmi,fuwei}@microsoft.com batmanflykgmail.com

the WMT22 News Translation Task (Kocmi et al., 2022). This potentially leads to unreliable evaluation results when using limited ground-truth references, as illustrated in Table 1.

Considering the above-mentioned issue, this paper attempts to improve the NLG evaluation benchmarks and make existing automatic metrics better reflect the actual quality of the hypotheses. We focus on increasing the number of reference texts to narrow the gap between automatic and human evaluation. The key idea is to leverage the excellent ability of existing LLMs to provide more high-quality references for a single sample. By enriching the diversity of the references while maintaining semantic consistency, we expand the coverage of the semantic expressions for evaluating the generated texts from _a single or few_ standard references to _a more diverse set_ of semantically equivalent references. In this way, our evaluation method can better approximate human evaluation criteria, as the improved scores shown in Table 1. In addition, increasing the number of references is _agnostic_ to specific task settings and can be integrated with various automatic metrics for evaluating different generation tasks.

To demonstrate the effectiveness of diversifying references, we conduct extensive experiments on the benchmarks from multiple NLG tasks. The experimental results demonstrate that incorporating multiple references can significantly improve the consistency between traditional evaluation metrics and human evaluation results. Surprisingly, it is even applicable in multilingual and multimodal text generation scenarios. Importantly, our approach is orthogonal with automatic metrics, enabling even the recent LLM-based evaluations (Kocmi and Federmann, 2023; Wang et al., 2023) to benefit from our diversified references and achieve SOTA correlation with human judges. _Therefore, incorporating more references for the NLG benchmark proves advantageous, requiring a one-time effort, and future researchers can reap its benefits._

## 2 Related Work

### Automatic Evaluation

Automatic evaluation metrics for natural language generation could be mainly categorized into two streams: reference-based and reference-free evaluation. The former involves measuring the quality of the hypothesis by comparing it with single or few ground-truth references, _e.g.,_ BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR Banerjee and Lavie (2005). They primarily focus on the n-gram overlaps between the hypothesis and the references. Recently, neural metrics have become a mainstream method to evaluate semantic similarity and usually have a higher correlation with human evaluation. The representative metrics include BERTScore Zhang et al. (2020), BLEURT Sellam et al. (2020), and recent methods involving LLMs Kocmi and Federmann (2023); Wang et al. (2023); Chiang and Lee (2023); Luo et al. (2023); Lu et al. (2023); Gao et al. (2023). Reference-free evaluations assess the hypothesis without the necessity of any reference. They often adopt neural-based models as a black box for evaluating semantic quality as well as grammatical fluency Zhao et al. (2020); Mehri and Eskenazi (2020); Hessel et al. (2021); Liu et al. (2023); Chen et al. (2023). However, the reference-free metrics has lower correlation with human compared to the reference-based ones Kocmi and Federmann (2023); Wang et al. (2023). In this work, we primarily focus on the reference-based automatic metrics, even without the need for altering their core implementation.

### Increasing the Reference Number

Initially, researchers attempt to utilize paraphrasing methods Bandel et al. (2022) to enrich the instances of training set Zheng et al. (2018); Khayralilah et al. (2020). Zhou et al. (2006b) use paraphrasing to enhance the evaluation of the summarization task. There are also prior works that employed paraphrasing in enhancing evaluations with machine translation, either by human paraphrasing Gupta et al. (2019); Freitag et al. (2020, 2020) or automatic paraphrasing Zhou et al. (2006a); Kauchak and Barzilay (2006); Thompson and Post (2020); Bawden et al. (2020, 2020). One recent study reports that the maximization of diversity should be favored for paraphrasing Bawden et al. (2020), which enhances the succeeding evaluation. Although current work showcases the promise of paraphrasing methods, they are confined to improving the correlation of specific metrics (_e.g.,_ BLEU and ROUGE) in certain tasks (_e.g.,_ translation and summarization). They neglect to explore the importance of the number of references, considering constraints such as the quality of automatic paraphrasing or the expense of human paraphrasing. Meanwhile, our investigation reveals that the majority of newly proposed NLG benchmarks in 2023 continue to rely on only one reference. Even those benchmarks 

incorporating multiple references typically feature no more than two or three ground truth. The advent of LLMs has facilitated a convenient and effective means of diversifying references to encompass the semantic space of samples. In this work, we design dedicated prompts tailored for LLMs and extensively investigate the imperative of augmenting the number of references in NLG benchmarks.

## 3 Methodology

This section first provides a formal definition by introducing several crucial aspects of NLG evaluation. We then describe our approach that leverages LLMs to enrich the semantic coverage of references, bridging the gap between automatic evaluation and human evaluation.

### NLG Evaluation Formulation

As for an NLG task, let \({\bf x}\) denote the input sequence associated with extra information (task goal, additional context, _etc_) and \({\bf y}^{*}\) denote the ground-truth reference provided by the benchmark. After a model or system generates the hypothesis sequence \(\hat{\mathbf{y}}\), the automatic evaluation of the metric \(\mathcal{M}\) can be represented as \(\mathcal{M}(\hat{\mathbf{y}}|{\bf x},{\bf y}^{*})\). Accordingly, we can also represent human evaluation as \(\mathcal{H}(\hat{\mathbf{y}}|{\bf x},{\bf y}^{*})\). Hence, to access the quality of the metric \(\mathcal{M}\), researchers usually calculate the correlation score with human evaluation \(\mathcal{H}\):

\[\rho(\mathcal{M}(\hat{\mathbf{y}}|{\bf x},{\bf y}^{*}),\mathcal{H}(\hat{ \mathbf{y}}|{\bf x},{\bf y}^{*})), \tag{1}\]

where \(\rho\) can be any correlation function such as Spearman correlation and Kendall's tau. An ideal metric is to maximize the correlation between automatic evaluation \(\mathcal{M}\) and human evaluation \(\mathcal{H}\).

Note that, \(\mathcal{H}\) is a subjective process and cannot be directly calculated. Intuitively, when a human assesses on the hypothesis \(\hat{\mathbf{y}}\), he or she will match \(\hat{\mathbf{y}}\) among various valid sentences, which can be illustrated as a semantic sentence space \(\mathbb{Y}\) formed in our brain based on human knowledge and common sense related to the ground-truth reference \({\bf y}^{*}\). Therefore, the human evaluation can be further described as \(\mathcal{H}(\hat{\mathbf{y}}|{\bf x},\mathbb{Y})\).

While researchers on NLG evaluation focus on proposing various implementations of \(\mathcal{M}\), we aim to improve the automatic evaluation benchmark using \(\mathcal{M}(\hat{\mathbf{y}}|{\bf x},A(\mathbb{Y}))\), where \(A(\mathbb{Y})\) is the approximation of \(\mathbb{Y}\) to instantiate the semantic space. \(A(\mathbb{Y})\) is defined as \(\{{\bf y}^{*},\tilde{\mathbf{y}}_{1},\ldots,\tilde{\mathbf{y}}_{n}\}\) to alleviate the bias and insufficiency of a single reference in representing the entire semantic space of the ground-truth references. To achieve this, we augment the reference with diverse expressions while retaining the same meaning, aiming to approximate the semantic space \(\mathbb{Y}\). In the traditional single-reference evaluation benchmark, \(A(\mathbb{Y})\) corresponds to \(\{{\bf y}^{*}\}\).

As the acquisition of \(A(\mathbb{Y})\) is costly for human annotation, we propose to leverage the superior capability of LLMs to generate high-quality and diverse references. With this approach, the automatic evaluation can be formulated as follows:

\[\mathcal{M}(\hat{\mathbf{y}}|{\bf x},A(\mathbb{Y}))=\mathcal{M}(\hat{\mathbf{ y}}|{\bf x},{\bf y}^{*},\tilde{\mathbf{y}}_{1},\ldots,\tilde{\mathbf{y}}_{ n}). \tag{2}\]

Traditional metrics, such as BLEU Papineni et al. (2002) and ChrF Popovic (2015), have built-in algorithms to handle multiple references, while for neural metrics, they only support a single reference and then aggregate the scores from each reference. In practice, the evaluation score under the multipleference setting can be calculated as follows:

\[\mathcal{M}(\hat{\mathbf{y}}|{\bf x},{\bf y}^{*},\tilde{\mathbf{y}}_{1}, \ldots,\tilde{\mathbf{y}}_{n})=\mathop{\mathcal{F}}_{i=0}^{n}\big{[}\mathcal{ M}(\hat{\mathbf{y}}|{\bf x},\check{\mathbf{y}}_{i})\big{]}, \tag{3}\]

where \(\check{\mathbf{y}}_{0}={\bf y}^{*}\) and \(\mathcal{F}\) is a function leveraged to aggregate scores of multiple diversified sequences, which can be the operation of max aggregation or mean aggregation.

### LLM Diversifying for Evaluation

Recently, LLMs have showcased remarkable capabilities across various NLP tasks. They have proven to be powerful aids in tasks such as text paraphrasing, text style transfer, and grammatical error correction Kaneko and Okazaki (2023). Therefore, we harness the potential of LLMs as the approximation function \(A\) to generate diverse expressions \(\tilde{\mathbf{y}}_{1},\ldots,\tilde{\mathbf{y}}_{n}\) while preserving the original semantics of the ground-truth reference \({\bf y}^{*}\).

#### 3.2.1 Paraphrasing Prompt

Following existing work Bawden et al. (2020), we provide the LLM with the paraphrasing prompt "Paraphrase the sentences: {reference}" to wrap the given reference and employ nucleus sampling Holtzman et al. (2020) to generate a variety of rephrased sentences. In our preliminary experiments, we apply the paraphrasing prompt to paraphrase ten sentences for each English reference sentence from the WMT22 Metrics Shared Task Freitag et al. (2022). We calculate a semantic diversity score1 of the rephrased sentences as 0.032.

 

Footnote 1: [http://www.cds.org/](http://www.cds.org/)

We further observe that rephrased sentences primarily involve word-level substitutions, with minimal modifications to the sentence structure. d

#### 3.2.2 Diversified Prompts

To improve the diversity of the reference sentences as suggested by Bawden et al. (2020), we explore several heuristic rules to obtain more diverse texts and cover the semantic space. Inspired by Jiao et al. (2023), we ask ChatGPT to provide instructions that cover different aspects of semantic expressions with the prompt: "_Provide ten prompts that can make you diversify the expression of given texts by considering different aspects._". According to the suggestions by Savage and Mayer (2006), we screen out ten diversifying instructions to promote the changes in words, order, structure, voice, style, _etc_, which are listed as follows:

[colframe=black!50]Change the order of the sentences:

Change the structure of the sentences:

Change the voice of the sentences:

Change the tense of the sentences:

Alter the tone of the sentences:

Alter the style of the sentences:

Rephrase the sentences while retaining the original meaning:

Use synonyms or related words to express the sentences with the same meaning:

Use more formal language to change the level of formality of the sentences:

Use less formal language to change the level of formality of the sentences:

Then, we also utilize the ten instructions to generate ten diversified sentences in total (_i.e.,_ one for each instruction). The semantic diversity score increases from 0.032 to 0.049, which demonstrates a significant diversity improvement among the sentences and verifies the effectiveness of our diverse prompts. Note that, our diversifying method is not just paraphrasing but attempts to cover different aspects of the reference expressions. Considering the strong cross-lingual generation capabilities of LLMs Muennighoff et al. (2022), we apply English instructions to diversify references in different languages (_e.g.,_ German and Russian). The diversified examples can be found in Tables 6, 7, 8.

#### 3.2.3 Discussion

Compared with existing work Freitag et al. (2020); Bawden et al. (2020) that utilizes paraphrasing for evaluation, we leverage the recent superior LLMs for diversifying the expressions of given reference. After supervised fine-tuning and reinforcement learning from human feedback, LLMs showcase excellent capability to follow the input instruction and align with human preference, which can not achieve by previous paraphrasing methods. To verify the effectiveness of LLMs, we further conduct experiments in Section 4.3 to compare them with traditional paraphrasing models. Moreover, we conduct experiments to evaluate the diversifying results of LLMs. We employ another excellent GPT 3.5 to judge whether the generated sentence conveys the same meaning of given reference. The results show that 94.6% of the generated sentences are suitable, which demonstrates the effectiveness and robustness of our diverse prompts. Note that, LLM diversifying is simple and convenient and does not need any post manual filtering. We conduct further experiments to verify it in Section 4.3.

## 4 Experiments

In this section, we deliberately select three different types of natural language generation tasks to verify the effectiveness of multiple references.

### Experimental Setup

#### 4.1.1 Benchmarks

We choose three meta evaluation benchmarks covering multilingual and multimodal scenarios. These metric benchmarks consist of human scores of the generated text (_i.e.,_\({\mathcal{H}}(\mathbf{y^{\prime}|x},\mathbb{Y})\)), and we can calculate their correlation with the automatic metric scores \({\mathcal{M}}(\mathbf{y^{\prime}|x},A(\mathbb{Y}))\) using multiple references.

* WMT22 Metrics Shared Task Freitag et al. (2022) includes the generated sentences of different competitor models in the WMT22 News Translation Task Kocmi et al. (2022). They require human experts to rate these sentences via the multidimensional quality metrics (MQM) schema. We use all three evaluated language pairs, including Chinese (Zh)\(\rightarrow\)English (En), English (En)\(\rightarrow\)German (De), and English (En)\(\rightarrow\)Russian (Ru). We leverage the standardized toolkit mt-metrics-eval V22 to calculate the segment-level Kendall Tau score and the 

system-level pairwise accuracy following Kocmi et al. (2021). Note that the overall system-level pairwise accuracy across three languages is the most important metric for translation evaluation Deutsch et al. (2023).
* SummEval Fabbri et al. (2021) comprises 200 summaries generated by each of the 16 models on the CNN/Daily Mail dataset See et al. (2017). Human judgements measure these summaries in terms of coherence, consistency, fluency, and relevance. We apply the sample-level Spearman score to measure the correlation.
* PASCAL-50S Vedantam et al. (2015) is a triple collection of 4,000 instances wherein each instance consists of one reference and two captions. Human annotators compare the two captions based on the reference and express their preference. We calculate the accuracy of whether the metric assigns a higher score to the caption preferred by humans. Our experiments follow the setups outlined by Hessel et al. (2021).

#### 4.1.2 Metrics

We evaluate a variety of automatic metrics covering different categories. Based on the taxonomy of existing work Sai et al. (2022), we select 17 metrics subdivided into five classes:

* Character-based metrics: ChrF Popovic (2015);
* Word-based metrics: BLEU Papineni et al. (2002), ROUGE-1/2/L Lin (2004), METEOR Banerjee and Lavie (2005), CIDEr Vedantam et al. (2015), and SPICE Anderson et al. (2016);
* Embedding-based metrics: BERTScore Zhang et al. (2020) and MoverScore;
* Trained metrics: BLEURT Sellam et al. (2020), Prism Thompson and Post (2020), COMET Rei et al. (2020), and BARTScore Yuan et al. (2021);
* LLM-based metrics: GEMBA-Dav3-DA Kocmi and Federmann (2023) and ChatGPT-eval Stars w/ ref) Wang et al. (2023);

The implementation of each metrics are detailed Appendix A.1. The metrics we used for each benchmark are listed in Table 2.

#### 4.1.3 Implementation Details

As for our approach, we utilize the gpt-3.5-turbo-instruct model as the LLM along with the instructions outlined in Section 3.2 to diversify the reference sentences into different expressions. When utilizing the OpenAI API, we set the temperature to 1 and the top_p to 0.9. In Equation 3, we employ the max aggregation and generate 10 diversified sentences (_i.e.,_ one for each instruction). We further analyze these hyper-parameters in Section 4.3.

In our experiments, the baseline method is the evaluation of various metrics over single-reference benchmarks, represented by **Single-Ref**, and the evaluation of our approach over multiple diversified references is denoted as **Div-Ref**.

### Experimental Results

The results of the three evaluation benchmarks over various automatic metrics are shown in the following subsections. We can see that enriching the number of references using our our LLM diversifying method shows a better correlation with human evaluation than the single-reference baseline. Our method is also compatible with existing SOTA LLM-based methods and can enhance them to achieve a higher correlation.

#### 4.2.1 Evaluation on Machine Translation

As shown in the figure 1, our Div-Ref method has shown consistent correlation improvements across all evaluation on the system-level accuracy when compared to the single-reference of the baseline system. Surprisingly, the SOTA metric GEMBA can still be enhanced when evaluated with more references. In terms of different languages, we observe that the diversifying methods are effective

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Categories** & **Metrics** & **Translation** & **Summarization** & **Caption** \\ \cline{2-5}
**Character** & ChrF & ✓ & – & – \\ \hline  & BLEU & ✓ & – & ✓ \\ \cline{2-5}  & ROUGE-1 & – & ✓ & – \\ \cline{2-5}  & ROUGE-2 & – & ✓ & – \\ \cline{2-5}
**Word** & ROUGE-L & – & ✓ & ✓ \\ \cline{2-5}  & METEOR & – & – & ✓ \\ \cline{2-5}  & CIDEr & – & – & ✓ \\ \cline{2-5}  & SPEC & – & – & ✓ \\ \hline \multirow{2}{*}{**Embedding**} & BERTscore & ✓ & ✓ & ✓ \\ \cline{2-5}  & MoverScore & – & ✓ & – \\ \hline \multirow{8}{*}{**Trained**} & BLEURUT & ✓ & – & – \\ \cline{2-5}  & Prism & ✓ & – & – \\ \cline{1-1} \cline{2-5}  & COMET & ✓ & – & – \\ \cline{1-1} \cline{2-5}  & BARTScore & ✓ & – & – \\ \hline \multirow{2}{*}{**LLM**} & GEMBA & ✓ & – & – \\ \cline{2-5}  & ChatGPT-eval & – & ✓ & – \\ \hline \hline \end{tabular}
\end{table}
Table 2: The summary of metrics evaluated on tasks.

 

across different languages. English and Russian references benefit more than the German ones, which may be due to the distinct multilingual ability of gpt-3.5-turbo. Notably, our approach showcases significant effects on the traditional BLEU metric, which can further facilitate the application due to its efficiency and universality. The large improvement further demonstrates the automatic metric may be not guilty but the evaluation benchmark needs more references.

#### 4.2.2 Evaluation on Text Summarization

In the summarization task, we select six metrics to examine the correlation against human evaluation

Figure 1: System-level pairwise accuracy (main aspect) and Kendall Tau correlation of segment-level score over the WMT22 Metrics Shared Task on three translation directions.

Figure 2: Spearman score of sample-level correlation over the SummEval benchmark on four evaluation aspects.

 

from four aspects: coherence, consistency, fluency, and relevance. According to the results shown in Figure 2, the Div-Ref method can make significant improvements in almost all dimensions compared to the traditional single-reference approach. We can see that the traditional word-based metrics (_e.g.,_ ROUGE) and the embedding-based metrics (_e.g.,_ BERTScore) perform closely, while LLM-based metric shows remarkable correlation with human evaluation. This phenomena further demonstrates the effectiveness of LLMs for NLG evaluation, as described by Wang et al. (2023). It should be noted that our method has further improved the LLM-based metric ChatGPT-eval in all dimensions. This also shows that our approach is effective in improving the correlation with human evaluation and the NLG benchmarks should include more references.

#### 4.2.3 Evaluation on Image Caption

In order to examine the effectiveness of our method for the image caption task, we expand the reference under four different settings to judge whether the metric assigns a higher score to the caption preferred by humans. The results of the image caption task are reported in Figure 3. For the HC and MM settings, which are difficult settings to judge two similar captions, Div-Ref exhibits enhancements in all metrics, particularly for SPICE, METEOR, and BERTScore. This verifies our approach can expand the semantic coverage of references to bridge the gap between automatic evaluation and human evaluation. Regarding HI and HM, Div-Ref still maintains the improvements in all metrics, except for a slight drop for BERTScore in the HM setting. Despite one of the candidate captions being incorrect or machine-generated, our method can strongly align different metrics with human preference, particularly for the SPICE metric. In comparison to the single-reference baseline, our approach yields a significant improvement of 3.6 points with SPICE in HI and 2.9 points for HM.

### Ablation Analysis

In this section, we examine the impact of various factors of increasing the reference numbers, which include the selection of diversifying models, the application of instruction prompts, the choice of the aggregation function, the effect of post-filtering, and the number of diversified references. The results can be found in Table 3 and 4 and Figure 4.

(1) Firstly, we compare the influence of our diversifying LLM gpt-3.5-turbo-instruct with three rephrasing PLMs PEGASUS-Paraphrasing3,

Figure 3: Accuracy score over the PASCAL-50S benchmark on four settings. HC denotes the two captions are correct and written by humans. HI denotes two human-written captions but one is irrelevant. HM denotes one caption is human-written and the other is model-generated. MM denotes two model-generated captions.

 

Parrot4, and QCPG (Bandel et al., 2022), which are fine-tuned on paraphrasing tasks. However, these three models only support English paraphrasing. We also incorporate another open-source LLMs, LLaMA-2-70b-chat, to diversify our references. From the results, we observe that gpt-3.5-turbo-instruct can outperform three PLMs and LLaMA-2-chat in all metrics, which showcases its superior capability in completing the semantic space of given reference.

Footnote 4: [https://huggingface.co/prithviida/parrot_paraphraser_on_T5](https://huggingface.co/prithviida/parrot_paraphraser_on_T5)

(2) Regarding the choice of instruction prompts, we first degrades the diverse prompts to the basic prompt mentioned in Section 3.2. We observe that the diverse prompts can achieve satisfactory results on English references (_i.e.,_ Zh-En), and may slightly reduce the performance on non-English languages (Table 4). Then, we further translate the English diverse prompts into respective language (_i.e.,_ instructing LLMs using the reference language), and find the gains of multilingual diverse prompts are also not obvious. We attribute the two results to that fact the diversifying ability of LLMs in non-English is not as good as that in English, since English is the dominant language. Besides, we analyze each kind of our diverse prompts in Appendix. We compare a mixture of one sentence per prompt with ten sentences per prompt. From the results in Table 5, we can find that mixing prompts is better than any individual prompt. This further demonstrates the effectiveness of our delicate prompts and they can cover a broader semantics range of reference sentences.

(3) Thirdly, we investigate the aggregation functions using the mean aggregation and the built-in multi-reference aggregation of BLEU and ChrF. We discover that when changing the aggregation from \(max\) to \(mean\), the correlation scores for most metrics have dropped, especially in the Chinese-to-English direction. This indicates that the highest-quality reference plays a dominant role in generation evaluation, and our approach to increasing the number of references significantly strengthens this probability. However, averaging multiple reference scores could introduce noise from low-quality reference scores. As for the built-in method of BLEU and ChrF, their performances are indistinguishable.

\begin{table}
\begin{tabular}{l l|c c|c c|c c|c c|c c|c c|c c} \hline \multicolumn{1}{c}{} & \multirow{2}{*}{**Settings**} & \multicolumn{2}{c|}{**BLEU**} & \multicolumn{2}{c|}{**ChrF**} & \multicolumn{2}{c|}{**BERTScore**} & \multicolumn{2}{c|}{**BLEURT**} & \multicolumn{2}{c|}{**Prism**} & \multicolumn{2}{c|}{**COMET**} & \multicolumn{2}{c}{**Average Gains**} \\  & & En-De & En-Ru & En-De & En-Bu & En-De & En-Bu & En-De & En-Bu & En-De & En-Bu & En-De & En-Bu & En-De & En-Bu \\ \hline \multirow{6}{*}{**Ours** & Single-Ref & 16.9 & 14.0 & 21.4 & 16.8 & 23.2 & 19.2 & 34.4 & 35.9 & 21.5 & 23.0 & 34.3 & 37.2 & 0.0 & 0.0 \\  & **Ours** (GPT 3.5+Diverse+Max) & 19.3 & 17.9 & 24.5 & 21.6 & 25.9 & 23.5 & 34.7 & 37.1 & 23.4 & 26.1 & 35.0 & 38.5 & +1.9 & +3.1 \\ \cline{1-1} \cline{2-19}
**Model** & LLAMA-2-70b-chat & 18.1 & 16.0 & 22.8 & 19.5 & 24.1 & 21.6 & 34.8 & 36.8 & 22.4 & 24.7 & 35.1 & 38.2 & +0.9 & +1.7 \\ \hline \multirow{2}{*}{**Prompt**} & Basic & 19.6 & 19.3 & 25.2 & 24.2 & 26.2 & 25.4 & 35.5 & 34.7 & 23.9 & 23.0 & 35.2 & 34.8 & +2.3 & +2.6 \\  & Multilingual & 18.9 & 19.1 & 22.4 & 22.2 & 23.9 & 24.2 & 37.3 & 37.1 & 26.4 & 26.1 & 38.7 & 38.9 & +2.7 & +3.6 \\ \cline{1-1} \cline{2-19}
**Aggregation** & Mean & 13.9 & 15.0 & 17.2 & 16.3 & 20.0 & 19.4 & 32.3 & 37.0 & 19.2 & 22.3 & 32.0 & 36.6 & -2.8 & +0.1 \\  & Built-in & 18.4 & 18.1 & 24.5 & 21.6 & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline \multicolumn{1}{c}{**Filtering subpar references**} & 19.4 & 17.9 & 24.8 & 21.6 & 26.0 & 23.5 & 34.8 & 37.1 & 23.4 & 26.1 & 35.1 & 38.5 & +0.2 & 0.0 \\ \hline \end{tabular}
\end{table}
Table 4: Ablation analysis in the English-to-German and English-to-Russia and directions using segment-level Kendall Tau correlation.

\begin{table}
\begin{tabular}{l l|c c|c c|c c|c c|c c|c c|c c} \hline \multicolumn{1}{c}{} & \multirow{2}{*}{**Setting**} & \multicolumn{2}{c|}{**BLEU**} & \multicolumn{2}{c|}{**ChrF**} & \multicolumn{2}{c|}{**BERTScore**} & \multicolumn{2}{c|}{**BLEURT**} & \multicolumn{2}{c|}{**Primn**} & \multicolumn{2}{c|}{**COMET**} & \multicolumn{2}{c}{**Average Gains**} \\  & & Syuen & Zh-En & System & Zh-En & System & Zh-En & System & Zh-En & System & Zh-En & System & Zh-En & System & Zh-En & System & Zh-En \\ \hline \multirow{6}{*}{**Mond**} & Single-Ref & 71.5 & 14.5 & 75.9 & 14.7 & 77.4 & 31.6 & 84.7 & 36.1 & 76.3 & 25.7 & 82.8 & 35.6 & 0.0 & 0.0 \\  & **Ours** (GPT 3.5+Diverse+Max) & 77.7 & 19.4 & **78.5** & 19.1 & **82.1** & 34.2 & **84.7** & 37.7 & **79.9** & **28.1** & **83.9** & **36.8** & **+3.0** & **+2.9** \\ \hline \multirow{6}{*}{**Model**} & PEGASUS & \(\times\) & 18.2 & \(\times\) & 18.5 & \(\times\) & 33.2 & \(\times\) & 37.0 & \(\times\) & 27.4 & \(\times\) & 36.0 & \(\times\) & +2.0 \\  & Parrot & 

(4) In addition, we attempt to filter the generated references considering some of them may be of low quality. We employ gpt-3.5-turbo to judge using the instruction: "_Sentence 1: [ref]nSentence 2: {div_reff}/uDo sentence 1 and sentence 2 convey the same meaning?ruvn_". After eliminating the reference unrecognized by gpt-3.5-turbo, we can find that the removal of low-quality sentences has minimal impact on correlation results. We speculate that our approach involves aggregating results from multiple references and selecting the one with the highest score, effectively disregarding those of inferior quality.

(5) Finally, we examine the influence of scaling the number of references. We utilize the diverse prompts to generate more references. From Figure 4, we observe a consistent upward trend in the overall performance as the number of references increases. For word-based metrics, this growth trend is more obvious. This experiment further shows that traditional benchmarks that relies on a single reference is very one-sided for NLG evaluation, and we need to provide multiple references for benchmarks. Considering that the performance of neural metrics tends to saturate when the quantity is high, over-generation may not lead to more significant gains, suggesting that the optimal cost-effective number may not exceed 20.

## 5 Conclusion

In this paper, we have investigated the effect of enriching the number of references in NLG benchmarks and verified its effectiveness. Our diversifying method, Div-Ref, can effectively cover the semantic space of the golden reference, which can largely extend the limited references in existing benchmarks. With extensive experiments, our approach yields substantial improvements in the consistencies between evaluation metrics and human evaluation. In future work, we will explore the current evaluation method on more NLG tasks, and also consider extending it to evaluate generation tasks in other modalities. It is also valuable to investigate whether paraphrasing can improve LLMs' training and utilization.

## Acknowledgement

This work was partially supported by Beijing Natural Science Foundation under Grant No. L233008 and 4222027. Xin Zhao is the corresponding author.

Figure 4: Kendall Tau correlation score _w.r.t._ the number of generated references in the Chinese-to-English direction on the WMT22 Metrics Shared Task.

\begin{table}
\begin{tabular}{c|c c c c c c} \hline \hline
**Prompts** & **BLEU** & **ChF** & **BERTScore** & **BLEURT** & **Prism** & **COMET** & **Average Gains** \\ \hline Single-Ref & 14.5 & 14.7 & 31.6 & 36.1 & 25.7 & 35.6 & 0.0 \\
**Ours (Mixing)– 



### Limitations

Despite conducting numerous experiments, further research is required to explore the number of references and the optimal diversifying techniques that can achieve a trade-off between time and effectiveness. Since using more references leads to more evaluation time, future work can explore strategies for mitigating these issues, possibly through the implementation of a selection mechanism that prioritizes sentences with diverse expressions while minimizing the overall number of reference sentences. Moreover, Our diverse prompts may fail in specialized domains, such as finance and biomedicine. Rewriting professional terms may lead to inaccuracy evaluation of the generated sentences. Future work can further investigate and validate the effectiveness of our method within these domains. Additionally, we can design more fine-grained prompts tailored to address the specific challenges posed by professional terminology. In addition, due to the high cost of text-davinci-003, we omit the experiments of GEMBA in the ablation analysis, which may lead to an incomplete analysis of LLM-based metrics. The OpenAI API also is non-deterministic, which may lead to different diversifying results for the same input. There is also a chance that OpenAI will remove existing models.

## References

* Alva-Manchego et al. (2021) Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitability of automatic evaluation metrics for text simplification. _Computational Linguistics_, 47(4):861-889.
* ECCV 2016_, pages 382-398, Cham. Springer International Publishing.
* Bandel et al. (2022) Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer, Ilya Shuayderman, Noam Slonim, and Liat Ein-Dor. 2022. Quality controlled paraphrase generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 596-609, Dublin, Ireland. Association for Computational Linguistics.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
* Bawden et al. (2020a) Rachel Bawden, Biao Zhang, Andre Tattar, and Matt Post. 2020a. ParBLEU: Augmenting metrics with automatic paraphrases for the WMT'20 metrics shared task. In _Proceedings of the Fifth Conference on Machine Translation_, pages 887-894, Online. Association for Computational Linguistics.
* Bowden et al. (2020b) Rachel Bawden, Biao Zhang, Lisa Yankovskaya, Andre Tattar, and Matt Post. 2020b. A study in improving BLEU reference coverage with diverse automatic paraphrasing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 918-932, Online. Association for Computational Linguistics.
* Celikyilmaz et al. (2020) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. _arXiv preprint arXiv:2006.14799_.
* Chen et al. (2023) Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. _arXiv preprint arXiv:2304.00723_.
* Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? _arXiv preprint arXiv:2305.01937_.
* Deutsch et al. (2023) Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Modifying kendall's tau for modern metric meta-evaluation. _arXiv preprint arXiv:2305.14324_.
* Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. _Transactions of the Association for Computational Linguistics_, 9:391-409.
* Freitag et al. (2020a) Markus Freitag, George Foster, David Grangier, and Colin Cherry. 2020a. Human-paraphrased references improve neural machine translation. In _Proceedings of the Fifth Conference on Machine Translation_, pages 1183-1192, Online. Association for Computational Linguistics.
* Freitag et al. (2020b) Markus Freitag, David Grangier, and Isaac Caswell. 2020b. BLEU might be guilty but references are not innocent. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 61-71, Online. Association for Computational Linguistics.
* neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.


Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. _arXiv preprint arXiv:2304.02554_.
* Gupta et al. (2019) Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_, pages 379-391, Stockholm, Sweden. Association for Computational Linguistics.
* Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7514-7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _International Conference on Learning Representations_.
* Jiao et al. (2023) WX Jiao, WX Wang, JT Huang, Xing Wang, and ZP Tu. 2023. Is chatgtprt a good translator? yes with gpt-4 as the engine. _arXiv preprint arXiv:2301.08745_.
* Kaneko and Okazaki (2023) Masahiro Kaneko and Naoaki Okazaki. 2023. Reducing sequence length by predicting edit operations with large language models. _arXiv preprint arXiv:2305.11862_.
* Kauchak and Barzilay (2006) David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_, pages 455-462, New York City, USA. Association for Computational Linguistics.
* Khayrollah et al. (2020) Huda Khayrollah, Brian Thompson, Matt Post, and Philipp Koehn. 2020. Simulated multiple reference training improves low-resource machine translation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 82-89, Online. Association for Computational Linguistics.
* Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novak, Martin Popel, and Maja Popovic. 2022. Findings of the 2022 conference on machine translation (WMT22). In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Kocmi and Federmann (2023) Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. _arXiv preprint arXiv:2302.14520_.
* Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to shship: An extensive evaluation of automatic metrics for machine translation. In _Proceedings of the Sixth Conference on Machine Translation_, pages 478-494, Online. Association for Computational Linguistics.
* Li et al. (2022) Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2022. A survey of pretrained language models based text generation. _arXiv preprint arXiv:2201.05273_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. GptoEval: Nlg evaluation using gpt-4 with better human alignment. _arXiv preprint arXiv:2303.16634_.
* Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error analysis promptting enables human-like translation evaluation in large language models: A case study on chatgpt. _arXiv preprint arXiv:2303.13809_.
* Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. _arXiv preprint arXiv:2303.15621_.
* Mehri and Eskenazi (2020)Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 681-707, Online. Association for Computational Linguistics.
* Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutauwika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. _arXiv preprint arXiv:2211.01786_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference 

on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Sai et al. (2022) Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for nlg systems. _ACM Comput. Surv._, 55(2).
* Savage and Mayer (2006) Alice Savage and Patricia Mayer. 2006. _Effective academic writing: the short essay_. Oxford University Press.
* See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Sulem et al. (2018) Elior Sulem, Omri Abend, and Ari Rappoport. 2018. BLEU is not suitable for the evaluation of text simplification. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 738-744, Brussels, Belgium. Association for Computational Linguistics.
* Thompson and Post (2020a) Brian Thompson and Matt Post. 2020a. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 90-121, Online. Association for Computational Linguistics.
* Thompson and Post (2020b) Brian Thompson and Matt Post. 2020b. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 90-121, Online. Association for Computational Linguistics.
* Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 4566-4575, Los Alamitos, CA, USA. IEEE Computer Society.
* Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. _arXiv preprint arXiv:2303.04048_.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In _Advances in Neural Information Processing Systems_, volume 34, pages 27263-27277. Curran Associates, Inc.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhang et al. (2023) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. _arXiv preprint arXiv:2301.13848_.
* Zhao et al. (2020) Wei Zhao, Goran Glavas, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1656-1671, Online. Association for Computational Linguistics.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 563-578, Hong Kong, China. Association for Computational Linguistics.
* Zheng et al. (2018) Renjie Zheng, Mingbo Ma, and Liang Huang. 2018. Multi-reference training with pseudo-references for neural translation and text generation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3188-3197, Brussels, Belgium. Association for Computational Linguistics.
* Zhou et al. (2006a) Liang Zhou, Chin-Yew Lin, and Eduard Hovy. 2006a. Re-evaluating machine translation results with paraphrase support. In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_, pages 77-84, Sydney, Australia. Association for Computational Linguistics.
* Zhou et al. (2006b) Liang Zhou, Chin-Yew Lin, Dragos Stefan Munteanu, and Eduard Hovy. 2006b. ParaEval: Using paraphrases to evaluate summaries automatically. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_, pages 447-454, New York City, USA. Association for Computational Linguistics.

 

Experimental Details

### Metric Implementation

The implementation details of each metric in different benchmarks are listed as follows:

* ChrF (Popovic, 2015): We utilize sentence-level Chrf from SacreBLEU5 for machine translation.

Footnote 5: [https://github.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu)

Footnote 6: [https://github.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu)

* BLEU Papineni et al. (2002): We utilize sentence-level BLEU from SacreBLEU6 for machine translation, and employ BLEU from pycocoevalcap7 for image caption. Footnote 7: [https://github.com/salaniz/pyococcevalcap](https://github.com/salaniz/pyococcevalcap)
* ROUGE-1/2/L Lin (2004): We utilize ROUGE-1/2/L from files2rouge8 for text summarization, and employ ROUGE-L from pycocoevalcap9 for image caption. Footnote 7: [https://github.com/mlp3/sacrebleu](https://github.com/mlp3/sacrebleu)
* METEOR Banerjee and Lavie (2005): We utilize METEOR from pycocoevalcap9 for image caption. Footnote 7: [https://github.com/solaniz/pycocoevalcap](https://github.com/solaniz/pycocoevalcap)
* CIDEr Banerjee and Lavie (2005): We utilize CIDEr from pycocoevalcap9 for image caption. Footnote 8: [https://github.com/oltrdy/files2rouge](https://github.com/oltrdy/files2rouge)
* SPICE Banerjee and Lavie (2005): We utilize SPICE from pycocoevalcap9 for image caption. Footnote 9: [https://github.com/galainz/pycocoevalcap](https://github.com/galainz/pycocoevalcap)
* BERTScore Zhang et al. (2020): We utilize BERTScore from its official repository10 for machine translation, text summarization, and image caption. Specially, we leverage Roberta-large for English reference sentences, while apply bert-base-multilingual-cased for other languages (_i.e.,_ German and Russia). Footnote 10: [https://github.com/thompson/prism](https://github.com/thompson/prism)
* MoverScore Zhao et al. (2019): We utilize MoverScore from its official repository11 for text summarization. Specially, we leverage the MNLI-BERT checkpoint. Footnote 11: [https://github.com/AltriBES/emlp19-moverscore](https://github.com/AltriBES/emlp19-moverscore)
* BLEURT Sellam et al. (2020): We utilize BLEURT from its official repository12 for machine translation. Specially, we leverage the BLEURT-20 checkpoint. Footnote 12: [https://github.com/google-research/bleurt](https://github.com/google-research/bleurt)
* Prism Thompson and Post (2020): We utilize Prism from its official repository13 for machine translation. Footnote 13: [https://github.com/thompson/prism](https://github.com/thompson/prism)
* COMET Rei et al. (2020): We utilize COMET from its official repository14 for machine translation. Specially, we leverage the Unbabel/wmt22-comet-da checkpoint. Footnote 14: [https://github.com/Unbabel/COMET](https://github.com/Unbabel/COMET)
* BARTSCore Yuan et al. (2021): We utilize BARTScore from its official repository15 for machine translation in the Chinese-to-English direction. Specially, we leverage the BARTscore+CNN+Para checkpoint. Footnote 15: [https://github.com/neulab/BARTScore](https://github.com/neulab/BARTScore)
* GEMBA Kocmi and Federmann (2023): We utilize GEMBA-Dav3-DA from its official repository16 for machine translation. Specially, we leverage direct assessment as the scoring task, and apply text-davinci-003 as the evaluation model with temperature=0. Footnote 16: [https://github.com/MicrosoftTranslator/GEMBA](https://github.com/MicrosoftTranslator/GEMBA)
* ChatGPT-eval Wang et al. (2023): We utilize ChatGPT-eval (Stars w/ ref) from its official repository17 for text summarization. Specially, we leverage the star prompt with reference, and apply gpt-3.5-turbo as the evaluation model with temperature=0. Footnote 17: [https://github.com/krystallan/chatgpt_as_nlg_evaluator](https://github.com/krystallan/chatgpt_as_nlg_evaluator)
 



\begin{table}
\begin{tabular}{l l} \hline \hline
**Source** & \\ I sincerely hope you get to find a resolution \\ \hline
**Ground-truth reference** & Ich hoffe wirklich, dass Sie eine Lösung finden werden \\ \hline
**Diversified references** & Ich drücke die Daumen, dass Sie eine Lösung ausarbeiten können. \\ Ich hoffe wirklich, dass Sie eine Lösung finden werden. \\ Ich hoffe, dass Sie eine Lösong finden werden, wirklich. \\ Ich hoffe wirklich, dass eine Lösung von Innen gefunden werden wird. \\ Ich hatte gehofft, dass Sie eine Lösung finden wurden. \\ Ich hoffe, dass Sie eine Lösung finden werden \\ Ich wünsche Ihnen aufrichtig, dass Sie eine Lösung finden werden. \\ Ich wunsche mir innigst, dass Sie eine Lösung finden werden. \\ Ich hoffe aufrichtig, dass Sie eine Lösung finden werden. \\ Ich hoffe wirklich, dass due eine Lösung findest. \\ \hline \hline \end{tabular}
\end{table}
Table 6: The diversified example of WMT22 Metrics Task in the Chinese-to-English direction. More examples can be found at [https://github.com/RUCAIBox/Div-Ref](https://github.com/RUCAIBox/Div-Ref).

\begin{table}
\begin{tabular}{l} \hline \hline
**Source** \\ I sincerely hope you get to find a resolution \\ \hline
**Ground-truth reference** \\ Ich hoffe wirklich, dass Sie eine Lösung finden werden \\ \hline
**Diversified references** \\ Ich drücke die Daumen, dass Sie eine Lösung ausarbeiten können. \\ Ich hoffe wirklich, dass Sie eine Lösung finden werden. \\ Ich hoffe, dass Sie eine Lösung finden werden, wirklich. \\ Ich hoffe wirklich, dass eine Lösung von Innen gefunden werden wird. \\ Ich hatte gehofft, dass Sie eine Lösung finden wurden. \\ Ich hoffe, dass Sie eine Lösung finden werden \\ Ich wünsche Ihnen aufrichtig, dass Sie eine Lösung finden werden. \\ Ich wünsche mir innigst, dass Sie eine Lösung finden werden. \\ Ich hoffe aufrichtig, dass Sie eine Lösung finden werden. \\ Ich hoffe wirklich, dass du eine Lösung findest. \\ \hline \hline \end{tabular}
\end{table}
Table 7: The diversified example of WMT22 Metrics Task in the English-to-German direction. More examples can be found at [https://github.com/RUCAIBox/Div-Ref](https://github.com/RUCAIBox/Div-Ref).





**Source**

I see it all the time in my line of work.

\begin{table}
\begin{tabular}{l} \hline
**Source** \\ I see it all the time in my line of work. \\ \hline
**Ground-truth reference** \\ \(A\) postoionino yuzku takoe v svoei skere deitel’nostri. \\ \(B\) postoionno yuzku takoe v svoei skere deitel'nosti. \\ \(B\) postoionno yuzku toki skere deitel’nostri. \\ \(B\) postoionno yuzku takoe v svoei skere deitel'nosti. \\ \(B\) vsroei skere deitel’nosti i akato nablodaka poldobone. \\ \(B\) veseda nablodano poldobnoe v svoei skere raboty. \\ \(B\) vsroei skere deitel’nosti a nepreryvno nablodano poldobnoe. \\ \(B\) postoionno nablodano poldobny vehtsi v svoei skere prodersional'noi deritel’nosti. \\ \(B\) veseda zto nablodano v svoei rabote. \\ \end{tabular}
\end{table}
Table 8: The diversified example of WMT22 Metrics Task in the English-to-Russian direction. More examples can be found at [https://github.com/RUCAIBox/Div-Ref](https://github.com/RUCAIBox/Div-Ref).