NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
Year: 2023

Abstract:
In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remain unclear. We present metric preference checklist as a framework to assess the effectiveness of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation. We show that automatic metrics provide a better guidance than human on discriminating system-level performance in Text Summarization and Controlled Generation tasks. We also show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly in Controlled Generation tasks. 1

heading: 1 Introduction
text: Natural Language Generation (NLG) refers to an automatic process to generate texts in one or more language categories that satisfy multiple desirable human-like qualities. For example, in Text Summarization (Novikova et al., 2017; Maynez et al., 2020; Bhandari et al., 2020; Fabbri et al., 2021), NLG
1Our code is available at https://github.com/inimah/metricpreference-checklist.
systems are expected to produce coherent, consistent, fluent, and relevant summarization outputs. In Dialogue Response Generation (See et al., 2019), the system outputs are mainly assessed based on aspects that are important in a typical human conversation, such as naturalness and engagingness. In Controlled Generation (Dathathri et al., 2020), the generation outputs are evaluated based on its relevance to the predefined topic category or sentiment category as control attributes.
A standard evaluation protocol in NLG for assessing the above human-like qualities involves conducting a human evaluation study or running an automatic evaluation, or both ways. A human evaluation study improves the reliability of evaluation process, particularly when the assessment is done by experts. It is also often infeasible to translate human evaluation aspects into an automatic statistical metric formulation due to its multi-dimensional abstractive properties (Birch et al., 2013; Hashimoto et al., 2019). However, human evaluation is known to be more costly and does not scale well (Howcroft et al., 2020; Freitag et al., 2021). Utilizing automatic metrics, on the other hand, is cost-effective and more feasible for large-scale evaluation data.
Recent works on automatic NLG evaluation metrics, such as CTRLEval (Ke et al., 2022), CTC (Deng et al., 2021), and UniEval (Zhong et al., 2022), have made progress in improving the correlation between automatic metrics and human up to 43% by developing human-aligned automatic metrics. Despite the advancements, there is a need for a standardized framework to assess the utility of these metrics in the context of discerning systemlevel performance. The reason is that an overall correlation score to human does not necessarily represents the metric effectiveness as an evaluation tool, as demonstrated by previous analysis studies on NLG automatic metrics (Caglayan et al., 2020; Hanna and Bojar, 2021; Sai et al., 2021, 2022). However, none of these works connect the correla-
1240
tion analysis to the metric effectiveness at addressing the main objective of NLG benchmarking. That is, for distinguishing system-level performance.
Our study addresses the above research gap by designing a metric preference checklist for measuring the effectiveness of automatic metrics in three NLG tasks: Text Summarization (TextSumm), Dialogue Response Generation (DiagGen), and Controlled Generation (CtrlGen). We introduce three types of assessment for evaluating NLG automatic metrics: Transfer experiment, Aspect-level evaluation, and System-level evaluation. The implications of our study are threefold:
• Verifying the faithfulness of automatic metrics to human preference is a necessary component for a more accurate interpretation of evaluation outcomes (section §6.1).
• Automatic metrics can be more discriminating than human (section §6.2).
• Benchmarking NLG systems via pairwise comparison provides more insights into the strengths and limitations of the systems w.r.t. desirable human-like qualities (section §6.3).

heading: 2 Related Work
text: Existing automatic metrics in NLG are mainly dominated by task-agnostic metrics – metrics that assess the quality of generation outputs without considering human evaluation aspects as context or
objective of the evaluation task (Sai et al., 2022). Task-agnostic metrics are highly adaptable across NLG tasks because the adaptation does not require task-specific design. For example, BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004a), which represent string-based metrics, are largely adopted in Neural Machine Translation (NMT) and Text Summarization. Perplexity (Jelinek et al., 1977; Brown et al., 1992) – a reference-less metric, is a standard evaluation metric in a Language Modeling-based NLG tasks, including Controlled Generation (Keskar et al., 2019; Dathathri et al., 2020). BERTScore (Zhang* et al., 2020) has been largely adopted in diverse NLG tasks, including NMT (Colombo et al., 2022), Text Summarization (Deutsch and Roth, 2021), and Dialogue System (Yeh et al., 2021). Nevertheless, currently available task-agnostic metrics are weakly correlated to human judgment (Novikova et al., 2017; Sai et al., 2021, 2022). A low correlation score introduces a criticism on the capability of automatic metrics at identifying the different quality of system outputs and their potential usage to substitute a costly human evaluation study.
Recent works (Deng et al., 2021; Ke et al., 2022; Zhong et al., 2022) have demonstrated that incorporating desirable human-like qualities as a training objective or contextual knowledge is the best-fit solution for improving the correlation level between automatic metrics and human. However, verifying whether a higher correlation represents a higher
human preference for ranking the quality of system outputs and ranking system performance, and vice versa, remains an underexplored query. Compared to the recent analysis studies that focus on validating the robustness (Caglayan et al., 2020; Hanna and Bojar, 2021; Chen et al., 2021; Vu et al., 2022), explainability (Kaster et al., 2021), reproducibility (Chen et al., 2022), and fairness (Sun et al., 2022) of the NLG evaluation metrics, our study focuses more on a systematic assessment by connecting the link between correlation score to the practical use of the metrics in NLG evaluation. That is, (i) for discriminating the system outputs based on desirable human-like qualities; and (ii) for ranking system performance.

heading: 3 Metric Preference Checklist
text: A standard evaluation protocol in NLG involves validating automatic metrics based on their correlation to human. Intuitively, a high correlation suggests a high agreement on discerning the quality of system outputs because low-quality outputs are penalized with lower scores, while high-quality outputs are rewarded with higher scores. However, currently available metrics are known to have a poor correlation to human. So, it is unclear to what extend current automatic metrics are capable of (i) identifying human-like quality of system outputs and (ii) discriminating performance between independent NLG systems.
To further investigate the above questions, we pose several relevant research questions as a metric preference checklist, as presented in Table 1. We define the assessment tasks for evaluating NLG automatic metrics into five (5) fine-grained aspects, as follows:

heading: 3.1 Transfer Experiment (Zero-Shot)
text: The assessment is designed to investigate whether the correlations between automatic metrics and human are consistent across NLG use cases. For measuring the adaptability of automatic metrics in new target domain, we define In-Domain (ID) and Out-of-Domain (OOD) use cases as follows 2:
In-Domain (ID) For learnable or tunable automatic metrics, we define ID data as the dataset in which the metrics are introduced. For example, UniEval (Zhong et al., 2022) is introduced with a subset of data from SummEval (Fabbri et al., 2021)
2We follow the categorization of OOD that is discussed in previous work by Arora et al. (2021).
and Topical-Chat (Mehri and Eskenazi, 2020). For task-agnostic metrics, such as Perplexity, BLEU, ROUGE, and BERTScore; the categorization of ID and OOD data is rather unknown. So, we define ID data based on a common sense perspective on how close a domain is to the NLG domain where the metric is introduced. For example, BLEU is originally introduced for a Neural Machine Translation (NMT) task (Papineni et al., 2002), yet the metric is widely adopted in Text Summarization (TextSumm). Thus, datasets in Text Summarization domain are considered to be ID samples for BLEU metric.
Semantic-Shift OOD Samples are drawn from the same domain or NLG task where the metric is introduced, but they do not necessarily contain overlapped semantic features with ID samples. For example, let consider ID samples {x, y} are drawn from a subset of SummEval and TopicalChat datasets introduced in UniEval benchmarking (Zhong et al., 2022). Semantic-Shift OOD samples are the superset {X,Y }, which are drawn from the original benchmark datasets of SummEval by Fabbri et al. (2021) and Topical-Chat by Mehri and Eskenazi (2020).
Domain-Shift OOD Samples are drawn from a new domain where the human evaluation aspects overlap with ID domain, but the background features are different. For example, CTRLEval (Ke et al., 2022) is firstly introduced and evaluated in a Controlled Generation task. Thus, samples from different NLG use cases, such as Text Summarization and Dialogue Response Generation are considered to be a Domain-Shift OOD samples.

heading: 3.2 System-level Evaluation
text: The task’s objective is to evaluate the effectiveness of the evaluation metrics at discerning the performance difference between independent NLG systems. For quantifying the degree to which the scores produced by automatic metrics are able to discern the performance between two different NLG systems, we utilize Kolmogorov-Smirnov (KS) as a statistical distance metric D:
D(P1, P2) = sup s
|P1(s)− P2(s)|, (1)
where P1 and P2 denote the empirical cumulative density function (cdfs) of scores based on metric M for system A and system B, where D ∈ [0, 1]. s denotes the evaluation scores as random variables
of metric M . D(.) = 0 indicates the two distributions are identical.

heading: 3.3 System-level Preference
text: The standard evaluation protocol in NLG consists of comparing the ranking of the systems based on the averaged evaluation scores. In many use cases, human and automatic metrics are in agreement about the system ranking. However, a prior study in Controlled Generation (Dathathri et al., 2020) shows that the assumption does not necessarily hold. Therefore, we design a task to compare the system ranking between automatic metrics and human as a similarity measure.
Definition 1. System-level preference Let a and b denote two independent NLG systems. We adopt the concept of utility function in human evaluation (Ethayarajh and Jurafsky, 2022) to measure systemlevel preference. The relation a ≺ b means that b is strictly preferred than a if and only if the utility of a < the utility of b:
a ≺ b ⇐⇒ u(a) < u(b). (2) a ≻ b means that a is preferred than b, while a ∼ b means that a and b are indiscernible. In this study, the utility function u(.) is the averaged evaluation scores for a particular NLG system.
Distance Measure To compute preference similarity between two metrics, we adopt Levenshtein
distance, which calculates the minimum number of insertions, deletions, and substitutions required to change one sequence into the other sequence.
di(P̂ , P ) = Lev(P̂ , P ), (3)
where P and P̂ can be expressed as two sequential orders of system-level preference. For example, let consider P = a ≺ b and P̂ = b ≺ a. Then, Levenshtein distance between P̂ and P is 2.
One of the limitations of Levenshtein distance is that the metric mainly calculates number of operations and does not take into account the sequence length differences. For example, the distance between two pairs P1 = {cdabe, abcde} and P2 = {cbed, abcde} are same, 4, even though the two pairs are composed of different sequences. To tackle this issue, we extend the distance metric formulation into a similarity measure by incorporating the total length of both sequences.
Definition 2. Preference similarity The similarity S between the two sequences P1 and P2 can be defined as:
S = (L1 + L2)− 2 ∗ Lev(P1, P2)
(L1 + L2) , (4)
where S denotes the similarity score; L1 and L2 are the length of P1 and P2 respectively. Using the above formula, the similarity between the first example pair P1 = {cdabe, abcde} is 0.2, while the similarity of the second pair P2 = {cbed, abcde} is 0.11.

heading: 3.4 Aspect-level Evaluation
text: NLG evaluation involves addressing qualitative questions, such as “Can the automatic metrics identify aspect-specific quality that is inferred in the generated texts?” For example, a dialogue system that uses the preceding conversation as a context when generating a new question or response can be considered more engaging and more faithful to the context than the system that outputs repetitive responses. Thus, an automatic metric can be considered adequately good if it can discern between low and high-quality system outputs. For measuring the capability of metrics on discerning aspect-level qualities, we utilize Kolmogorov-Smirnov (KS), as described in Eq. 1.

heading: 4 Experiment

heading: 4.1 Datasets and Metrics 3
text: We consider publicly available author-annotated benchmark datasets in three NLG tasks, as listed in Table 2. For automatic metrics, we consider commonly used task-agnostic automatic metrics in NLG evaluation and the recent proposal of humanaligned automatic metrics, as listed in Table 3.

heading: 4.2 Evaluation Setup
text: ID vs OOD samples We classify benchmark datasets as target evaluation data into In-Domain (ID) and Out-of-Domain (OOD) categories, as shown in Table 3. The configuration of the data split is explained in section § 3.1.
Level of quality We split samples in each benchmark dataset into three categories (if applicable) based on their corresponding human ratings: low quality (rating < 3); moderate (rating = 3); and high quality (rating > 3). The split is disjointly applied to each human evaluation aspect.
Easy vs. Hard samples We split samples in each benchmark dataset into two categories: Easy and Hard. First, The rank of systems is obtained by averaging their corresponding human scores. Easy pair is composed of two systems with the large performance difference (e.g. systems with the lowest vs. highest human scores), while Hard pair contains systems with a close performance score.

heading: 5 Results, Analysis, and Discussion

heading: 5.1 Transfer Experiment
text: Figure 1 shows the correlation level between automatic metrics and human ratings across NLG domains (ID and OOD). The result is summarized below.
Low Correlation in transfer experiment. We observe that the correlation level of automatic metrics deteriorates sharply on target datasets with Semantic-Shift OOD and Domain-Shift OOD, particularly for tunable metrics, such as LM-based Perplexity, BERTScore, and human-aligned metrics (CTC, CtrlEval, UniEval). In general, the notably low correlation is observed in Controlled Generation (CtrlGen) task. UniEval’s correlation scores to human are considered moderately high in TextSumm (0.341) and DiagGen (0.298), but
3Details are provided in Appendix.
the metric does not correlate well with human in CtrlGen (0.006). The result suggests the remaining challenges of adapting human-aligned automatic metrics to a new task or domain, regardless whether the target task has similar dimensions of desirable human-like qualities.

heading: 5.2 Aspect-level Evaluation
text: Figure 2-3 shows aspect-level evaluation of automatic metrics in Text Summarization (TextSumm) and Controlled Generation (CtrlGen). Our main observations are as follows:
UniEval performs best in TextSumm Multiaspect human-aligned metric (UniEval) is observed to have superior performance (up to 0.579) at distinguishing between different levels of quality in UniEval-summ and summ-Eval. However, the discriminative power of the metric is less visible in Newsroom and Controlled Generation (CtrlGen) task. In Newsroom, both BLEU and BERTScore are more discriminative than human-aligned metrics (CTC, CTRlEval, UniEval).
BERTScore is comparably good in TextSumm BERTScore has an adequately good discriminative property (KS=0.557) in UniEval-summ, comparable to multi-aspect human-aligned metric (UniEval) with KS=0.579. In Newsroom, BERTScore consistently has a higher performance score in three sample categories (Lo-Hi, Lo-Mod, Hi-Mod) than human-aligned metrics (CTC, CtrlEval, UniEval). The finding suggests that the characteristics of datasets in Text Summarization domain adequately fit with automatic metrics based on semantic similarity of text embeddings.
Higher KS is not necessarily highly agreeable Perplexity has the highest KS score for distinguishing between low and high quality outputs in UBER data. In contrast, the metric’s aspect-level preference is not in alignment with human.

heading: 5.3 System-level Evaluation
text: Figure 4-6 show the effectiveness of the metrics at discerning system-level performance. Our main observations are as follows:
BLEU is more discriminative in Newsroom In general, apart from BLEU in Newsroom, the remaining metrics’ KS scores across three NLG tasks are considered low-to-moderate (≤ 0.6). We further inspect the reason why BLEU performs considerably well in Newsroom and discover that the data is mainly composed of outputs from two types of NLG systems: extractive vs. abstractive summarization systems. We also observe that in the Newsroom dataset, abstractive systems are often voted lower (averaged score = 2.5) than extractive systems (averaged score =3.85). Such characteristic of human ratings in Newsroom is a good fit for surface-level metric (BLEU), because the metric is more likely to penalize abstractive systems with zero score (0.0) and extractive systems with a higher score (e.g. 1.0).
Automatic metrics are more discriminating than human When human struggles to distinguish between different system-level performances, automatic metrics are observed to be more discriminative. For example, in UniEval-summ (Hard), human has a considerably low score (KS =0.145), while UniEval has a higher KS score (KS =0.269).
In Newsroom (Hard), BLEU, BERTScore, and UniEval are more discriminative (KS > 0.4) than human (KS=0.163). The possible reason for this particular use case is that Hard sample pairs are mainly composed of systems from a similar source or origin. For example, in Persona-Chat (USR-PC), the Hard sample category is composed of a pair of human reference systems: Original Ground Truth, New Human Generated. In Newsroom, Hard sample pairs consist of models from the same category (e.g. extractive-based systems). In UBERTopic, where low KS scores are more visible across human and automatic metrics, both Easy and Hard pairs consist of systems that are derived from one pretrained Language Model.
Multi-aspect human-aligned metric is not always dominant In Persona-Chat (USR-PC), a single aspect human-aligned metric (CTC) has a higher KS score (0.386) and higher preference similarity (0.888) than a multi-aspect metric (UniEval), in which KS =0.218 and similarity=0.833. In
UBER-Topic, UniEval has the lowest KS score (0.025 for Easy pairs, 0.027 for Hard pairs). We find that the less distinctiveness of UniEval is mainly due to a high alignment between UniEval and multi-dimensional human evaluation aspects. For example, in Persona-Chat (USR-PC), the agreement between human evaluation aspects is low. The three aspects (Understandable, Natural, and Engaging) yield a different system rank than the remaining aspects. Thus, a high alignment to interaspect disagreement may necessarily introduce a lower KS.

heading: 5.4 Visualizing Pairwise System Ranking
text: We compare pairwise win fractions of NLG systems based on human ratings and automatic metrics in this study. The objectives are: (i) to better reason on why automatic metrics are more discriminating than human and (ii) to inspect the agreement level between metrics on system ranking.
Notice that the results of pairing evaluation, as
shown in Figure 7, are consistent with our empirical findings in Figure 4-6, particularly for preference similarity with human. The system rankings based on BERTScore F1 and single-aspect CTC metrics are more similar to human on Relevance. Perplexity is more discriminating than human, but its similarity to human (Fluency) is lower. We also observe that although automatic metrics are more discriminating than human ratings
in general, human voting on Relevance aspect can discern system-level performance more effectively than BERTScore and CTC-E Relevance. The result suggests that although a binary voting scheme in a human evaluation study may be less insightful than rating or error correcting protocol, the approach is cost-effective for performance selection based on a particular evaluation aspect.

heading: 6 Implications

heading: 6.1 Faithfulness to Human Preference
text: We show that both low correlation scores and low discriminative power (KS scores) do not represent low faithfulness to human preference. In Controlled Generation, we observe that metrics with lower correlation and lower KS score, such as BERTScore-F1 and single-aspect CTC, on the contrary have a higher similarity with human on system-level preference and ranking. The result suggests the importance of verifying the metric’s correlation score to its faithfulness to human preference, particularly for NLG use cases with poor correlation score (e.g. ρ < 0.2) and low agreement on system ranking.

heading: 6.2 Discriminating System-level Performance
text: We show that automatic metrics can be more discriminating than human, particularly when NLG systems are derived from the same training objective or encoding scheme. In contrast, for human evaluation aspect that is measured based on a binary voting scheme, such as Relevance in Controlled Generation, we observe that the scores based on the corresponding aspect are more distinctive than automatic metrics.

heading: 6.3 Guidance to System Selection
text: We show that benchmarking NLG systems and evaluation metrics via pairwise comparison provides more insights into the agreement level for
selecting the best-performed system. Low agreement between metrics on ranking system-level performance suggests at least two scenarios. First, the automatic metrics are not able to capture the human-like qualities inferred in texts as key factors for discriminating system outputs. Second, each metric focuses on a particular evaluation aspect among multi-dimensional human-like qualities. For example, Fluency focuses on penalizing repetition and grammatical errors, while Relevance focuses on measuring the closeness between the generation outputs and the given control attribute (e.g. topic category). For guiding the selection of the best-performed system, the second scenario allows a fine-grained assessment to scrutinize both strengths and limitations of the system based on desirable human-like qualities.

heading: 7 Conclusion
text: We introduce the metric preference checklist as a framework for analyzing the effectiveness of currently available NLG automatic metrics. We show the importance of verifying the preference similarity between automatic metrics and human, regardless of their correlation scores. We also find that automatic metrics are more discriminating than human for discerning system-level performance, except for human evaluation aspect with a binary voting protocol. Lastly, we show the implication of current work on guiding the selection of the best-performed system based on pairwise system ranking.

heading: Limitations
text: Robustness to perturbations Our empirical study does not explore the connection between the discriminative power of automatic metrics based on the proposed metric preference checklist and their robustness to simple perturbations or other natural language phenomena that may occur in texts or NLG use cases.
Metric Fairness (Social Bias) Our study does not include an investigation of metric fairness or social bias issues that may be introduced by Language Model-based NLG evaluation Metrics.
Single-aspect vs. Multi-aspect Our current empirical experiments mainly explore the discriminative power of evaluation metrics in single-aspect experiment setup (section §5.2). It may also be interesting to inspect to what extend the metrics can identify multi-aspect levels of quality, particularly when there exists disagreement between human evaluation aspects. For example, instead of disjointly splitting samples into {low Engagingness, moderate Engagingness, high Coherence}, samples can be divided based on the joint aspects, such as {low Engagingness and low Coherence}.
Universal input-output structure Our experiments are mainly carried on publicly available author-annotated human evaluation benchmark datasets. Thus, we do not guarantee the universal input-output structure and a uniform naming system across datasets or tasks. For example, UniEval - Topical Chat data (UniEval-TC) (Zhong et al., 2022) and USR - Topical Chat (USR-TC) (Mehri and Eskenazi, 2020) use a different naming system for human evaluation aspects, yet the aspects refer to the same dimension of human-like qualities.
Dependency of NLG Systems When comparing outputs from two different NLG systems, the systems are presumably independent. However, in many NLG use cases, this assumption is not fully accurate. For example, in Controlled Generation task, the systems originate from one pretrained Language Model as an encoder model. In inference or decoding stage, the encoder’s probability outputs are used as inputs for multiple decoding schemes, such as the use of Log-Likelihood ranking, distance scoring as filter, etc (Dathathri et al., 2020), yielding n-systems to compare with. As a result of this setup, the generation outputs from these n-systems are often less diverse and less distinguishable than
the outputs from two independent systems that do not share the same encoding scheme or training objective.

heading: Ethics Statement
text: The purpose of this study is not to provide an immutable checklist to define what makes a good NLG evaluation metrics. Instead, the main objective is to introduce an extended perspective on how to assess metric-level performance beyond a correlation analysis. Our empirical experiments are carried on previously reported human evaluation data and NLG use cases under ACL Ethics Policy. Human evaluation datasets are extracted from peerreviewed scientific publications by Mehri and Eskenazi (2020) in ACL 2020; Dathathri et al. (2020) in ICRL 2020; Ke et al. (2022) in ACL 2022; and Zhong et al. (2022) in EMNLP 2022, as we have listed in our Experiment section. Our empirical findings are not necessarily representative for NLG use cases and datasets that are not covered in this study. However, our metric preference checklist can be easily adopted as fine-grained analysis to measure the effectiveness of new NLG automatic evaluation metrics, regardless of their overall correlation scores to human judgments.

heading: Acknowledgment
text: We thank the anonymous reviewers for the constructive feedback, which has greatly improved the final version of the paper. This research has been partially supported by the Dutch Research Council (NWO) and Indonesian Endowment Fund for Education (LPDP) Scholarship under Beasiswa Pendidikan Indonesia (BPI) – ID Number 0003194/SC/D/9/LPDP2016. The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

heading: A Appendix

heading: A.1 Modification Post Reviews
text: We thank reviewers for the constructive feedback. We list the modification of the paper based on reviewers’ suggestion as follows.
• We add the visualization of pairwise system ranking (section §5.4) to accomodate the reviewers’ suggestion on linking the current work to the objectives of NLG evaluation, particularly for reasoning and guiding model selection,
• We add Implications (§6) to improve the clarity of the paper,
• We add Related Work in the main page (section §2) to clarify the positioning of current proposed framework,
• We add Background in Appendix for providing detail information on NLG tasks and automatic metrics used in this study.
• We improve the presentation of the paper by highlighting the core points and the implications of the study for future works. We also correct the grammatical errors found in the manuscript. The revision is particularly done for Abstract, Introduction, Related Work, and Conclusion section.

heading: A.2 Background

heading: A.2.1 NLG Tasks
text: Our empirical study is mainly carried on three (3) NLG tasks: Controlled Generation, Dialogue Response Generation, and Text Summarization.
1. Controlled Generation (CtrlGen) (Dathathri et al., 2020) is firstly introduced as Conditional Language Modeling (Keskar et al., 2019). In a general setup of CTRLGen, NLG systems are mainly trained based on a language modeling objective where the task is to predict next token or word given the preceding sequence of tokens. During inference stage, the trained system is perturbed with an external control attribute (e.g. topics, sentiment labels, aspects of sentiment) to generate texts that are semantically linked to the control attribute. All tasks in CtrlGen can be categorized as open-ended NLG tasks because ground truth human references are not provided by default. The quality of NLG system outputs is defined based on how semantically close the generation outputs to the corresponding control attribute, which can be aligned to several human-likeness aspects, such as coherence, consistency, fluency, and relevance.
End-to-End NLG Systems We measure the performance of the following systems based on previous work on in Controlled Generation task (Dathathri et al., 2020): B: Baseline, unchanged pretrained GPT-2 Language Model. BR: Sampling B r times based on Log Likelihood ranking and distance-based ranking. BC: For each decoding step, update latent representation H̃t based on attribute model log likelihood loss. BCR: Combine approach from BC (updating H̃t) and BR (sampling and output ranking).
2. Dialogue Response Generation (DiagGen) is NLG use case in neural conversational domain, which can be further divided into an investigation of multi-turn dialogue response generation in a Persona Chat domain (See et al., 2019); or single response generation in Topical Chat and Persona Chat domains (Mehri and Eskenazi, 2020; Zhong
et al., 2022). In this study, we focus on the evaluation of the latter category, where the quality of NLG system outputs is mainly assessed based on how good the machine responses to the preceding conversation. The goodness is mainly defined based on several aspects of human-likeness, such as understandability, naturalness, coherence, engagingness, and groundedness.
End-to-End NLG Systems For Persona-Chat dialogue response generation (USR-PC), we compare the performance of the following systems based on (Mehri and Eskenazi, 2020; Zhong et al., 2022): Systems based on pretrained models in ParlAI 4 for CONVAI2 competition (Colombo et al., 2022), i.e. Seq2Seq – a Sequence-to-Sequence model trained on Persona Chat, KV-MemNN – Key Value Profile Memory Network, Language Model – LSTMbased Language Model, Seq2Seq, and human annotated references – Human Generated Old, and Human Generated New. For Topical-Chat (USRTC and UniEval-TC), the systems are: Human annotations – Human Generated Old, Human Generated New, and four systems that origin from Transformers with different decoding systems, such as Nucleus Decoding p = 0.3, Nucleus Decoding p = 0.5, Nucleus Decoding p = 0.7, Argmax Decoding – greedy decoding.
3. Neural Text Summarization (TextSumm) (Grusky et al., 2018; Fabbri et al., 2021) focuses on a compression type of NLG where the main objective is to generate a concise version of texts, yet maintaining the salient information expressed in the document sources. The quality of system outputs is mainly assessed based on human evaluation aspects that fit into the objective of the task, such as coherence, consistency, fluency, and relevance.

heading: End-to-End NLG Systems In Newsroom
text: dataset (Grusky et al., 2018), the systems are divided into Extractive approach:
• TextRank (Mihalcea and Tarau, 2004) – unsupervisedly rank sentences in document to form a summary with an approach similar to Google PageRank ();
• Extractive Oracle (Fragments) – Fragments F(A,S) are sets of shared sequences of tokens in A = ⟨a1, a2, . . . , an⟩ and S = ⟨s1, s2, . . . , sm⟩
4https://github.com/facebookresearch/ParlAI/ tree/main/projects/convai2
Abstractive approach:
• Sequence-to-Sequence (Seq2Seq) / Attention, Tensorflow implementation of (Rush et al., 2015)5
and Mixed approach:
• Pointer Generator (See et al., 2017) with copying (Vinyals et al., 2015; Gulcehre et al., 2016) and coverage (Tu et al., 2016) mechanism;
• Lower Bound (Lede-3) – baseline approach, by copying the first sentence, first paragraph, or first k words as the summary
In summEval dataset, systems are divided into Extractive:
• M1, NEUSUM (Zhou et al., 2018) – scoring and selecting sentences based on hierarchical representation of a document;
• M2, BanditSum (Dong et al., 2018) – contextual bandit approach of summarization where the document is seen as context and the sequence of sentences to be included in the summary as action;
• M3, LATENT (Zhang et al., 2018) – views sentences in document as relevance binary labels of latent variables;
• M4, REFRESH (Narayan et al., 2018) – a reinforcement approach by focusing on combining individually high-scoring sentences;
• M5, RNES (Wu and Hu, 2018) – improving REINFORCE network by combining coherence model and ROUGE scores as a reward;
• M6, JECS (Xu and Durrett, 2019) – scoring possible constituency-based compressed units;
• M7, STRASS (Bouscarrat et al., 2019) – selecting sentences based on the closest embeddings to the document embedding;
and Abstractive:
• M8, Pointer Generator (See et al., 2017) – encoder decoder model where the decoder can generate samples based on the log-likelihood of words in vocabulary or copy words from the sentence source;
5https://modelzoo.co/model/textsum
• M9, Fast-abs-rl (Chen and Bansal, 2018) – improves Pointer Networks with ROUGE-L reward of REINFORCE;
• M10, Bottom-up (Gehrmann et al., 2018) – decoding method with content selection model to restrict the copy attention distribution of pretrained Pointer Generation Network during inference;
• M11, Improve-abs (Kryściński et al., 2018) – augments the decoder with external LSTMbased Language Model and RL-based objective;
• M12, Unified-ext-abs (Hsu et al., 2018) – aligns word-level attention scores of abstractive model with sentence level attention based on the probability outputs of extractive model;
• M13, ROUGESal (Pasunuru and Bansal, 2018) – improves reinforcement approach by using three types of rewards: keyphrase-based salience, entailment-based, and ROUGEbased reward;
• M14, Multi-task (Ent+QG) (Guo et al., 2018) – a multi-task learning approach with question and entailment generation as auxiliary tasks;
• M15, Closed book decoder (Jiang and Bansal, 2018) – introduces copy-less and attention-less decoder on Pointer Generator Network;
• M16, SENECA (Sharma et al., 2019) – combines entity-aware content selection module and abstractive generation module;
• M17, T5 (Raffel et al., 2022) – improves Transformers-based architecture by exploring the limitation of various transfer learning approaches;
• M18, NeuralTD (Böhm et al., 2019) – define RL-based reward function based on 2500 human evaluation outcomes ;
• M19, BertSum-abs (Liu and Lapata, 2019) – extend BERT with document-level encoder;
• M20, GPT-2 (Ziegler et al., 2019) – finetune GPT-2 on human summaries with a reinforcement learning framework;

heading: A.2.2 Types of automatic Metrics
text: Task-agnostic metrics Task-agnostic metric refers to a category of NLG evaluation metric that does not need task-specific design or contextual knowledge prior to its utilization in a new NLG task.
• Surface-level refers to automatic metrics that mainly assess the quality of system outputs based on word-overlapping or string-based matching techniques between the generation outputs and human-generated references. Our study specifically focuses on two surfacelevel-based similarity metrics: Bilingual Evaluation Understudy (BLEU) (Papineni et al.,
2002) – computes n-gram precision of the generation outputs w.r.t. the corresponding ground truth references; Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004b) – measures how good the system at recalling n-grams from human text references;
• Semantic similarity refers to metrics that measure the similarity between system outputs and text references based on the distance of textual features X in an embedding space X ∈ R. In many cases, the mapping from texts to the corresponding vector representations R requires a Deep Neural Network as an encoder, such as by utilizing pretrained Language Models (BERT) (Devlin et al., 2019) or word embeddings (Bengio et al., 2003; Mikolov et al., 2013a,b). In this study, we focus on investigating BERTScore (Zhang* et al., 2020) to assess to what degree the generation outputs are similar to the given contexts (e.g. text sources, reference summaries, contextual knowledge, or control attributes);
• Language Model-based metric refers to evaluation metric that define the quality of generation outputs by linking the outputs to the surprisal score of an independent pre-trained Language Model – where the surprisal of a word is mainly described as the negative logarithm of the word probability given preceding context words. Perplexity (Brown et al., 1992) is an example of automatic evalution metric that is defined based on the entropy of Language Model. Given machine-generated texts as the inputs of a pretrained LM (e.g. GPT-2), Perplexity scores are the exponents of Negative Log-Likelihood (NLL) of the inputs;
Human-aligned metrics refers to automatic metrics that translate multi-dimensional explainable human evaluation aspects (e.g. Coherence, Consistency) into measureable statistical features of an evaluation metric. We further classify humanaligned automatic metrics into two categories as follows:
• Single-aspect views multi-dimensional human-like aspects or qualities as independent entities.
– CTC (Deng et al., 2021) – is an automatic metric that the main objective is to align information between input, context, and output texts in Compressionbased NLG (Summarization), Transduction-based NLG (Style Transfer), and Creation-based NLG (Dialogue Response Generation). The alignment function is estimated by Embedding Matching (E), Discriminative Model (D), and Aggregated Regression (R). For example, in a compression task, Consistency aspect is described as the average of the alignment score (fE(.), fD(.), or fR(.)) between the summarization outputs y and the source x. Although CTC metric assesses the quality of system outputs based on multiple human evaluation aspects, the aspects are measured independently. Recent report () also discloses that CTC scores are bias to particular human-like aspect or quality. For example, CTC-E Consistency is highly correlated to consistency score based on human ratings, but it cannot explain the other human evaluation aspects. Therefore, our study classifies the metric as single-aspect human-aligned metric;
– CtrlEval (Ke et al., 2022) – is unsupervised reference-less metric in Controlled Generation (Dathathri et al., 2020). The metric translates three human evaluation aspects: Consistency, Coherence, Relevance into a Text Infilling objective. That is, given the input I = (X, a, Y ) consisting of prefix sentence X , control attribute a, and the generation output Y , the score is calculated by projecting pair of sequences from I to N -number of pattern evaluators, where each pattern evaluator’s score is estimated by the log probability outputs of pretrained model.;
• Multi-aspect introduces a unifying perspective of multi-aspect human-like qualities via multi-task and continual learning objectives.
– UniEval (Zhong et al., 2022) – re-frames evaluation aspect as a Boolean Question Answering (QA) objective. For example, for a Coherence aspect, given a summarization output and the corresponding
document source, the metric calculates the performance score by modeling a binary classification task (Yes/No) for a question “Is this a coherent summary of the document?”. Given n-multi dimensional aspects d = (d1, . . . , dn), the generation outputs x, reference texts y (if applicable), and context c, the quality of the system outputs is measured based on the probability of the system generating words that can be either classified as positive and negative samples for addressing question qi:
si = P (“Yes”|x, y, c, qi)
P (“Yes”|x, y, c, qi) + P (“No”|x, y, c, qi) (5)

heading: A.3 Assessment setups

heading: Data Preprocessing
text: • summEval, Newsroom, UniEval-summ (TextSumm) – We use standard data preprocessing: we remove punctuation and nontextual (i.e. numeric and abbreviation) features; we also substitute latin abbreviation, such as i.e. to id est and e.g. to exempli gratia; prior to using the data to calculate the scores based on Perplexity, CTC, CtrlEval, and UniEval metrics. Specific to CtrlEval, we mainly utilize tf-idf weights in (Ke et al., 2022) 6, but we additionally generate relevant prompt and verbal dictionary for the summarization task. as shown in Table 4.
• USR-PC, USR-TC, UniEval-TC (DiagGen) – Specific to CTC-based evaluator, the format of references (list of personas) as relevancebased attribute is adjusted accordingly to follow the input-output structure of the pretrained evaluator. That is by transforming lineseparable personas into a single line of text input separated by a character “||”.
• UBER-Topic, CTRL-Topic, CtrlEval-Topic (CtrlGen) – Data preprocessing follows the procedur in Text Summarization task. Since the nature of benchmark datasets in Controlled Generation is reference-less and openendedness - no human-generated texts as
6https://github.com/thu-coai/CTRLEval
ground truth references, we use the concatenation between control attribute (topic category, such as “Science”) and its corresponding list of relevant keywords as a means of reference.
References and Human-like Aspects Our study uses the following frame of references, which are dependent to the target NLG evaluation task or benchmark dataset and the characteristic of automatic metrics:
• summEval (TextSumm) – The dataset uses n-references (n = 11) as ground truth humangenerated summaries. For each system output and the corresponding references, the score based on BLEU, ROUGE, BERTScore, and human ratings (Coherence, Consistency, Fluency, Relevance) are already included in dataset. For BLEU, ROUGE, and BERTScore, we average the metric scores based on 1-reference and 11-references.
Our work additionally compute the scores based on Perplexity, CTC, CtrlEval, and UniEval metrics. Perplexity mainly uses the system’s outputs as the input x of the metric. For CTC, we use 1-reference only as the ground truth target and average the scores based on embedding-based CTC (CTCE), discriminator-based CTC (CTC-D), and regressor-based CTC (CTC-R) w.r.t. the two aspects of evaluation: “Consistency” and “Relevance”. The inputs for CTC metric are x = {docs, hypos, refs} – where docs denotes document source to be summarized, hypos denotes the system’s generation outputs, and refs is ground truth human-generated summaries.
For CtrlEval and UniEval, we use 11- references as evaluation target for the metrics. For CtrlEval, the performance score is computed based on “Coherence” aspect by solely utilizing the system outputs as the input sources for pretrained GPT-2.
For UniEval, the evaluator is pretrained on summarization task for assessing four aspects: “Coherence”, “Consistency”, “Fluency”, and “Relevance”. For assessing “Coherence” and “Consistency” aspects, UniEval uses document source and the system outputs as the inputs for pretrained evaluator. The system outputs is used solely as inputs for measuring “Fluency”, while the generation
outputs and ground truth references are compared for measuring “Relevance” aspect.
• Newsroom (TextSumm) – The evaluation setup for Newsroom dataset is similar to summEval, except that Newsroom does not include ground truth human references. Instead, the title of articles is used as a means of reference for assessing the quality of system outputs.
• UniEval-summ (TextSumm) – is a subset of summEval. Therefore, the evaluation setup follows the configuration in summEval data.
• USR-PC (DiagGen) – is composed of three source of textual inputs for the evaluation metrics: persona of the model (NLG system) and human evaluators as a background knowledge (fact), the preceding dialogue as a context, and the system responses (generation outputs).
BLEU, ROUGE are computed by comparing between the system responses and the concatenation of document source and factual or contextual knowledge (i.e. list of personas in USR-PC and document title in USR-TC). While, BERTScoreis computed by comparing between system’s responses and document sources.
CTC scores are measured based on “Engagingness” and “Groundedness” (Use Knowledge) aspects, two aspects out of total five aspects based on human ratings (Understandable, Natural, Maintains Context, Engaging, Use Knowledge). CTC-based engagingness is measured by utilizing (i) the concatenated version of factual knowledge (personas) and dialogue history, and (ii) system responses as inputs to be compared. While, CTC-based groundedness measures the relevance of information by inspecting how the system responses comply with the predefined factual knowledge.
CtrlEval scores are measured based on “Coherence”, “Consistency”, and “Relevance” aspects. CtrlEval-Coherence uses the concatenation of dialogue history and system response as input. CtrlEval-Consistency measures how consistent the system response w.r.t. the prefix or dialogue history. While, CtrlEvalRelevance compares the degree of relevance
between the generated responses and the predefined personas.
UniEval scores are computed based on human evaluation aspects included in USR-PC data: UnieEval-Understandability, UniEvalNaturalness, UniEval-Coherence, UniEvalEngagingness, UniEval-Groundedness, and UniEVal-Overall; given dialogue histories as source, list of personas as contextual knowledge, and the system responses as output to be evaluated.
• USR-TC, UniEval-TC (DiagGen) – The main difference between USR-TC and USRPC is that the two benchmarks use different factual knowledge as a means of reference for model or metric. In USR-PC, the reference is the predefined list of model and human personas as multi-turn agents in a dialogue system. While, in USR-TC, the predefined knowledge-grounded conversation is used as a means of reference for evaluating systems and metrics in this study.
• UBER-Topic, CTRL-Topic, CtrlEval-Topic (CtrlGen) – are mainly composed of prefixes, the perturbed version of generation outputs, and control attributes (i.e. topic categories) as textual inputs for the evaluation metrics. The contextual knowledge is constructed by concatenating topic category as control attribute for each prefix sample and the corresponding list of keywords as a pointer to particular topic or domain.
BERTScore is defined based on the comparison between the system’s generated outputs and the control attributes as contextual knowledge. For each system output, we construct

heading: A.4.1 Transfer Experiment

heading: A.4 Experiment Results
text: tasks (ID and OOD).
Sample Analysis In this section, we sample data in In-Domain (ID) and Out-of-Domain subsets to further analyze the contexts in which automatic metrics are not in alignment with human judgments. The samples are mainly grouped based on the agreement-level of multi-aspect human ratings (low vs. high) across ID and OOD subsets (Figure 1a) and NLG use cases (Figure 1b).

heading: A.4.2 Aspect-level Evaluation
text: Figure 9 shows Kolmogorov-Smirnov (KS) scores for aspect-level evaluation in Dialogue Response Generation (DiagGen) and the corresponding similarity score to human preference.

heading: A.4.3 System-level Evaluation
text: Table 17-19 show Kolmogorov-Smirnov (KS) scores of both human and automatic metrics as a measure of metric’s capability at distinguishing performance differences between independent NLG systems. Table 20-22 show the preference similarity between human and automatic metrics at deciding the performance rank of the systems.

heading: A.5 Packages
text: We use publicly available Python Packages for running the experiments, as listed in Table 9. The prerequisite installation is provided in the shared implementation code.

heading: A.6 Hyperparameters
text: BLEU Package: evaluate, https://huggingface. co/spaces/evaluate-metric/bleu. Parameters: ’brevity_penalty’: 1.0 (default).
ROUGE Package: evaluate, https: //huggingface.co/spaces/evaluate-metric/rouge.
BERTScore Package: evaluate, https: //huggingface.co/spaces/evaluate-metric/ bertscore. Model: “roberta-large_L17_noidf_version=0.3.12(hug_trans=4.25.1)”.
Perplexity Package: evaluate, https:// huggingface.co/spaces/evaluate-metric/perplexity. Model: “gpt2”.
CTC Package: CTC. For Embedding-based alignment (CTC-E), we use BERTAligner/BERT embedding (default). For discriminative alignment (CTC-D), we use “roberta-large”. For regressive alignment (CTC-R), we use BLEURTAligner.
CtrlEval Package: CtrlEval. Model: “google/pegasus-large”. We use default configuration in https://github.com/thu-coai/CTRLEval. We reuse the TfIdf features of the original work. For the other required external knowledge (prompt and verbal list), we adjust accordingly to the objective of target NLG task. The prompt and verbal files are provided in the shared data and code implementation.
UniEval Package: UniEval. We use two types of pretrained evaluators in https://github.com/ maszhongming/UniEval: UniEval-sum and UniEvaldialog. We re-use the multi-dimensional human evaluation aspects of the corresponding pretrained evaluators. We adjust the configuration of inputsoutputs of the evaluators based on the target NLG tasks.

heading: A.7 Computing Resources
text: Experiments were done in computing nodes of a HPC cluster with specifications of 4 GPUs Nvidia Tesla V100 (16GB RAM, 2560 tensor cores, 10480 CUDA cores, compute capability 7.0). 1 CPU Intel Xeon E5-2698v4 @ 2.2GHz (40 hyperthreads, RAM: 256GB).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation section is after Conclusion (Section 7) and before Reference list
3 A2. Did you discuss any potential risks of your work? Yes, The potential risks are included in Limitations and Ethics Statement.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract. Introduction (Section 1)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4.1. Datasets are listed and accompanied by the citation of the original paper in Table 2.
3 B1. Did you cite the creators of artifacts you used? section 4.1 and Table 2.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Limitations, Ethics Statement, and Appendix section A.2 (Background)
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Limitations, Ethics Statement, and Appendix section A.2 (Background)
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Limitations, Ethics Statement, and Appendix section A.2 (Background), Appendix A.3
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix section A.2 (Background)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 2
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section A.5 Appendix: Packages.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: • We improve the presentation of the paper by highlighting the core points and the implications of the study for future works. We also correct the grammatical errors found in the manuscript