Venue: NAACL 2024

# X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented

Instruction Tuning with Auxiliary Evaluation Aspects

 Minqian Liu\({}^{\spadesuit}\) Ying Shen\({}^{\spadesuit}\) Zhiyang Xu\({}^{\spadesuit}\) Yixin Cao\({}^{\clubsuit}\)

**Eunah Cho\({}^{\heartsuit}\) Vaibhav Kumar\({}^{\heartsuit}\) Reza Ghanadan\({}^{\heartsuit}\) Lifu Huang\({}^{\spadesuit}\) \({}^{\spadesuit}\)Virginia Tech \({}^{\heartsuit}\)Amazon Inc. \({}^{\spadesuit}\)Fudan University  {minqianliu, yings, zhiyangx, lifuh}@vt.edu {eunahch, kvabh, ghanadan}@amazon.com caoyixin2011@gmail.com**

###### Abstract

Natural Language Generation (NLG) typically involves evaluating the generated text in various aspects (e.g., consistency and naturalness) to obtain a comprehensive assessment. However, multi-aspect evaluation remains challenging as it may require the evaluator to generalize to any given evaluation aspect even if it's absent during training. In this paper, we introduce X-Eval, a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. X-Eval consists of two learning stages: the vanilla instruction tuning stage that improves the model's ability to follow evaluation instructions, and an enhanced instruction tuning stage that exploits the connections between fine-grained evaluation aspects to better assess text quality. To support the training of X-Eval, we collect AspectInstruct, the first instruction tuning dataset tailored for multi-aspect NLG evaluation spanning 27 diverse evaluation aspects with 65 tasks. To enhance task diversity, we devise an augmentation strategy that converts human rating annotations into diverse forms of NLG evaluation tasks, including _scoring, comparison, ranking_, and _Boolean question answering_. Extensive experiments across three essential categories of NLG tasks: dialogue generation, summarization, and data-to-text coupled with 21 aspects in meta-evaluation, demonstrate that X-Eval enables even a lightweight language model to achieve a comparable if not higher correlation with human judgments compared to the state-of-the-art NLG evaluators like GPT-4. 1

Footnote 1: The source code, model checkpoints, and datasets are publicly available at [https://github.com/VT-NLP/XEval](https://github.com/VT-NLP/XEval) for research purposes.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8560–8579 June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Recent advancements of pre-training Chung et al. (2022); Touvron et al. (2023a, b), prompting Brown et al. (2020); Wei et al. (2022b); Wang et al. (2023); Yao et al. (2023); Qi et al. (2023), and instruction tuning Wei et al. (2022a) have improved the quality of machine generated texts by a significant degree. Nevertheless, the evaluation of various Natural Language Generation (NLG) tasks still lags far behind compared with the rapid progress of large language models (LLMs). Previous similarity-based metrics such as ROUGE Lin (2004), BLUE Papineni et al. (2002), METEOR Banerjee and Lavie (2005), and BERTScore Zhang* et al. (2020) predominantly measures the similarity between the generated and reference text, failing to accurately reflect the quality of generated text Gehrmann et al. (2023), especially for open-ended generation tasks.

To obtain a more comprehensive assessment of text quality, multi-aspect evaluation Fabbri et al. (2021) has been proposed to evaluate the generated text from multiple fine-grained evaluation _aspects_, such as fluency and consistency. While most

Figure 1: Illustration of X-Eval for multiple seen and unseen fine-grained evaluation aspects across various NLG tasks. The unseen aspect (i.e., Interestingness) is highlighted in _italics_. The text to be evaluated is highlighted with underline. In this example, each evaluation score is from 0 to 1. The higher score indicates better quality.



existing studies Mehri and Eskenazi (2020); Yuan et al. (2021); Zhong et al. (2022) consider a closed set of aspects, in many realistic scenarios, the users may need to evaluate the text with their customized aspects and specifications, calling for building an evaluator that can be flexibly extended to any _unseen_ aspects without the need of training data. Recent studies Fu et al. (2023); Liu et al. (2023) propose to leverage large language models (LLMs) such as GPT-4 OpencAI (2023) as NLG evaluators, yielding promising zero-shot performance on unseen aspects. However, such evaluations, especially with proprietary LLMs, are cost-intensive, time-consuming, and pose concerns about data privacy and reproducibility.

In this work, we propose X-Eval, an automatic evaluation framework that can conduct fine-grained evaluation on both seen and unseen aspects across various NLG tasks with a single model, as illustrated in Figure 1. X-Eval follows a two-stage training paradigm: we first instruction-finetune an open-source language model to equip it with the capability of following human-written instructions for evaluation. Then, motivated by the observation that evaluation aspects usually exhibit interconnections Fu et al. (2023) and thus their evaluations can benefit each other, we introduce an additional training stage to finetune the model on the instruction-tuning tasks enriched with the evaluations of a set of _auxiliary aspects_, which are expected to provide clues for evaluating the target aspect and encourage consistent evaluations across multiple aspects. During training, for each target aspect, we take all the remaining aspects defined in the corresponding dataset as auxiliary aspects and incorporate their gold evaluations into the instructions for the second-stage tuning. During inference, given the target aspect, we first select a set of auxiliary aspects based on the similarity of the aspect definitions and predict the evaluation result for each auxiliary aspect using the trained model. We then re-perform the evaluation for each target aspect by incorporating the results of auxiliary aspects.

To support our proposed two-stage training of X-Eval, we construct AspectInstruct, the first multi-aspect evaluation instruction tuning dataset spanning 27 diverse evaluation aspects over 65 tasks. This dataset is anchored around three core categories of NLG tasks: dialogue, summarization, and data-to-text. In light of insights from previous studies in instruction tuning Wei et al. (2022); Xu et al. (2023), which emphasize the advantage of task diversity in enhancing zero-short generalization, we further augment the dataset by converting the original human rating data into diverse forms of NLG evaluation tasks, including _scoring, comparison, ranking_ and _Boolean question answering_. In addition, to incorporate auxiliary aspects, we manually create templates that convert the numerical evaluation scores of each aspect into descriptions in natural language.

The main advantages of our approach are highlighted as follows: **(1) Generalization ability:** we introduce X-Eval that can be flexibly generalized to evaluate the unseen NLG tasks or the aspects customized by user instructions in a zero-shot manner with a single model; **(2) Strong performance with high efficiency:** with significantly less amount of model parameters (780M), X-Eval achieves strong performance compared to the state-of-the-art LLM-based evaluators (including GPT-4) demonstrated through comprehensive experiments; **(3) Reference-free and open-source:** our evaluator does not require gold reference to perform evaluation and it is more reliable and transparent thanks to its open-source nature.

## 2 Related Work

Similarity-based MetricsThe previously dominant text evaluation paradigm is to predict a one evaluation score, where most of them are similarity-based metrics, including metrics that measure the surface overlap between the generated and reference text, such as ROUGE Lin (2004), BLUE Papineni et al. (2002), and METEOR Banerjee and Lavie (2005), as well as metrics measuring the distance between the contextualized embeddings of the generated text and the reference as the similarity score, such as BERTScore Zhang* et al. (2020) and MoverScore Zhao et al. (2019). Although these metrics are widely adopted, they often overlook fine-grained aspects and later study Gehrmann et al. (2023) has proven that they fail to truly capture the quality of text with the coarse-grained score.

Multi-Aspect MetricsTo conduct a more holistic evaluation, recent studies Wang et al. (2020); Huang et al. (2020) propose to evaluate the NLG systems via multiple fine-grained aspects. UniEval Zhong et al. (2022) proposes to re-frame NLG evaluation into a QA format and perform multi-aspect evaluation with a single model via continual learning Madotto et al. (2021); Liu et al. ( 

2022; Liu and Huang, 2023). However, UniEval cannot maintain robust performance when generalizing to novel aspects. To obtain an evaluator that can be generalized to customized aspects, some recent studies Fu et al. (2023); Liu et al. (2023) harness proprietary LLMs to perform fine-grained evaluation in a zero-shot manner. However, due to the closed-source nature, these evaluation metrics suffer from issues of reproducibility and are prohibitively expensive. More recently, some concurrent studies Xu et al. (2023); Jiang et al. (2023); Mehri and Shwartz (2023) propose to extract instruction-following data from proprietary LLMs for finetuning a more lightweight model as the evaluator. Nevertheless, they still require high costs to call the APIs to obtain a large amount of training data and it is non-trivial to ensure the data are of high quality. In addition, to the best of our knowledge, we are the first to meticulously curate the instruction-tuning dataset and train an instruction-based evaluator for dialogue evaluation.

## 3 AspectInstruct

### Problem Definition

Multi-aspect automatic text evaluation aims to evaluate the quality of NLG system's output \(x\) given a set of evaluation aspects \(\mathcal{A}\) (e.g., coherence, naturalness and so on), and optionally an additional set of texts \(\mathcal{S}\) (e.g., the source documents for text summarization, or context for dialogue evaluation). The evaluation task can be formulated as:

\[c=f(x,\mathcal{S},a)\]

where \(a\in\mathcal{A}\) is the fine-grained aspect to be evaluated, and \(f(\cdot)\) is the scoring function that provides an assessment \(c\) w.r.t. the aspect \(a\).

### Data Collection

We aim to build a unified automatic evaluation framework that can assess the text quality for both seen and unseen evaluation aspects across various NLG tasks via instruction tuning. To this end, we build an instruction-tuning dataset tailored for multi-aspect evaluation, namely AspectInstruct, with the following steps:

Existing Dataset CollectionWe first collect 10 existing evaluation datasets with human annotations for 3 representative categories of NLG tasks, including dialogue generation Sai et al. (2020); Gunasekara et al. (2020); Pang et al. (2020); Gopalakrishnan et al. (2019); Mehri and Eskenazi (2020), text summarization Volske et al. (2017); Fabbri et al. (2021); Wang et al. (2020); Zhong et al. (2022), and data-to-text Wen et al. (2015).

Task AugmentationThe original datasets we collect only contain numerical scores annotated by humans, which severely limits the diversity of instruction-tuning tasks. Thus, we further derive diverse forms of evaluation tasks from the original annotations to enhance task diversity. Denote the ground truth score for text \(x_{i}\) as \(y_{i}\). We derive four types of tasks based on this annotation: **(1) Scoring:** we ask the model to directly predict a discrete score (e.g., in the Likert scale) where we map the continuous ground truth \(y_{i}\) into a discrete scale; **(2) Comparison:** we sample two texts \(x_{i}\) and \(x_{j}\) for an identical context, e.g., two versions of summaries for the same source document, and ask the model to select the text with the higher evaluation score; **(3) Ranking:** we further extend the comparison task into ranking by sampling three candidates under the same context and ask the model to predict the correct ranking of the candidates based on the text quality; **(4) Boolean Question Answering:** we also formulate evaluation as a Boolean QA task following Zhong et al. (2022) by asking the model a question such as "_Is this response fluent?_" and let the model predict "_Yes_" or "_No_".

Instruction CreationFinally, we define a unified instructions format for tasks included in AspectInstruct. Each instruction consists of three parts: (1) _task description_ that briefly introduces the evaluation task, (2) _aspect definition_, and (3) _evaluation protocol_ that details what the model should output to perform the evaluation. We present the detailed procedure for instruction annotation in Appendix A.1. We provide an example of the original annotation, and the derived evaluation tasks along with the curated instructions in Figure 6 in Appendix A.2. The full list of evaluation aspects and the collected instructions can be found in Appendix A.3.

StatisticsIn total, we construct 65 tasks in AspectInstruct, where we split 32 tasks and 14 seen aspects for instruction tuning and 33 tasks and 13 unseen aspects for meta-evaluation. We collect 72,637 instances in total with 55,602 instances for training and 17,035 instances for inference. Note that there is no overlap among the datasets used for training and inference. We consider two aspects that have identical aspect names but are in 

different NLG tasks as distinct aspects. We include more details about the source datasets, constructed instruction-tuning tasks, and the number of instances of each task in Appendix A.2.

## 4 X-Eval

### Two-Stage Instruction Tuning

Figure 2 presents an overview of X-Eval, which consists of two stages of instruction tuning:

Vanilla Instruction TuningThe first training stage aims to equip the model with the ability to follow instructions to perform diverse evaluation tasks. We adopt Flan-T5 Chung et al. (2022), an open-source language model as the base model for our evaluator. Based on Flan-T5, we further perform standard instruction tuning on the mixture of four types of tasks: _scoring, comparison, ranking_, and _Boolean QA_, as elaborated in Section 3.2.

Instruction Tuning with Auxiliary AspectsThrough our study, we discern that certain evaluation aspects could be interrelated. As evidence, in dialogue evaluation Gopalakrishnan et al. (2019) the aspect naturalness usually shows a notable correlation with engagingness. When a dialogue response is not natural, it is very likely that human considers the response to be not engaging. While these two aspects are not interchangeable given their different definitions, the evaluation of one aspect can offer useful clues for the evaluation of another potentially related aspect. Motivated by this, we enrich our training regimen with an additional instruction tuning stage to leverage potential connections to the target evaluation aspect.

More precisely, for each instruction-tuning task detailed in Section 3.2, we augment it based on the ground truth evaluation results of a predefined set of auxiliary aspects which are all other aspects collected in the source dataset. To convert the evaluation results of auxiliary aspects into natural language that can be fed into the input, we employ a template-based verbalizer, denoted as \(v(\cdot)\), which takes in an aspect \(a\) and its evaluation score \(s\) for an instance, mapping it into a verbalized evaluation \(h=v(s,a)\). For example, with the aspect Consistency on Data2Text and the evaluation score 0.9 out of 1.0, the verbalized result is phrased as "_This sentence is consistent with the source."_ (see more details in Appendix B). We construct the set of verbalized results \({\cal H}\) with the verbalizer for each auxiliary aspect (except for the target aspect). This set \({\cal H}\) is then concatenated into the additional

Figure 2: **Illustration of our X-Eval framework. The left section depicts our two-stage training approach: vanilla instruction tuning on diverse tasks and subsequent training on instruction tasks enriched with auxiliary aspects. The right section illustrates the inference pipeline with auxiliary aspects.**



set of texts in the evaluator's input. The model then undergoes the second training stage on the instruction tasks enriched with these evaluation results.

### Inference with Auxiliary Aspects

At the inference stage, we perform the following steps to evaluate the text on the target aspect: **First**, we select a set of auxiliary aspects for the target aspect. Based on the definitions of the target aspect and a pool of candidate aspects, we employ Sentence-T5 Ni et al. (2022) to encode the definitions and measure the similarity between the sentence embeddings of target aspect definition and each candidate aspect definition. We select the aspects with top-\(k\) similarity scores as the auxiliary aspects to limit inference cost, where \(k\) is a hyperparameter. **Second**, we run an inference process using the Boolean QA task format, where the model predicts either _"Yes"_ or _"No"_, as outlined in Section 3.2, on each auxiliary aspect. We convert the prediction into natural language results with the verbalizer. These verbalized results, denoted as \({\cal H}\), are subsequently integrated into the additional set of texts \({\cal S}\) for evaluating the target aspect. **Finally**, given the input enhanced by auxiliary aspects, we adopt the same Boolean QA format to compute the evaluation score \(c\) for the target aspect:

\[c=\frac{P(``Yes''|x,{\cal S},a)}{P(``Yes''|x,{\cal S},a)+P(``No"|x,{\cal S},a)}\]

where \(P(\cdot)\) denotes the probability of the model generating a specific word. The pseudo-code of our inference pipeline is in Algorithm 1.

```
Input: Set of evaluation aspects \({\cal A}\), Target aspect \(a_{t}\), NLG system's output \(x\), Additional set of texts \({\cal S}\), Scoring function \(f(\cdot)\), Evaluation verbailzer \(v(\cdot)\), Similarity measure \(sim(\cdot)\), Sentence encoder \({\cal E}\) Output:Target score \(c_{t}\)// Determine top-\(k\) auxiliary aspects \(L\leftarrow\left\{(ism({\cal E}(a),{\cal E}(a_{t})),a)~{}|~{}a\in{\cal A} \setminus\{a_{t}\}\right\}\) Sort L in descending order based on similarity \({\cal A}^{R}\leftarrow\text{first $k$ respects from sorted $L$}\) // Generate verbalized evaluation results for auxiliary aspects Initialize me empty auxiliary evaluation set \({\cal H}\)for\(a_{r}\in{\cal A}^{R}\)do// Score for auxiliary aspect \(c_{r}\gets f(x,{\cal S}_{r},a_{r})\) // Add verbalized evaluation to the auxiliary evaluation set \({\cal H}\leftarrow[{\cal H};v(c_{r},a_{r})]\) \({\cal S}_{r}\leftarrow[{\cal S}_{r};{\cal H}]\) // Evaluate the target aspect \(c_{t}\gets f(x,{\cal S}_{t},a_{t})\) return\(c_{t}\)
```

**Algorithm 1**Inference Pipeline

## 5 Experiment Setup

Meta EvaluationWe meta-evaluate our X-Eval on the test split of AspectInstruct, where the details of the test set are introduced as follows. For text summarization, we adopt SummEval Fabbri et al. (2021) and QAGS Wang et al. (2020). For dialogue generation, we employ TopicalChat Gopalakrishnan et al. (2019) and FED Mehri and Eskenazi (2020). For data-to-text generation, we utilize SFHOT & SFRES Wen et al. (2015). AspectInstruct contains the following _unseen_ aspects: topic depth (DEP), likeability (LIK), understandability (UND), flexibility (FLE), informativeness (INF), inquisitiveness (INQ), interestingness (INT), specificity (SPE), correctness (COR), and semantic appropriateness (SEM). More detailed descriptions of the test splits, as well as seen and unseen evaluation aspects, are be found in Appendix A.4.

Implementation DetailsWe adopt Flan-T5-large (with ~780M parameters) as our base language model for subsequent finetuning. Without specification, we pick the top-1 aspect during inference, i.e., \(k=1\). More implementation details can be found in Appendix C.

BaselinesWe compare our X-Eval with the following state-of-the-art NLG evaluation metrics: **(1) UniEval**Zhong et al. (2022) is a unified multi-aspect evaluator that re-frames the evaluation process as a Boolean QA task; **(2) GPTScore**Fu et al. (2023) is a multi-faceted and training-free evaluation framework that utilizes the output probabilities from LLMs to score generated texts; **(3) G-Eval**Liu et al. (2023) proposes to leverage large language models such as GPT-3.5 or GPT-4 to assess the text quality with form-filling paradigm in a training-free manner; **(4) ROUGE-L**Lin (2004); **(5) DynaEval**Zhang et al. (2021); **(6) BERTScore**Zhang* et al. (2020); **(7) MoverScore**Zhao et al. (2019); **(8) USR**Mehri and Eskenazi (2020); **(9) BARTScore**Yuan et al. (2021). We include more details of baselines (4)-(9) in Appendix C due to space limit.

Variants of X-EvalWe design several variants of X-Eval for ablation studies: **(1) X-Eval w/o 



**Training** denotes the original Flan-T5 (without any further finetuning on our proposed AspectInstruct); **(2) X-Eval w/o Instructions**: based on Flan-T5, we only conduct prompt-based multi-task training and inference in the same way as Zhong et al. (2022) where we ask the model to answer Boolean questions without using aspect definitions; **(3) X-Eval w/o Stage-Two Tuning**: for this variant, we only conduct vanilla instruction tuning in Stage 1 based on Flan-T5. During inference, we directly perform evaluation based on the instructions without using auxiliary aspects.

## 6 Main Results

We report the main results of dialogue evaluation in Table 1 and Table 2, summarization in Table 3 and Table 9 , and data-to-text in Table 4. Each table is divided into three sections: the top section delineates the performance of traditional metrics and evaluators based on lightweight language models. The middle section shows the performance of the evaluators based on GPTs Brown et al. (2020); OpenAI (2023) that are proprietary and much larger than our approach. The bottom section shows the performance of X-Eval and its variants.

Results of Dialogue Evaluation on FEDTo assess X-Eval's ability to generalize to _unseen_ aspects, we present the Spearman correlation on FED in Table 1. X-Eval surpasses the baselines in the top section. Also, X-Eval matches the performance of GPT-based baselines with much fewer parameters. The bottom section of the table highlights the improvement achieved by two-stage tuning, incorporating instructions, and integrating auxiliary aspects. It is worth noting that UniEval achieves notably poor performance on dialogue-level evaluation on FED, which is probably due to UniEval being overfitted to turn-level evaluation and failing to generalize to dialogue-level evaluation.

Results of Dialogue Evaluation on Topical-Chat We also evaluate the performance for the _seen_ aspects on Topical-Chat and report the results in Table 2. Notably, in addition to the superior performance over lightweight baselines, X-Eval also

\begin{table}
\begin{tabular}{l|c c c c c c|c c c c c|c c c} \hline \multicolumn{1}{c|}{\multirow{2}{*}{**Metrics**}} & \multicolumn{4}{c|}{**Dialogue-level**} \\  & **DEP** & **LIK** & **UND** & **FLE** & **INF** & **INQ** & **AVG** & **IFT** & **SPE** & **CORR** & **SEM** & **UND** & **AVG** \\ \hline BARTScore Yuan et al. (2021) & 0.082 & 0.099 & -0.115 & 0.093 & 0.092 & 0.062 & 0.052 & 0.159 & 0.083 & 0.076 & 0.100 & 0.120 & 0.128 \\ DynaEul Zhang et al. (2021) & 0.498 & 0.416 & 0.365 & 0.383 & 0.426 & 0.410 & 0.416 & 0.327 & 0.346 & 0.242 & 0.202 & 0.200 & 0.263 \\ UniEval Zhou et al. (2022) & 0.046 & 0.009 & -0.024 & -0.003 & -0.070 & 0.085 & 0.030 & 0.435 & **0.381** & 0.125 & 0.051 & 0.082 & 0.215 \\ \hline GPTScore GPT-3-d01 Ju et al. (2023) & **0.669** & **0.634** & 0.524 & 0.515 & **0.602** & 0.503 & **0.574** & 0.501 & 0.214 & 0.434 & **0.444** & 0.365 & 0.392 \\ GPTScore GPT-3-d03 Fu et al. (2023) & 0.341 & 0.184 & 0.196 & 0.072 & 0.317 & -0.101 & 0.168 & 0.224 & 0.151 & 0.428 & 0.405 & 0.311 & 0.304 \\ G-Eval GPT-3-eject Liu et al. (2023) & 0.339 & 0.392 & 0.123 & 0.344 & 0.232 & 0.101 & 0.259 & 0.30 & 0.280 & 0.430 & 0.390 & 0.274 & 0.335 \\ G-Eval GPT-4tri Tai et al. (2023) & 0.583 & 0.614 & **0.602** & **0.587** & 0.510 & **0.551** & 0.573 & **0.506** & 0.368 & **0.522** & 0.443 & **0.485** \\ \hline X-Eval Ourts & 0.583 & 0.436 & 0.588 & 0.324 & 0.480 & 0.497 & 0.485 & 0.421 & 0.370 & 0.492 & 0.376 & 0.332 & 0.398 \\ - w/o Training & 0.377 & 0.387 & 0.394 & 0.424 & 0.370 & 0.417 & 0.395 & 0.250 & 0.175 & 0.296 & 0.289 & 0.225 & 0.247 \\ - w/o Instructions & 0.350 & 0.333 & 0.495 & 0.355 & 0.425 & 0.435 & 0.399 & 0.477 & 0.353 & 0.203 & 0.255 & 0.211 & 0.300 \\ - w/o Stage-Two Tuning & 0.388 & 0.324 & 0.555 & 0.384 & 0.582 & 0.437 & 0.445 & 0.372 & 0.282 & 0.418 & 0.329 & 0.311 & 0.342 \\ \hline \end{tabular}
\end{table}
Table 1: **Meta-evaluation on dialogue** based on _unseen_ aspects in terms of dialogue-level and turn-level Spearman (\(\rho\)) correlations on FED. The best overall results are highlighted in **bold**. We also highlight the best results excluding GPT-based metrics with underline. †: our re-implementation, where we adopt our annotated instructions and aspect definitions as inputs to OpenAI’s API to obtain the performance of G-Eval on FED.

\begin{table}
\begin{tabular}{l|c c c c|c c c|c c c} \hline \hline \multicolumn{1}{c|}{\multirow{2}{*}{**Metrics**}} & \multicolumn{2}{c|}{**Naturalness**} & \multicolumn{2}{c|}{**Coherence**} & \multicolumn{2}{c|}{**Engagingness**} & \multicolumn{2}{c|}{**Grundedness**} & \multicolumn{2}{c}{**AVG**} \\  & \(r\) & \(\rho\) & \(r\) & \(\rho\) & \(r\) & \(\rho\) & \(r\) & \(\rho\) & \(r\) & \(\rho\) \\ \hline ROUGE-L Lin (2004) & 0.176 & 0.146 & 0.193 & 0.203 & 0.295 & 0.300 & 0.310 & 0.327 & 0.243 & 0.244 \\ BERTscore Zhang et al. (2020) & 0.226 & 0.209 & 0.214 & 0.233 & 0.317 & 0.335 & 0.291 & 0.317 & 0.262 & 0.273 \\ USR Mehri and Eskenazi (2020b) & 0.337 & 0.325 & 0.416 & 0.377 & 0.456 & 0.465 & 0.222 & 0.447 & 0.358 & 0.403 \\ UniEval Zhong et al. (2022) & 0.480 & 0.512 & 0.518 & 0.609 & 0.544 & 0.563 & 0.462 & 0.456 & 0.501 & 0.535 \\ \hline G-Eval (GPT-3.5) Liu et al. (2023) & 0.532 & 0.539 & 0.519 & 0.544 & **0.660** & **0.691** & 0.586 & 0.567 & 0.574 & 0.585 \\ G-Eval (GPT-4) Liu et al. (2023) & **0.549** & **0.565** & **0.594** & 0.605 & 0.627 & 0.631 & 0.531 & 0.551 & **0.575** & 0.588 \\ \hline X-Eval Ourts & 0.417 & 0.478 & 0.558 & 0.622 & 0.449 & 0.593 & **0.734** & **0.728** & 0.540 & **0.605** \\ - w/o Training & 0.054 & 0.051 & 0.063 & 0.073 & 0.258 & 0.298 & 0.427 & 0.436 & 0.200 & 0.214 \\ - w/o Instructions & 0.415 & 0.452 & 0.560 & 0.574 & 0.397 & 0.532 & 0.690 & 0.701 & 0.515 & 0.565 \\ - w/o Stage-Two Tuning & 0.396 & 0.446 & 0.581 & **0.642** & 0.408 & 0.569 & 0.725 & 0.706 & 0.528 & 0.592 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Turn-level Pearson (\(r\)) and Spearman (\(\rho\)) correlations on _seen_ aspects on Topical-Chat. The best overall results are highlighted in **bold**. We also highlight the best results excluding GPT-based metrics with underline.



surpasses all GPT-based metrics in averaged Spearman correlation. We notice that the correlation of X-Eval on groundedness is notably higher than other baselines. One plausible reason is that Flan-T5 has been finetuned on related tasks such as natural language inference Chung et al. (2022), as X-Eval w/o Training has achieved decent performance without finetuning on AspectInstruct.

Results of Summarization EvaluationWe use summary-level Spearman and Kendall-Tau correlation to assess various evaluators on SummEval. Note that all the aspects in SummEval are seen aspects. From Table 3, X-Eval surpasses lightweight evaluators in averaged Spearman correlation and outperforms both GPTScore and G-Eval (GPT-3.5). G-Eval (GPT-4) consistently excels across all aspects. We speculate this may stem from GPT-4's strong ability to handle long input contexts. In addition, we report the results on QAGS in Table 9 in Appendix due to the space limit.

Results of Unseen NLG Task EvaluationIn this experiment, we evaluate X-Eval on the unseen data-to-text generation task. Table 4 shows that while X-Eval experiences a slight performance loss in naturalness compared to G-Eval (GPT-4), it consistently excels over all other baselines across all aspects. This underscores the generalization capability of X-Eval on unseen NLG tasks.

## 7 Discussions

Ablation Study of Instruction Tuning TasksWe conduct ablation studies to investigate the contribution of incorporating diverse forms of evaluation tasks during instruction tuning. Table 5 shows the averaged Spearman correlation on each meta-evaluation dataset. In general, X-Eval trained on the combination of all forms of evaluation tasks, including _scoring, comparison, ranking_, achieves the highest averaged correlation for nearly all tasks.

\begin{table}
\begin{tabular}{l|c c|c c|c} \hline \hline \multirow{2}{*}{**Metrics**} & \multicolumn{2}{c|}{**SFRES**} & \multicolumn{2}{c|}{**SFHOT**} & \multirow{2}{*}{**AVG**} \\  & **NAT** & **INFO** & **NAT** & **INFO** & \multicolumn{1}{r}{**AVG**} \\ \hline ROUGE-L & 0.169 & 0.103 & 0.186 & 0.110 & 0.142 \\ BERTScore & 0.219 & 0.156 & 0.178 & 0.135 & 0.172 \\ MOVERScore & 0.190 & 0.153 & 0.242 & 0.172 & 0.189 \\ BARTScore & 0.289 & 0.238 & 0.288 & 0.235 & 0.263 \\ UniEval (Summ) & 0.333 & 0.225 & 0.320 & 0.249 & 0.282 \\ \hline GTFScore & 0.190 & 0.232 & 0.036 & 0.184 & 0.161 \\ G-Eval (GPT-3.5)† & 0.144 & 0.118 & 0.072 & 0.102 & 0.109 \\ G-Eval (GPT-4+) & **0.351** & 0.189 & **0.338** & 0.198 & 0.269 \\ \hline X-Eval (Ours) & 0.316 & **0.265** & 0.322 & **0.310** & **0.303** \\ - w/o Training & 0.240 & 0.192 & 0.207 & 0.262 & 0.225 \\ - w/o Instructions & 0.303 & 0.255 & 0.297 & 0.277 & 0.283 \\ - w/o Stage-Two Tuning & 0.322 & 0.257 & 0.311 & 0.292 & 0.295 \\ \hline \end{tabular}
\end{table}
Table 4: Spearman correlation on the data-to-text NLG task. NAT and INFO indicate Naturalness and Informativeness, respectively. The best results are highlighted in **bold**. †: our re-implementation.

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c} \hline \hline \multirow{2}{*}{**Metrics**} & \multicolumn{2}{c|}{**Coherence**} & \multicolumn{2}{c|}{**Consistency**} & \multicolumn{2}{c|}{**Fluency**} & \multicolumn{2}{c|}{**Relevance**} & \multicolumn{2}{c}{**AVG**} \\  & \(\rho\) & \(\tau\) & \(\rho\) & \(\tau\) & \(\rho\) & \(\tau\) & \(\rho\) & \(\tau\) & \(\rho\) & \(\tau\) \\ \hline ROUGLE-L Lin (2004) & 0.128 & 0.099 & 0.115 & 0.092 & 0.105 & 0.084 & 0.311 & 0.237 & 0.165 & 0.128 \\ MOVERScore Zhao et al. (2019) & 0.159 & 0.118 & 0.157 & 0.127 & 0.129 & 0.105 & 0.318 & 0.244 & 0.191 & 0.148 \\ BERTSCore Zhang et al. (2020) & 0.284 & 0.211 & 0.110 & 0.090 & 0.193 & 0.158 & 0.312 & 0.243 & 0.225 & 0.175 \\ BARTScore Yuan et al. (2021) & 0.448 & 0.342 & 0.382 & 0.315 & 0.356 & 0.292 & 0.356 & 0.273 & 0.385 & 0.305 \\ UniEval Zhong et al. (2022) & 0.495 & 0.374 & 0.435 & 0.365 & 0.419 & 0.346 & 0.424 & 0.327 & 0.443 & 0.353 \\ \hline GPTScre Fu et al. (2023) & 0.434 & – & 0.449 & – & 0.403 & – & 0.381 & – & 0.417 & – \\ G-Eval (GPT-3.5) Liu et al. (2023) & 0.440 & 0.335 & 0.386 & 0.318 & 0.424 & 0.347 & 0.385 & 0.293 & 0.401 & 0.320 \\ G-Eval (GPT-4) Liu et al. (2023) & **0.582** & **0.457** & **0.507** & **0.425** & 0.455 & **0.378** & **0.547** & **0.433** & **0.514** & **0.418** \\ \hline X-Eval Ours & 0.530 & 0.382 & 0.428 & 0.340 & **0.461** & 0.365 & 0.500 & 0.361 & 0.480 & 0.362 \\ - w/o Training & 0.187 & 0.131 & 0.193 & 0.152 & 0.135 & 0.104 & 0.444 & 0.325 & 0.240 & 0.178 \\ - w/o Instructions & 0.458 & 0.333 & 0.414 & 0.328 & 0.395 & 0.309 & 0.496 & 0.359 & 0.441 & 0.333 \\ - w/o Stage-Two Tuning & 0.536 & 0.385 & 0.413 & 0.326 & 0.455 & 0.360 & 0.503 & 0.363 & 0.476 & 0.359 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Summary-level Spearman (\(\rho\)) and Kendall-Tau (\(\tau\)) correlations of different metrics on SummEval. All aspects are _seen_ aspects. The best overall results are highlighted in **bold**. We also highlight the best results excluding GPT-based metrics with underline.

\begin{table}
\begin{tabular}{l|c|c|c|c|c} \hline \hline \multirow{2}{*}{**Metrics**} & \multicolumn{2}{c|}{**Topic.**} & \multicolumn{2}{c|}{**FED**} & \multicolumn{1}{r|}{**Summ.**} & \multicolumn{1}{r|}{**D2T**} & \multicolumn{1}{r}{**AVG**} \\ \hline X-Eval (w/o STT) & **0.592** & **0.375** & **0.480** & 0.295 & **0.436** \\ - w/o Scoring & 0.547 & 0.281 & 0.438 & **0.300** & 0.392 \\ - w/o Comparison & 0.554 & 0.347 & 0.448 & 0.293 & 0.411 \\ - w/o Ranking & 0.591 & 0.354 & 0.433 & 0.252 & 0.408 \\ - w/o QA & 0.579 & 0.357 & 0.418 & 0.284 & 0.410 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation study on stage one instruction tuning task type (Spearman correlation). ”w/o STT” denotes the model does not use Stage-Two Tuning. The best results are highlighted in **bold**.

 

Effect of the Scale of Language Model BackbonesWe adopt the same training and inference pipelines for the backbones with different scales to show the effect of the models' size and justify the use of Flan-T5-large. Specifically, we additionally experiment with Flan-T5-small (80M), Flan-T5-base (250M), and Flan-T5-xl (3B) as the backbone models, and term our X-Eval respectively. The results are shown in Figure 3. From Figure 3, the evaluators' performance consistently increases as the model size increases in general. However, when we upgrade the backbone model from Flan-T5-large to Flan-T5-xl, the performance improvement becomes less significant. Given the trade-off between efficiency and performance, we select Flan-T5-large as the default backbone model of X-Eval in our experiments. We include a more detailed performance analysis of the effect of language model backbones in Appendix C.

Error Propagation from Auxiliary Aspects during InferenceDuring inference, X-Eval may predict inaccurate evaluations for auxiliary aspects. To investigate their impact, we tailor several baselines: (1) directly applying the model after two-stage tuning to evaluate without auxiliary aspects; (2) using the ground truth ("GT") evaluation results instead of predicted results for auxiliary aspects (upperbound), and; (3) using random evaluation results for auxiliary aspects (lowerbound). From Table 6, removing auxiliary aspects makes the overall performance drop. The variant with GT results gains improvement in all aspects, which indicates the error in the evaluation of auxiliary aspects does impact the performance of target aspects, but not to a large degree. Using random results, on the other hand, deteriorates the performance significantly.

Effect of Hyperparameter \(k\)We examine the choice of \(k\) in selecting top-\(k\) auxiliary aspects during inference. Table 7 shows that inference with the top-\(1\) auxiliary aspect generally achieves better correlation. We speculate that this may stem from the error propagation during inference on auxiliary aspects, where using more auxiliary aspects potentially introduces more inaccuracies, offsetting their potential performance benefits.

Qualitative Correlation Analysis on Instruction TuningTo further investigate the effect of instruction tuning, in Figure 4, we visualize the correlation of our X-Eval and Flan-T5 (i.e., "X-Eval w/o Training") based on naturalness on Topical-Chat and consistency on SummEval. The red lines are linear regression fits to show how well the predicted scores correlate to human judgments linearly. Before instruction tuning, the predicted scores are more uniformly distributed regardless of ground truth scores, which results in poor correlation. On the contrary, our X-Eval can predict scores that not only achieve better correlation but also are more distinctive (either close to 1 or 0), showing the effectiveness of our instruction tuning.

Figure 3: Effect of the scale of language model backbones. For each meta-evaluation benchmark, we report the average Spearman correlation on all the aspects. X-Eval-large (780M) is the default backbone language model throughout all the experiments if there is no specification.

 

Visualization of Auxiliary Aspect SelectionIn Figure 5, we also report the cosine similarity between the sentence embeddings of the aspect definitions used in turn-level dialogue evaluation as the qualitative analysis of our aspect selection strategy. In general, our strategy can select semantically related aspects for target-aspect evaluation.

Analysis of Auxiliary Aspect Selection StrategyWe also experimented to compare the performance of selecting auxiliary aspects based on seen, unseen, or all aspects, as well as randomly selecting aspects regardless of the definitions. We set the number of auxiliary aspects to 1 in this experiment. From Table 8, selecting the auxiliary aspect based on all the aspects achieves the best overall performance. Also, we observe a substantial performance degradation when the auxiliary aspect is randomly selected, which shows the effectiveness of our aspect selection strategy.

## 8 Conclusion

In this work, we present X-Eval, a novel two-stage instruction-tuning framework for text evaluation across both seen and unseen aspects. To facilitate training, we collect AspectInstruct, the first instruction-tuning dataset for multi-aspect evaluation. Extensive experiments on meta-evaluation benchmarks demonstrate that with significantly fewer parameters, X-Eval achieves a comparable if not higher correlation with human judgments compared to the state-of-the-art NLG evaluators.

## 9 Limitations

Limitation of Data CollectionIn this work, we mainly target evaluation tasks in English. Future work can explore evaluation tasks in a more diverse language setting and augment our AspectInstruct dataset. In addition, our dataset focuses on a limited subset of NLG tasks including dialogue, summarization, and data2text. More NLG tasks

\begin{table}
\begin{tabular}{l|c|c|c|c|c} \hline \hline
**Selection** & **Topic.** & **FED** & **Summ.** & **D2T** & **AVG** \\ \hline Top-1 & **0.605** & 0.434 & **0.480** & **0.303** & **0.456** \\ Top-3 & 0.602 & 0.414 & 0.466 & 0.278 & 0.440 \\ Top-5 & 0.598 & **0.435** & 0.463 & 0.275 & 0.443 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Effect of different \(k\) in selecting auxiliary aspect in terms of averaged Spearman correlation. The best results are highlighted in **bold**.

Figure 4: The scatter plots of correlation between human scores and predicted scores of X-Eval and Flann T5, respectively.

Figure 5: Cosine similarity scores of the sentence embeddings of aspect definition in turn-level dialogue evaluation. Naturalness (NAT), coherence (COH), engagingness (ENG), and groundedness (GRO) are seen aspects, while the rest are unseen aspects.

 

can be considered in the future.

Inference EfficiencyOur algorithm may require multiple rounds of predictions to generate evaluation results from auxiliary aspects in the inference time. While this process imposes additional computational costs, given that the backbone we used is lightweight (with 780M parameters) and efficient, our approach is still significantly more efficient than the evaluators that are much larger, e.g., GPT-4. We leave exploring more efficient inference strategies for future work.

Error PropagationDuring inference, the evaluation results of auxiliary aspects may contain some errors. The errors may affect the final evaluation of the target aspect. We leave developing more robust inference algorithms to address the error propagation problem for future works.

## Acknowledgments

This research is partially supported by a research award from the Amazon-Virginia Tech Initiative and award No. 2330940 from the Secure and Trustworthy Cyberspace program of the National Science Foundation (NSF). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* S. Banerjee and A. Lavie (2005)Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72. Cited by: SS1.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, Arvind Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. CoRRabs/2005.14165. External Links: Link, 2005.14165 Cited by: SS1.
* H. Won Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. Shaine Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Y. Zhao, Y. Huang, A. M. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei (2022)Scaling instruction-finetuned language models. CoRRabs/2210.11416. External Links: Link, 2005.14165 Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Vol. , pp. 4171-4186. External Links: Document, ISSN 1574-147X Cited by: SS1.
* A. R Fabbri, W. Kryscinski, B. McCann, C. Xiong, R. Socher, and D. Radev (2021)Summeval: re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics9, pp. 391-409. External Links: Link, 2005.14166 Cited by: SS1.
* J. Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu (2023)Gptscore: evaluate as you desire. CoRRabs/2302.04166. External Links: Link, 2302.04166 Cited by: SS1.
* S. Gehrmann, E. Clark, and T. Sellam (2023)Repairing the cracked foundation: a survey of obstacles in evaluation practices for generated text. J. Artif. Intell. Res.77, pp. 103-166. External Links: Link, 2005.14165 Cited by: SS1.
* K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra, A. Venkatesh, R. Gabriel, and D. Hakkani-Tur (2019)Topical-chat: towards knowledge-grounded open-domain conversations. External Links: 1905.02771 Cited by: SS1.
* R. Chulaka Gunasekara, S. Kim, L. Fernando D'Haro, A. Rastogi, Y. Chen, M. Eric, B. Hedayatnia, K. Gopalakrishnan, Y. Liu, C. Huang, D. Hakkani-Tur, J. Li, Q. Zhu, L. Luo, L. Linden, K. Huang, S. Shayandeh, R. Liang, B. Peng, Z. Zhang, S. Shukla, M. Huang, J. Gao, S. Mehri, Y. Feng, C. Gordon, S. Hossein Alavi, D. R. Traum, M. Eskenzi, A. Beirami, E. Cho, P. A. Crook, A. De, A. Geramifard, S. Kottur, S. Moon, S. Poddar, and R. Subba (2020)Overview of the ninth dialog system technology challenge: DSTC9. CoRRabs/2011.06486. External Links: Link, 2011.06486 Cited by: SS1.
* E. J. Hu, y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022)LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, 

Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. GRADE: automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 9230-9240. Association for Computational Linguistics.
* Jiang et al. (2023) Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2023. Tigerscore: Towards building explainable metric for all text generation tasks. _CoRR_, abs/2310.00752.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. _arXiv preprint arXiv:1910.13461_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Liu et al. (2022) Minqian Liu, Shiyu Chang, and Liftu Huang. 2022. Incremental prompting: Episodic memory prompt for lifelong event detection. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 2157-2165, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Liu and Huang (2023) Minqian Liu and Liftu Huang. 2023. Teamwork is not always good: An empirical study of classifier drift in class-incremental information extraction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2241-2257, Toronto, Canada. Association for Computational Linguistics.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chengquang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment. _CoRR_, abs/2303.16634.
* Madotto et al. (2021) Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwha Moon, Paul Crook, Bing Liu, Zhou Yu, Eujoon Cho, Pascale Fung, and Zhiguang Wang. 2021. Continual learning in task-oriented dialogue systems. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7452-7467, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Mehri and Eskenazi (2020) Shikib Mehri and Maxine Eskenazi. 2020a. Unsupervised evaluation of interactive dialog with dialogpt. _arXiv preprint arXiv:2006.12719_.
* Mehri and Eskenazi (2020) Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. pages 681-707.
* Mehri and Shwartz (2023)Shuhaiy Mehri and Vered Shwartz. 2023. Automatic evaluation of generative models with instruction tuning. _CoRR_, abs/2310.20072.
* Ni et al. (2022) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pretrained text-to-text models. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 1864-1874.
* OpenAI (2023) OpenAI. 2023. Qpt-4 technical report. _ArXiv_, abs/2303.08774.
* Pang et al. (2020) Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3619-3629, Online. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Qi et al. (2023) Jingu Yuan Qi, Zhiyang Xu, Ying Shen, Mingqian Liu, Di Jin, Qifan Wang, and Liftu Huang. 2023. The art of SOCRATIC QUESTIONING: zero-shot multimodal reasoning with recursive thinking and self-questioning. _CoRR_, abs/2305.14999.
* Sai et al. (2020)Ananya B Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. _Transactions of the Association for Computational Linguistics_, 8:810-827.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gaudier Izacard, Xavier Martinet, Marie-Anne Lachatav, Timothee Lacroix, Bapistte Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LDAA: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Touvron et al. (2021) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlyov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyiu Fu, Brian Fuller, Cynthia Gao, Vedanuj Gosowami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Innan, Marcia Kardas, Viktor Kerkez, Madain Khabsa, Isabel Koumann, Artem Koreney, Punit Singh Kouura, Marie-Anne Lachatav, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molbybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runeta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Rambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas 

Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. CoRRabs/2307.09288.
* Volske et al. (2017) Michael Volske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, Copenhagen, Denmark. Association for Computational Linguistics.
* Wang et al. (2020a) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5008-5020. Association for Computational Linguistics.
* Wang et al. (2020b) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020b. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228.
* Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
* Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
* Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS.
* Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745.
* Xu et al. (2023a) Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023a. INSTRUCTSCORE: towards explainable text generation evaluation with automatic feedback. CoRRabs/2305.14282.
* Xu et al. (2023b) Zhiyang Xu, Ying Shen, and Lifu Huang. 2023b. Multi-Instruct: Improving multi-modal zero-shot learning via instruction tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence 11445-11465, Toronto, Canada. Association for Computational Linguistics.
* Yao et al. (2023) Shunya Yao, Qian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. CoRRabs/2305.10601.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems34:27263-27277.
* Zhang et al. (2021) Chen Zhang, Yiming Chen, Luis Fernando D'Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021. DynaCEval: Unifying turn and dialogue level evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics.
* Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Morverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 563-578.
* Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chengquang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023-2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

## Appendix A More Details on AspectConstruct

### Annotation Protocol of Instructions

We depict the annotation process for the instructions in AspectConstruct as follows. To curate the definition for each aspect, we first refer to the definition of the aspect in the original annotation guideline. When a definition is absent from the guideline, three human annotators (graduate students studying in computational linguistics or natural language processing areas) construct and revise the definition until they reach an agreement. The task descriptions and evaluation protocols are also written by three human annotators in similar annotation protocols.

### Augmenting Instruction-tuning Tasks

We show the seen aspects, their corresponding source datasets where we collect the training data, 

constructed tasks, and the number of training instances for each task in Table 11 and Table 12. For the way we count the number of aspects, we treat the aspects with the same name but in different NLG tasks as different aspects. For example, the _naturalness_ aspect in dialogue evaluation and data2text evaluation are considered different under these two settings, although they have the same aspect name. More specifically, in our AspectInstruct dataset, _understandability_ is counted twice for dialogue-level and turn-level dialogue evaluation; _naturalness_ is counted twice for turn-level dialogue evaluation and data-to-text evaluation; _informativeness_ is counted twice for dialogue-level dialogue evaluation and data-to-text evaluation. We also include an example of how we augment instruction-tuning tasks from the original annotation in Figure 6.

### Aspect Definition

We present the annotated definitions in AspectInstruct in the following. We show the definitions of seen aspects on dialogue evaluation on Table 13, unseen aspects on dialogue evaluation on Table 14, and the aspects on summarization on Table 15.

Source Datasets for Meta Evaluation SummEval Fabbri et al. (2021)is an evaluation benchmark for summarization which contains human ratings of 100 summaries along four evaluation dimensions: fluency, coherence, consistency, and relevance.

QAGS Wang et al. (2020)is a benchmark for identifying and evaluating hallucinations in the summarization task. It aims to measure the factual inconsistencies of generated summaries.

Figure 6: An illustrative example of augmented instruction-tuning tasks from the original annotation. The definition of the aspect is highlighted in purple. The annotated task instructions and the constructed output labels are highlighted in the corresponding colors for each task.

 

Topical-Chat (Gopalakrishnan et al., 2019)is a knowledge-grounded human-human conversation dataset. Following Zhong et al. (2022), we utilize human ratings collected by Mehri and Eskenazi (2020b) for Topical-Chat as the benchmark for evaluating dialog response generation. The assessment consider five aspects: naturalness, coherence, engagingness, groundedness, and understandability.

FED (Mehri and Eskenazi, 2020a)is an evaluation benchmark for fine-grained dialog evaluation. It comprises human annotations evaluated across eighteen dialog aspects at both the turn-level and the dialog-level.

SFHOT & SFRES (Wen et al., 2015)are evaluation benchmarks for data-to-text task. They provide information about restaurants and hotels in San Francisco. The generated text is evaluated based on two aspects: informativeness and naturalness.

## Appendix B More Details on X-Eval

Pseudo-code of Inference PipelineWe provide the pseudo-code of our proposed inference pipeline for X-Eval in Algorithm 1.

More Details on Verbalizer \(v\) and its Templates We design a template-based verbalizer to convert the evaluation results of auxiliary aspects into natural language evaluation that can be integrated into the instructions. More formally, the inputs of the verbalizer \(v\) contain aspect \(a\) and evaluation score \(s\) (in the range of 0-1). We first adopt a threshold \(\delta\) (we set \(\delta=0.5\) throughout all experiments) to get a _binary_ label that indicates the quality is _"positive"_ (if \(s>\delta\)) or _"negative"_ (if \(s\leq\delta\)). Given this label and the aspect \(a\), we map the results into a template in natural language accordingly. The verbalized results will then be integrated into the instructions. We construct the templates for each aspect by deriving from aspect definition. We apply the annotation protocol that three human annotators revise the templates together until they reach a consensus. We show the verbalized templates in Table 16 for dialogue evaluation and Table 17 for summarization evaluation.

## Appendix C More Details on Experiments

More Implementation DetailsWe use the checkpoint released on HuggingFace for Flan-T5-large2. In the first training stage, we set the number of epochs to 2, the learning rate to 5e-05, and the maximum source length to 1024. The second training stage shares the same setup except the number of epochs set to 1. We set the maximum source length during inference to 2048 and pick the top-1 aspect during inference, i.e., \(k=1\). We use sentence-T5-large3 to compute the embeddings for aspect definition for auxiliary aspect selection. All the experiments are conducted on NVIDIA A40 GPUs including both training and inference.

Footnote 2: [https://huggingface.co/google/flan-t5-large](https://huggingface.co/google/flan-t5-large)

Footnote 3: [https://huggingface.co/sentence-transformers/sentence-t5-large](https://huggingface.co/sentence-transformers/sentence-t5-large)

More Details on BaselinesWe include more details for the following baselines that are omitted in the main paper due to page limit: **(4) ROUGE-L**Lin (2004) counts the overlap (i.e., longest common subsequence) between the text to be evaluated and reference to indicate text quality; **(5) DynaCwal**Zhang et al. (2021) adopts a graph convolutional network to model dialogue's structure to facilitate evaluation; **(6) BERTScore**Zhang* et al. (2020) is a similarity-based evaluator. It uses the contextualized representation from BERT Devlin et al. (2019) to compute the similarity between the generated text and reference; **(7) MoverScore**Zhao et al. (2019) goes beyond BERTScore by utilizing soft alignments and new aggregation methods on the layer-wise information; **(8) USR**Mehri and Eskenazi (2020b) is an unsupervised and reference-free evaluation metric to measure multiple desirable qualities of dialog; **(9) BARTScore**Yuan et al. (2021) is a unified evaluator based on BART Lewis et al. (2019), which uses the average likelihood of the model output as the metric. Note that for all single-aspect metrics, we compute the correlation between the single predicted evaluation and the human rating of each fine-grained aspect, respectively.

More Results on the Effect of the Scale of Language Model BackbonesWe further conducted an experiment on using another language model backbone. Specifically, we adopt LLaMA-7Bchat Touvron et al. (2023a) as the backbone model and adopt LoRA parameter-efficient tuning Hu et al. (2022) during the two-stage instruction tuning. We report the performance in Table 10.

 



\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c} \hline
**Model** & **\# Parameters** & **TopicalChat** & **SummEval** & **FED-Dialog** & **FID-Turn** & **Data2Text** & **AVG** \\ \hline X-Eval-large (Default Ver.) & 780M & 0.605 & 0.480 & 0.485 & 0.398 & 0.303 & 0.454 \\ X-Eval-LLaMA-LoRA & 7B & 0.519 & 0.448 & 0.427 & 0.351 & 0.337 & 0.416 \\ \hline \end{tabular}
\end{table}
Table 10: Effect of the scale of language model backbones. For each meta-evaluation benchmark, we report the average Spearman correlation on all the aspects.

\begin{table}
\begin{tabular}{p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}} \hline
**Aspect** & **Datasets** & **Task** & **\# Instances** \\ \hline Accuracy & TL;DR (Völske et al., 2017) & \begin{tabular}{c} Scoring \\ Boolean QA \\ Comparison \\ Ranking \\ \end{tabular} & \begin{tabular}{c} 5,000 \\ 5,000 \\ 599 \\ \end{tabular} \\ \hline Coherence & TL;DR (Völske et al., 2017), UniEval (Zhong et al., 2022) & \begin{tabular}{c} Scoring \\ Boolean QA \\ Comparison \\ Ranking \\ \end{tabular} & \begin{tabular}{c} 5,000 \\ 5,000 \\ 5,000 \\ 734 \\ 425 \\ \end{tabular} \\ \hline Coverage & TL;DR (Völske et al., 2017) & \begin{tabular}{c} Scoring \\ Boolean QA \\ Comparison \\ Ranking \\ \end{tabular} & \begin{tabular}{c} 5,000 \\ 4,354 \\ 1,028 \\ 964 \\ \end{tabular} \\ \hline Consistency & UniEval (Zhong et al., 2022) & \begin{tabular}{c} Boolean QA \\ 15,000 \\ 15,000 \\ \end{tabular} \\ \hline Fluency & UniEval (Zhong et al., 2022) & \begin{tabular}{c} Boolean QA \\ 15,000 \\ \end{tabular} \\ \hline Relevance & UniEval (Zhong et al., 2022) & \begin{tabular}{c} Boolean QA \\ \end{tabular} & 
\begin{tabular}{c} 15,000 \\ 15,000 \\ \end{tabular} \\ \hline \end{tabular}
\end{table}
Table 11: The full list of apects, the corresponding datasets and tasks on summarization evaluation collected in the training split of AspectInstruct.

\begin{table}
\begin{tabular}{p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}} \hline
**Aspect** & **Datasets** & **Task** & **\# Instances** \\ \hline Relevance & DailyDialog++ (Sai et al., 2020) & \begin{tabular}{c} Scoring \\ Boolean QA \\ Comparison \\ \end{tabular} & \begin{tabular}{c} 2,000 \\ 2,000 \\ 2,000 \\ 2,000 \\ 2,000 \\ 2,000 \\ 2000 \\ \end{tabular} \\ \hline Coherence & HolisticDial (Pang et al., 2020); DSTCP9 (Sunasekara et al., 2020); UniEval (Zhang et al., 2022) & \begin{tabular}{c} Scoring \\ Boolean QA \\ \end{tabular} & \begin{tabular}{c} 2,400 \\ 17,200 \\ 2,200 \\ \end{tabular} \\ \hline Consistency & DSTCP9 (Sunasekara et al., 2020) & \begin{tabular}{c} Scoring \\ Boolean QA \\ \end{tabular} & \begin{tabular}{c} 2,200 \\ 2,200 \\ 2,200 \\ \end{tabular} \\ \hline Diversity & DSTCP9 (Sunasekara et al., 2020) & \begin{tabular}{c} Scoring \\ Boolean QA \\ \end{tabular} & \begin{tabular}{c} 2,200 \\ 2,200 \\ 15,000 \\ \end{tabular} \\ \hline Groundedness & UniEval (Zhong et al., 2022) & \begin{tabular}{c} Boolean QA \\ \end{tabular} & \begin{tabular}{c} 15,000 \\ 15,000 \\ \end{tabular} \\ \hline Naturalness & UniEval (Zhong et al., 2022) & \begin{tabular}{c} Boolean QA \\ \end{tabular} & \begin{tabular}{c} 15,000 \\ 15,000 \\ \end{tabular} \\ \hline Fluency & HolisticDial (Pang et al., 2020) & Scoring & 
\begin{tabular}{c} 200 \\ 200 \\ \end{tabular} \\ \hline \end{tabular}
\end{table}
Table 12: The full list of apects, the corresponding datasets and tasks on dialogue evaluation collected in the training split of AspectInstruct. “NOTA” indicates the comparison task consists of the case of “None Of The Above”, where the quality of two candidates is tied.





\begin{table}
\begin{tabular}{l|p{284.5pt}}
**Aspect** & **Definition** \\ Naturalness & Naturalness in dialogue evaluation refers to the degree to which a response in a conversational context mirrors the characteristics, language use, and structure typical of a human conversational partner. \\ Coherence & Coherence refers to the logical and consistent interconnection of utterances and exchanges throughout a conversation. It represents the extent to which a dialogue system maintains relevance, consistency, and meaningful progression within the discourse, ensuring that the flow and structure of the conversation align with expected conversational norms and the ongoing context. \\ Engagingness & Engagingness in the context of dialogue evaluation refers to the degree to which a response fosters continued interaction, maintains or elevates interest, and stimulates a compelling exchange of ideas, emotions, or information between participants. \\ Groundedness & Dialogue groundedness measures how well does the response use the given fact. A response with weak groundedness means the response does not mention or refer to the fact at all. A response with good groundedness means the response uses the fact well. \\ Relevance & Relevance in dialogue evaluation refers to the measure of applicability, pertinence, or connection of a given response to the preceding conversational context and/or the explicitly posed question or statement. \\ Fluency & Fluency in dialogue evaluation refers to the degree of fluidity, coherence, and linguistic correctness in a generated response. It encompasses not only the grammatical and syntactic accuracy but also the seamless flow of ideas, the smooth transition between topics, and the naturalness of the language used, echoing human-like conversation patterns. \\  & \\ \end{tabular}
\end{table}
Table 13: The full list and definitions of _seen_ aspects on dialogue evaluation collected in AspectInstruct.





\begin{table}
\begin{tabular}{p{108.4pt}|p{256.9pt}}
**Aspect** & **Definition** \\ Topic Depth & Topic depth refers to the ability of a dialogue system to engage in extensive, detailed, and multi-turn discussions on a particular subject. \\ Likeability & Likeability refers to the degree to which an interactive system presents a pleasant, engaging, and affable conversational style that resonates positively with the user. \\ Understandability & Understandability reflects the ability of a conversational system to correctly parse and interpret user inputs, reflect an appropriate comprehension of the context, and generate contextually relevant responses. \\ Flexibility & Flexibility measures the system’s capacity to understand and react appropriately to a wide range of conversational scenarios, and not merely those for which it was explicitly programmed or trained. It implies the capacity to engage in a diverse array of topics, offer meaningful responses in unexpected situations, and adjust conversational strategies based on the evolving context or user input. \\ Informativeness & Informativeness refers to the quality and relevance of the information that a dialogue system provides in response to user inputs. It captures the system’s ability to offer novel, detailed, accurate, and appropriate information that aligns with the user’s requests or needs. \\ Inquisitiveness & Inquisitiveness pertains to the consistent exhibition of the capacity to ask meaningful, contextually appropriate, and well-timed questions within a conversation by a dialogue system. This behavior is exhibited in the pursuit of greater comprehension, clarifying ambiguities, furthering the dialogue, or driving deeper engagement with the conversation partner. \\ Interestingness & Interestingness refers to the degree to which a response stimulates engagement, thought, or emotional reaction in the average user, fostering a desire to continue the conversation or explore the topic further. It is a measure of the response’s capacity to capture the user’s attention and maintain their engagement over time. \\ Specificity & Specificity measures to what degree the response is unique, personalized, or pertinent to the specific details of the preceding user inputs or dialogue context, as opposed to being generic, universally applicable, or independent of the conversational specifics. \\ Correctness & Correctness in dialogue evaluation measures to the extent to which a generated response correctly reflects, comprehends, and addresses the salient elements, inferences, and implications in the preceding conversation context. \\ Semantic Appropriateness & Semantic appropriateness is the measure of the extent to which a response in a dialogue maintains logical, meaningful, and contextually fitting alignment with the preceding discourse elements, while adhering to the rules and principles of the language used in the conversation. \\  & \\ \end{tabular}
\end{table}
Table 14: The full list and definitions of _unseen_ aspects on dialogue evaluation collected in AspectInstruct.





\begin{table}
\begin{tabular}{p{56.9pt}|p{341.2pt}} \hline
**Aspect** & **Definition** \\ \hline Accuracy & The accuracy aspect measures how the factual information in the summary accurately matches the post. A summary is accurate if it doesn’t say things that aren’t in the article, it doesn’t mix up people, and generally is not misleading. If the summary says anything at all that is not mentioned in the post or contradicts something in the post, it should be considered as an inaccurate summary. \\ \hline Coherence & The coherence aspect measures how coherent is the summary on its own. A summary is coherent if, when read by itself, it’s easy to understand and free of English errors. A summary is not coherent if it’s difficult to understand what the summary is trying to say. Generally, it’s more important that the summary is understandable than it being free of grammar errors. \\ \hline Coverage & The coverage aspect measures how well does the summary cover the important information in the post?” A summary has good coverage if it mentions the main information from the post that’s important to understand the situation described in the post. A summary has poor coverage if someone reading only the summary would be missing several important pieces of information about the situation in the post. A summary with good coverage should also match the purpose of the original post (e.g. to ask for advice). \\ \hline Consistency & The consistency aspect measures the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. You also need to penalize summaries that contained hallucinated facts. \\ \hline Fluency & Fluency measures the quality of individual sentences. A fluent summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. \\ \hline Relevance & Relevance measures the selection of important content from the source. The summary should include only important information from the source document. You should penalize summaries which contain redundancies and excess information. \\ \hline \end{tabular}
\end{table}
Table 15: The full list and definitions of aspects of summarization evaluation collected in AspectInstruct.





\begin{table}
\begin{tabular}{l|p{213.8pt}} \hline
**Aspect** & **Verbalizer Template** \\ \hline Naturalness & NEG: The response is unnatural. POS: The response is natural. \\ \hline Coherence & NEG: The response drastically changes topic or ignores the conversation history. POS: The response is on topic and strongly acknowledges the conversation history. \\ \hline Engagingnes & NEG: The response is generic and dull. POS: The response is interesting or presents an interesting fact. \\ \hline Groundedness & NEG: Given the interesting fact that the response is conditioned on, the response does not mention or refer to the fact at all. POS: Given the interesting fact that the response is conditioned on, the response uses the fact well. \\ \hline Relevance & NEG: The response is not relevant to the conversation. POS: The response is relevant to the conversation. \\ \hline Fluency & NEG: The response is not fluently written. POS: The response is fluently written. \\ \hline Topic Depth & NEG: The system cannot discuss topics in depth. POS: The system is able to discuss topics in depth. \\ \hline Likeability & NEG: The system cannot display a likeable personality. POS: The system is able to display a likeable personality. \\ \hline Understandability & NEG: The response is difficult to understand. You do not know what the person is trying to say. POS: The response is understandable. You know what the person is trying to say. \\ \hline Flexibility & NEG: The system is not flexible and adaptable to the user and their interests. POS: The system is flexible and adaptable to the user and their interests. \\ \hline Informativeness & NEG: The system is not informative throughout the conversation. POS: The system is informative throughout the conversation. \\ \hline Inquisitiveness & NEG: The system is not inquisitive throughout the conversation. POS: The system is inquisitive throughout the conversation. \\ \hline Interestingness & NEG: To the average person, the response is not interesting. POS: To the average person, the response is interesting. \\ \hline Specificity & NEG: The response is too generic and not specific to the conversation. POS: The response is specific to the conversation. \\ \hline Correctness & NEG: There was a misunderstanding of the conversation. POS: The response is correct in the context of the conversation. \\ \hline Semantic Appropriateness & NEG: The response is not semantically appropriate. POS: The response is semantically appropriate. \\ \hline \end{tabular}
\end{table}
Table 16: The full list of verbalizer templates that are used to convert the evaluation results of auxiliary aspects for dialogue evaluation collected in AspectInstruct. “POS” and ”NEG” indicate _“positive”_ and _"negative”_, respectively.





\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline \hline
**Aspect** & **Verbalizer Template** \\ \hline Accuracy & NEG: The factual information in the summary cannot accurately match the post. It says things that aren’t in the article, it mixes up people, or generally is misleading. \\  & POS: The factual information in the summary accurately match the post. It doesn’t say things that aren’t in the article, it doesn’t mix up people, and generally is not misleading. \\ \hline Coherence & NEG: The summary is not coherent as it lacks a logical flow and has disjointed information, making it difficult to understand the main topic or argument. \\  & POS: The summary is well-structured and well-organized and it is built from sentence to sentence to a coherent body of information about a topic. \\ \hline Coverage & NEG: The summary has poor coverage on the important information in the post, e.g., someone reading only the summary would be missing several important pieces of information about the situation in the post. \\  & POS: The summary has good coverage since it mentions the main information from the post that’s important to understand the situation described in the post and also match the purpose of the original post. \\ \hline Consistency & NEG: The summary is not factually consistent with the original post as it introduces factual inaccuracies or hallucinated facts that are not present in or supported by the original source document. \\  & POS: The summary has good factual alignment between the summary and the summarized source. It contains only statements that are entailed by the source document. \\ \hline Fluency & NEG: The summary is not fluent as it contains formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. \\  & POS: This is a fluent summary as it generally does not have formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. \\ \hline Relevance & NEG: This summary is not relevant to the source document as it contains redundancies or excess information. \\  & POS: The summary generally includes relevant content, capturing some key points from the source. \\ \hline \hline \end{tabular}
\end{table}
Table 17: The full list of verbalizer templates that are used to convert the evaluation results of auxiliary aspects for summarization evaluation collected in AspectInstruct. ”POS” and ”NEG” indicate _“positive”_ and _"negative”_, respectively.