DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
Year: 2023

Abstract:
Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. Specifically, most of the wellperformed metrics are required to train on evaluation datasets of specific NLG tasks and evaluation dimensions, which may cause over-fitting to task-specific datasets. Furthermore, existing metrics only provide an evaluation score for each dimension without revealing the evidence to interpret how this score is obtained. To deal with these challenges, we propose a simple yet effective metric called DecompEval. This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models (PLMs) without training on evaluation datasets, aiming to enhance the generalization ability. To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result. Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability1.

heading: 1 Introduction
text: Recently, pre-trained language models (PLMs) such as GPT (Brown et al., 2020), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020) have achieved promising performance in natural language generation (NLG) tasks, such as text summarization (Zhang et al., 2020a) and dialogue generation (Zhang et al., 2020c). As the quality of gen-
∗ Corresponding author 1The codes are available at https://github.com/
kepei1106/DecompEval
erated texts gradually approaches that of humanwritten texts, there is an increasing demand for automatic evaluation metrics of generated texts.
However, existing evaluation metrics are still struggling to measure the quality of generated texts accurately. Traditional metrics such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) rely on n-gram overlap between generated texts and reference texts, which fail to detect the issues in the content of generated texts (Gehrmann et al., 2022). Recent works resort to model-based evaluation metrics to compute the similarity between generated texts and reference texts based on contextual representations from pre-trained models (Zhao et al., 2019; Zhang et al., 2020b) or adopt the score of language modeling (Yuan et al., 2021) / masked language modeling (Ke et al., 2022; Colombo et al., 2022) for evaluation. Other works choose to train evaluation models on the evaluation datasets to fit human scores (Shen et al., 2017; Sellam et al., 2020) or distinguish human-written texts from negative samples (Guan and Huang, 2020; Zhong et al., 2022), aiming to obtain higher correlations with human judgments in various evaluation dimensions (such as coherence and consistency) of specific datasets.
We argue that there are two main challenges in building an evaluation metric for text generation: 1) Generalization Ability: Most of the existing metrics that have high correlations with human judgments on evaluation datasets are directly trained on the corresponding datasets (Sellam et al., 2020; Guan and Huang, 2020; Zhong et al., 2022). This may result in over-fitting to task-specific data and harm their generalization ability to other NLG tasks and dimensions (Ke et al., 2022). 2) Interpretability: Although recently proposed evaluation metrics can measure the quality of generated texts from multiple dimensions, they only provide an evaluation score for each dimension without giving evidence to interpret how they predict this score
9676
(Ke et al., 2022; Zhong et al., 2022). To deal with these challenges, we propose a simple yet effective evaluation metric called DecompEval. Firstly, to improve the generalization ability, we formulate NLG evaluation as an instruction-style question answering (QA) task, and utilize instruction-tuned pre-trained language models (Chung et al., 2022) to solve this task without training on task-specific data. The instruction-style question consists of an instruction, the input of NLG evaluation, and a yes/no question, e.g., "Answer the following yes/no question ... Is this a coherent response given the dialogue history?" for the evaluation of coherence in dialogue generation, where the specific evaluation input is omitted. Secondly, we propose a question decomposition strategy to make the evaluation process more interpretable, instead of directly making instructiontuned PLMs answer the original question. This strategy decomposes the question into the subquestions which sequentially evaluate the corresponding dimension of each sentence in the generated texts. Then, we recompose these subquestions with their answers generated by the PLM as evidence to make the PLM answer the original question, which is used to compute the final evaluation result. The evidence can promote the understanding of the evaluation process by indicating the potential problematic sentences that affect the evaluation score.
Our main contributions are as follows:
• We propose an evaluation metric called DecompEval, which formulates NLG evaluation as an instruction-style QA task, and solves it with instruction-tuned PLMs via question decomposition.
• We conduct experiments on the benchmark datasets for evaluating text summarization and dialogue generation. Experimental results show that DecompEval can achieve state-ofthe-art performance in untrained metrics.
• We empirically show that DecompEval can generalize to other evaluation dimensions and tasks (such as data-to-text generation) better than all the baselines, while improving the interpretability via decomposed subquestions with their answers.

heading: 2 Related Work

heading: 2.1 Evaluation for Language Generation
text: Evaluation is a long-standing task in the field of NLG (Celikyilmaz et al., 2020), which becomes more critical with the rapid development of PLMs. There are two main categories of automatic evaluation metrics, i.e., untrained and trained metrics (Sai et al., 2020). Untrained metrics without training on specific datasets of evaluation tasks or related tasks aim to measure the relationship among source texts, generated texts, and reference texts via n-gram overlap (Papineni et al., 2002; Banerjee and Lavie, 2005; Lin, 2004), semantic similarity (Zhao et al., 2019; Zhang et al., 2020b), or language modeling / masked language modeling scores (Yuan et al., 2021; Ke et al., 2022; Colombo et al., 2022). In comparison, trained metrics are commonly trained on the evaluation datasets to fit human scores (Shen et al., 2017; Sellam et al., 2020) or distinguish human-written texts from negative samples (Guan and Huang, 2020; Zhong et al., 2022), aiming to achieve higher correlations with human judgments on specific datasets. Among these metrics, there are some similar works which re-frame NLG evaluation as QA tasks and adopt the generated answers or generation probabilities as evaluation results (Deutsch et al., 2021; Zhong et al., 2022).
The most similar work to our method is UniEval (Zhong et al., 2022). UniEval re-frames NLG evaluation as a Boolean QA task and trains the evaluation model on the pseudo data constructed from the evaluation dataset and other related datasets in a unified Boolean QA format. Compared with UniEval, our method is untrained since we transform NLG evaluation to an instruction-style QA task that can be solved by instruction-tuned PLMs without further training. Also, our method can provide some evidence (i.e., the answers to decomposed subquestions) to interpret how the model reaches the evaluation result, instead of only providing a final evaluation score.

heading: 2.2 Instruction-Tuned Pre-Trained Models
text: Instruction learning (Weller et al., 2020) which trains PLMs to follow human instructions has attracted much attention recently since it shows the strong zero-shot cross-task generalization ability. To improve instruction understanding, existing works adopt instruction tuning (Wei et al., 2022) which trains PLMs on massive tasks described
via instructions with multi-task learning, such as FLAN (Wei et al., 2022; Chung et al., 2022), T0 (Sanh et al., 2022), and InstructGPT (Ouyang et al., 2022). Other works systematically study instruction tuning in specific areas such as dialogue systems (Gupta et al., 2022) and multi-modal learning (Xu et al., 2022).
In comparison, our work is the first to explore the potential of instruction-tuned PLMs in the evaluation of NLG without further training. We show that equipped with well-designed input prompts and suitable question decomposition, instructiontuned PLMs can sequentially measure the quality of each sentence and finally recompose all the subquestions with their answers to obtain surprisingly great evaluation results in an unsupervised fashion.

heading: 3 Method

heading: 3.1 Task Definition and Model Overview
text: Given the context c, the model-generated text x, and the reference text r, our goal is to acquire the evaluation results from different individual dimensions, respectively. The context contains different contents in various NLG tasks. Also, the context and the reference may be omitted, which depend on the evaluation task and dimension. We assume that the generated text consists of n sentences, i.e., x = (x1, x2, · · · , xn).
As shown in Figure 1, our main idea is to formulate NLG evaluation as an instruction-style QA task and solve this task with instruction-tuned PLMs
via question decomposition. Our proposed method consists of three steps. First of all, we transform the input of NLG evaluation into an instruction-style question which contains an instruction s, the input of evaluation tasks (c, x, r), and a yes/no question q for each dimension (§3.2). Then, we decompose this question into the subquestions {sqt}nt=1, which evaluate each sentence xt(1 ≤ t ≤ n) in the generated text x respectively and acquire the answers {at}nt=1 to these subquestions via the instructiontuned PLM Pθ (§3.3). The answer to each subquestion is appended to the input prompt of the PLM, which may help to solve subsequent subquestions as in-context examples. Finally, we recompose all the subquestions with their answers as evidence and make the instruction-tuned PLM answer the original question, which can be used to compute the evaluation result (§3.4).

heading: 3.2 Instruction-Style QA Task Formulation
text: To improve the generalization ability of evaluation metrics, we formulate NLG evaluation as an instruction-style QA task that can be solved by instruction-tuned PLMs in an unsupervised fashion. As shown in Figure 1, the instruction-style question contains three parts:
• Instruction: The design of instructions depends on the data format of instruction-tuned PLMs. In this paper, we adopt yes/no questions (Zhong et al., 2022) to measure the quality of generated texts. Thus, we follow
Chung et al. (2022) to devise the instruction as s ="Answer the following yes/no question.".
• Evaluation Input: The original input (c, x, r) for NLG evaluation mentioned in §3.1 are incorporated with task-specific descriptive texts. For example, we add the text "dialogue history:", "response:", and "reference:" before c, x, and r respectively for evaluating dialogue generation.
• Yes/No Question: We finally devise a yes/no question to assess the specific dimension of generated texts. For example, the yes/no question assessing the coherence of generated texts in dialogue generation is q ="Is this a coherent response given the dialogue history?".

heading: 3.3 Question Decomposition and Subquestion Answering
text: To interpret how the model predicts the evaluation score, we devise a question decomposition strategy inspired by the existing works in the QA community (Min et al., 2019; Perez et al., 2020; Zhou et al., 2023), rather than force the instruction-tuned PLM to answer the original question directly. This strategy splits the generated text based on sentences and sequentially queries the quality of each sentence via subquestions. The subquestions with their answers generated by the PLM are expected to act as evidence to illustrate how the PLM arrives at the final evaluation score. We simply select sentences as the decomposition criterion instead of using external off-the-shelf models (Perez et al., 2020; Deutsch et al., 2021) because sentences are shown to be important basic units for deriving the evaluation result of the whole generated text (Amplayo et al., 2023).
Specifically, to answer the subquestion sqt(1 ≤ t ≤ n) for measuring the quality of the t-th sentence xt, we combine the instruction s, the evaluation input (c, x, r), the previous subquestions with their answers {(sqj , aj)}t−1j=1, and the current subquestion sqt as the input prompt It =( s, c, x, r, {(sqj , aj)}t−1j=1, sqt ) . Then, we compare the generation probability of "yes" / "no" from the instruction-tuned PLM to determine the answer:
at = { yes, Pθ(yes|It) > Pθ(no|It) no, Pθ(yes|It) ≤ Pθ(no|It) (1)
t = 1, 2, · · · , n
The answer at is appended to the current input prompt It, which becomes the in-context examples of It+1 helping to solve the next subquestion sqt+1. All these subquestions with their answers can serve as evidence to improve the interpretability by indicating potential low-quality sentences in the generated text that affect the evaluation score.

heading: 3.4 Question Recomposition for Evaluation
text: To recompose all the subquestions with their answers to acquire the final evaluation result, we append the original yes/no question mentioned in §3.2 to the end of the last subquestion and its answer. The instruction-tuned PLM is expected to leverage all these information as evidence to answer the original question and obtain the evaluation result.
Specifically, given the instruction s, the evaluation input (c, x, r), all the subquestions with their answers {(sqt, at)}nt=1, and the original question q as the input prompt, we compute the evaluation score using the generation probability of answer words (i.e., yes and no) from the instruction-tuned PLM (Ke et al., 2022; Zhong et al., 2022):
f(l) = Pθ(l|s, c, x, r, {(sqt, at)}nt=1, q) (2)
score = f(l = yes)
f(l = yes) + f(l = no) (3)

heading: 4 Experiment

heading: 4.1 Dataset
text: We follow Zhong et al. (2022) to adopt two benchmark datasets to test the performance of DecompEval. The statistics of these datasets are shown in Table 1. SummEval (Fabbri et al., 2021): This dataset is a benchmark for evaluation metrics of text summarization. It covers the generated summaries from recent summarization models on the CNN/DailyMail (CNNDM) dataset (Hermann et al., 2015). For each generated summary, it provides the human scores from four dimensions including fluency, coherence, consistency, and relevance. Topical-Chat (Gopalakrishnan et al., 2019): This dataset is a benchmark for knowledge-grounded dialogue generation. Mehri and Eskénazi (2020) collects human annotations for the models trained on Topical-Chat. For each generated response, it provides the human scores from five dimensions2
2We use the description of dimensions in the existing work (Zhong et al., 2022) for fair comparison, which is slightly different from the original paper (Mehri and Eskénazi, 2020).
including naturalness, coherence, engagingness, groundedness, and understandability. Following Zhong et al. (2022), we use the first four dimensions in the main result (§4.4) and the last dimension to test the generalization ability (§4.5).

heading: 4.2 Implementation Detail
text: We choose FLAN-T5 (Chung et al., 2022) as our base model, which is obtained by training T5 (Raffel et al., 2020) on 1.8K tasks described via instructions3. We use FLAN-T5-XL with 3B parameters in the main result and also explore other model scales in §4.8. We follow Zhong et al. (2022) to set the input length to be 1,024. We design the input prompts based on the data formats of FLAN-T5, the evaluation tasks and dimensions. More details about the specific design of input prompts for each dataset / dimension and the sensitivity analysis are included in Appendix A.
As for the evaluation on two datasets, we directly compute summary-level / turn-level evaluation scores for SummEval / Topical-Chat based on our method in most of the dimensions, respectively, except fluency / consistency on SummEval and engagingness on Topical-Chat. For these dimensions, we follow Zhong et al. (2022) to obtain the evaluation scores via averaging (for fluency / consistency on SummEval) (Laban et al., 2022) or cumulating (for engagingness on Topical-Chat) (Deng et al., 2021) individual evaluation results of constituent sentences for fair comparison.

heading: 4.3 Baseline
text: We choose several state-of-the-art untrained and trained metrics as our baselines: MoverScore (Zhao et al., 2019): This metric relies on Earth Mover’s Distance (Rubner et al., 2000) between generated texts and reference texts based on the contextual representations from PLMs. BERTScore (Zhang et al., 2020b): This metric computes the similarity between generated texts
3Although the instruction-tuning datasets of FLAN-T5 cover the CNNDM dataset (Chung et al., 2022), they do not include the generated summaries with human evaluation scores, ensuring no data leak in the experiment.
and reference texts based the contextual representations from BERT (Devlin et al., 2019). USR (Mehri and Eskénazi, 2020): This metric combines the evaluation results of masked language models and dialogue retrieval models which are trained on the dialogue evaluation dataset. BARTScore (Yuan et al., 2021): This metric utilizes the generation probabilities of BART (Lewis et al., 2020) to measure the relationship among source texts, generated texts, and reference texts with different inputs and outputs. We use two variants BARTScore and BARTScore (CNNDM) in the original paper. The latter adopts BART finetuned on the CNNDM dataset as the base model. CTRLEval (Ke et al., 2022): This metric formulates evaluation dimensions as multiple text infilling tasks and uses the ensemble of generation probabilities from PEGASUS (Zhang et al., 2020a) as the evaluation results. UniEval (Zhong et al., 2022): This metric reframes NLG evaluation as a Boolean QA task. It conducts multi-task learning on the related datasets and continual learning on the dimensions of the evaluation dataset with a unified QA format. We use two variants UniEval (Summ) and UniEval (Dial) in the original paper, which are trained on all the dimensions of SummEval and the first four dimensions of Topical-Chat, respectively.
In addition, we also select traditional evaluation metrics based on n-gram overlap like BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) as baselines. We directly re-print the experimental results of baselines if their original papers adopt the same benchmark datasets as ours. Otherwise, we implement the baselines based on the codes and model parameters released by the original papers.

heading: 4.4 Main Result
text: Following Liu et al. (2021) and Zhong et al. (2022), we adopt summary-level Spearman (ρ) and Kendall (τ ) correlation coefficients between human judgments and automatic metrics to assess the performance on the SummEval dataset. The results in Table 2 show that DecompEval achieves state-ofthe-art performance in untrained metrics, indicating the effectiveness of our proposed instructionstyle QA formulation and question decomposition method. Especially, DecompEval can even beat the best-performing trained metric UniEval (Summ) in the dimension of consistency, which shows the po-
tential of instruction-tuned PLMs in the evaluation of generated texts.
We also conduct experiments on the TopicalChat dataset and report turn-level Pearson (r) / Spearman (ρ) correlation coefficients in Table 3 as the existing works (Mehri and Eskénazi, 2020; Zhong et al., 2022) do. Similarly, DecompEval beats all the untrained baselines and even outperforms the trained baseline USR in most of the dimensions. This indicates that DecompEval can successfully adapt to the evaluation of dialogue generation without training on specific datasets. We also find that DecompEval can outperform UniEval (Dial) in the dimension of groundedness. We conjecture that DecompEval may be good at measuring the consistency between generated texts and con-
texts, thereby performing extremely well on consistency in text summarization and groundedness in dialogue generation.

heading: 4.5 Generalization Ability
text: Generalization ability is essential because new evaluation dimensions and tasks may emerge without sufficient data. Thus, we study whether DecompEval can generalize at the dimension / task level better than untrained and trained baselines.

heading: 4.5.1 Generalization to Other Dimensions
text: To compare the performance of DecompEval and untrained / trained baselines on other dimensions, we follow Zhong et al. (2022) to adopt the dimension of understandability on the Topical-Chat
dataset to conduct experiments. The results in the top of Figure 2 show that DecompEval can outperform all the competitive untrained / trained baselines and achieve best performance in the dimension of understandability, which shows its strong dimension-level generalization ability. From the bottom of Figure 2, we can observe that DecompEval maintains stable performance in all these dimensions. In comparison, the trained baseline UniEval (Dial) which is trained on the first four dimensions of Topical-Chat except understandability cannot surpass DecompEval in the evaluation of understandability. The performance of UniEval (Dial) also degrades obviously in understandability compared with the other dimensions, which demonstrates the potential side-effect of over-fitting to specific dimensions.

heading: 4.5.2 Generalization to Other NLG tasks
text: To investigate how DecompEval performs compared with untrained / trained baselines in other NLG tasks in addition to text summarization and dialogue generation, we follow Yuan et al. (2021) and Zhong et al. (2022) to adopt two data-to-text generation datasets SFRES and SFHOT (Wen et al., 2015). These two datasets cover generated texts from structured data in the domain of restaurants and hotels. For each generated text, they provide human scores from two dimensions, i.e., naturalness and informativeness. The number of samples in SFRES / SFHOT is 1,181 / 875, respectively.
The results are shown in Table 4. Our proposed metric DecompEval can still achieve state-of-theart performance in untrained metrics and outperform the trained baselines in most of the dimen-
Dataset SFRES SFHOT
Metric Nat. Info. Nat. Info.
Trained Metric
BARTScore (CNNDM) 0.289 0.238 0.288 0.235 UniEval (Summ) 0.333 0.225 0.320 0.249 UniEval (Dial) 0.291 0.194 0.291 0.196
Untrained Metric
sions. Thus, we believe that DecompEval can successfully improve the generalization ability to multiple NLG tasks via the full utilization of the instruction-tuned PLM without further training. We also illustrate the average of Spearman correlation coefficients in all the dimensions of each dataset in Table 5. Compared with our proposed metric, UniEval (Summ) and UniEval (Dial), as the bestperforming trained metrics on the SummEval and Topical-Chat datasets, respectively, obtain obviously worse performance on the evaluation datasets which they are not trained on, indicating limited task-level generalization ability.

heading: 4.6 Interpretability
text: To verify whether the subquestions with their answers are reliable evidence to interpret the evaluation score, we conduct human evaluation on the generated answers to subquestions. We randomly select 200 subquestions from each dimension of the Topical-Chat dataset. Three annotators are hired to answer these subquestions with yes or no according to the evaluation input, where the human-annotated labels are determined via majority voting. The
Naturalness 17%
14%
60%
9% Coherence
34%
9% 32%
25%
Groundedness
36%
2% 22%
40%
Understandability
27%
7%
53%
13%
Generic Text Elliptical Sentence First-Person Sentence Others
Figure 4: Type distributions of generated texts in the error cases of human evaluation.
human-annotated labels are used as ground-truth labels to measure the quality of generated answers.
The results in Figure 3 show that the accuracy of each dimension is above 0.7, indicating reasonable performance of subquestion answering which serves as interpretable evidence. We manually check the error cases and find that they include three typical types of generated texts, i.e., generic texts (Li et al., 2016), elliptical sentences, and firstperson sentences. The type distributions of generated texts in the error cases are shown in Figure 4. We can observe that generic texts (such as "That is true.") dominate the generated texts in the error cases of coherence / groundedness, while first-person sentences (such as "I think ...") appear more frequently in those of naturalness / understandability. These two types of generated texts are mostly not contradictory to the evaluation input, thereby being commonly recognized by our metric. However, generic texts can only provide limited information while first-person sentences may contain irrelevant contents regarding the evaluation input. Thus, annotators tend to regard them as low-quality ones.
We also provide the case study in Appendix B
to show the interpretable evaluation process of DecompEval.

heading: 4.7 Ablation Study
text: To further investigate the effectiveness of each part in our metric, we conduct detailed ablation studies. We build the following three ablation models which remove three important parts of input prompts in §3.4, respectively: 1) w/o Instruction indicates the model without the instruction s; 2) w/o Decomp. Q&A denotes the model without the decomposed subquestions with their answers {(sqt, at)}nt=1; 3) w/ Prefix Yes/No Que. means moving the yes/no question q to the prefix of the evaluation input behind the instruction. We find that our metric without this yes/no question fails to achieve reasonable performance possibly because it contains the information about evaluation tasks and dimensions.
The results are shown in Table 6. We can observe that all these three parts contribute to final performance. The decomposed subquestions with their answers play a more important role in most of the dimensions, indicating their positive impact as evidence on the model performance in addition to the interpretability. As for instructions, the performance of DecompEval without instructions does not degrade obviously. We conjecture that the yes/no question has explicitly conveyed the information to make the instruction-tuned PLM answer with yes or no. Thus, the impact of instructions may be weakened. The position of yes/no questions has also affected the model performance. From the experimental results, the question in the end of input prompts can obtain better performance than that in the middle part.

heading: 4.8 Analysis on Model Scale
text: We further conduct experiments on the scale of base models, which may impact the capacity of following instructions to evaluate generated texts. We choose FLAN-T5-Base and FLAN-T5-Large additionally, and compare their performance with
FLAN-T5-XL used in our main experiments. The results in Table 7 show that the performance of DecompEval improves on most of the dimensions as the number of parameters in the base model increases. We also find that there is a relatively large margin between the performance of FLANT5-Base/Large and FLAN-T5-XL, especially in the dimensions of naturalness, coherence, and groundedness. This phenomenon is accordant to the findings of existing works (Chung et al., 2022; Wei et al., 2022), where the zero-shot capacity of instruction following mainly emerges in the models of sufficiently large scales.

heading: 5 Discussion
text: Applicability in Non-English Languages: Although the benchmark datasets in the experiment are mainly in English, our method can be also applied to non-English languages. Since our base model FLAN-T5 has some multilingual ability (Chung et al., 2022), we can design instructionstyle questions / subquestions and answer words in the target language to apply DecompEval to non-English evaluation tasks. DecompEval can also adapt to stronger instruction-tuned multilingual PLMs for better applicability in non-English languages. We will further investigate the extensibility of our method to non-English evaluation tasks in the future work.

heading: 6 Conclusion
text: We present an untrained evaluation metric called DecompEval, which formulates NLG evaluation as an instruction-style QA task, and utilizes instruction-tuned PLMs to solve this task via question decomposition. Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics, which also exhibits better dimension-level / task-level generalization ability than trained metrics and improves the interpretability.
Limitations
The limitation of our work includes the following aspects: 1) The instruction-style question which measures the quality of generated texts from different dimensions still needs manual design. Although the questions in our experiment have already involved typical dimensions in text summarization, dialogue generation, and data-to-text generation, we admit that it is hard to cover all the dimensions in various NLG tasks. We believe that this is not a severe problem because we can refer to the definition and human annotation instructions (Mehri and Eskénazi, 2020) of each dimension, which are commonly formulated as questions. We leave the exploration of automatically constructing instruction-style questions for multiple dimensions of NLG evaluation as future work. 2) Due to the limitation of computational resources, the largest base model used in our experiment is FLAN-T5-XL with 3B parameters. Since the ability of instruction following is related to the model scale (Wei et al., 2022), we leave the exploration of adopting larger instruction-tuned PLMs such as FLAN-T5-XXL and OPT-IML (Iyer et al., 2022) as future work.

heading: Acknowledgements
text: This work was supported by the NSFC project (Key project with No. 61936010). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.

heading: A Prompt Design
text: We show the specific prompt design for each dimension of SummEval, Topical-Chat, and SFRES / SFHOT in Table 8, 9, and 10, respectively. The instruction used in all the datasets is s ="Answer the following yes/no question.", as mentioned in §3.2. We refer to the definition and human annotation instructions of each dimension (Fabbri et al., 2021; Mehri and Eskénazi, 2020; Wen et al., 2015) as well as the existing works on QA for evaluation (Deutsch et al., 2021; Zhong et al., 2022) to design evaluation inputs and yes/no questions. The format of subquestions is similar to yes/no questions, where the sentence to be measured is added to the middle part.
To investigate the sensitivity of input prompts, we construct seven grammatical yes/no questions for each dimension of Topical-Chat, covering the original one and three types of lexical variations, i.e., auxiliary verb replacement, synonym replacement, and word reordering. For example, the original yes/no question for naturalness in Table 9 is "Is this response natural to the dialogue history?". After auxiliary verb replacement, the question may start with another auxiliary verb, such as "Does this response have a natural body to the dialogue history?". Similarly, after synonym replacement, the question may have some words which are replaced with their synonyms, such as "Is this response natural given the dialogue history?". As for word reordering, the question may be composed of reordered words, such as "Is this a natural response to the dialogue history?". Note that the subquestions are perturbed accordingly. Then, we illustrate the mean value and standard deviation over the original prompt and perturbed prompts of each dimension in Figure 5, showing the stable performance of DecompEval faced with variations.

heading: B Case Study
text: We provide evaluation cases on the Topical-Chat and SummEval datasets in Table 11 and 12, respectively. We can observe that DecompEval can provide the evaluation scores which are the most accordant to human scores. Also, the subquestions with their answers can act as evidence to indicate the potential low-quality sentence which impacts the overall quality. For example, in Table 11, the second sentence which mentions the concert seems not to be coherent to the topic in the dialogue history (i.e., the crab feast at a stadium). Similarly, in Table
12, the third sentence about patient satisfaction is not relevant to the reference. In comparison, the evaluation scores of other metrics deviate from human scores, while they cannot provide evidence to demonstrate how they predict the evaluation scores.

heading: C Analysis on Decomposition Strategy
text: To judge how useful our decomposed subquestions with generated answers are for interpreting final evaluation scores, we ask the same three annotators to assign an interpretability score to selected samples in §4.6. We adopt a 1-3 Likert scale, where 1 / 2 / 3 means that the decomposition can hardly / partly / comprehensively help understand how the model reaches final scores, respectively. The average interpretability scores over all the selected samples are 2.84 / 2.76 / 2.74 / 2.67 for naturalness / coherence / groundedness / understandability, respectively, showing that our decomposition strategy based on sentences is mostly useful for interpreting final evaluation scores of multiple dimensions.

heading: D Experimental Detail
text: D.1 License of Datasets and Models The licenses of datasets and base models used in our experiments include MIT for the SummEval dataset and Apache-2.0 for the Topical-Chat dataset and the FLAN-T5 model.
D.2 Implementation Detail We use NLTK4 to split generated texts into sentences for the construction of subquestions. As for the computation of Pearson, Spearman, and Kendall correlation coefficients, we use the APIs from SciPy5.
4https://www.nltk.org 5https://scipy.org
D.3 Inference Time
The inference time on the SummEval / TopicalChat dataset is about 28 / 5 minutes, respectively. We test our model on 1 NVIDIA A100 GPU.
D.4 Human Evaluation
The annotation instruction of human evaluation in §4.6 contains two main parts: 1) A subquestion with its corresponding instruction and evaluation input in the same format as Figure 1; 2) An explanation of NLG tasks and dimensions to be measured, which is re-printed from the original paper about benchmark datasets (Mehri and Eskénazi, 2020).
In addition, all the other contents shown to annotators are from the original dataset of Topical-Chat (Gopalakrishnan et al., 2019). We manually check these contents before annotation to avoid potential risks.
We recruit three graduate students as annotators to complete this task. We pay each annotator $0.07 for every subquestion. The payment is determined based on the difficulty of tasks and the length of subquestions.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: 7 (after the section of conclusions)
7 A2. Did you discuss any potential risks of your work? Our work cannot produce new contents. Our main goal is to build a state-of-the-art evaluation metric for text generation which shows great generalization ability and interpretability.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix D.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not create new datasets. We follow many existing works to use benchmark datasets for fair comparison.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4.1, 4.5.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1, 4.5.2
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4.2, Appendix D.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Our proposed method is unsupervised which does not need hyperparameter search. We only need to follow existing works to set the input length for fair comparison.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Our proposed unsupervised method can directly provide deterministic experimental results in a run.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D.2
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4.6

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix D.4

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix D.4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix D.4
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We recruit annotators to measure the quality of texts in benchmark datasets, which is regarded as human labels to test the model performance.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We recruit three graduate students as annotators without collecting demographic and geographic information.

heading: Future Work
text: We leave the exploration of automatically constructing instruction-style questions for multiple dimensions of NLG evaluation as future work. 2) Due to the limitation of computational resources, the largest base model used in our experiment is FLAN-T5-XL with 3B parameters. Since the ability of instruction following is related to the model scale (Wei et al. , 2022), we leave the exploration of adopting larger instruction-tuned PLMs such as FLAN-T5-XXL and OPT-IML (Iyer et al. , 2022) as future work