Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Year: 2023

Abstract:
Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lacking. Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high interannotator agreement. (2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems on three datasets. (3) We conduct a comparative study of four human evaluation protocols, underscoring potential confounding factors in evaluation setups. (4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings have important implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators’ prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

heading: 1 Introduction
text: Human evaluation plays an essential role in both assessing the rapid development of summarization systems in recent years (Lewis et al., 2020a; Zhang et al., 2020a; Brown et al., 2020; Sanh et al., 2022; He et al., 2022) and in assessing the ability of automatic metrics to evaluate such systems as a proxy
∗ Equal contribution
for manual evaluation (Bhandari et al., 2020; Fabbri et al., 2022a; Gao and Wan, 2022). However, while human evaluation is regarded as the gold standard for evaluating both summarization systems and automatic metrics, as suggested by Clark et al. (2021) an evaluation study does not become “gold” automatically without proper practices. For example, achieving a high inter-annotator agreement among annotators can be difficult (Goyal et al., 2022), and there can be a near-zero correlation between the annotations of crowd-workers and expert annotators (Fabbri et al., 2022a). Also, a human evaluation study without a large enough sample size can fail to find statistically significant results due to insufficient statistical power (Card et al., 2020).
Therefore, we believe it is important to ensure that human evaluation can indeed serve as a solid foundation for evaluating summarization systems and automatic metrics. For this, we propose using a robust human evaluation protocol for evaluating the salience of summaries that is more objective by dissecting the summaries into finegrained content units and defining the annotation task based on those units. Specifically, we introduce the Atomic Content Unit (ACU) protocol for summary salience evaluation (§3), which is modified from the Pyramid (Nenkova and Passonneau, 2004) and LitePyramid (Shapira et al., 2019) protocols. We demonstrate that with the ACU protocol, a high inter-annotator agreement can be established among crowd-workers, which leads to more stable system evaluation results and better reproducibility.
We then collect, through both in-house annotation and crowdsourcing, RoSE, a large human evaluation benchmark of human-annotated summaries with the ACU evaluation protocol on recent state-of-the-art summarization systems, which yields higher statistical power (§4). To support evaluation across datasets and domains, our benchmark consists of test sets over three summarization datasets, CNN/DailyMail (CNNDM) (Nalla-
4140
pati et al., 2016), XSum (Narayan et al., 2018), and SamSum (Gliwa et al., 2019), and annotations on the validation set of CNNDM to facilitate automatic metric training. To gain further insights into the characteristics of different evaluation protocols, we conduct human evaluation with three other protocols (§5). Specifically, we analyze protocol differences in the context of both fine-tuned models and large language models (LLMs) in a zero-shot setting such as GPT-3 (Brown et al., 2020). We find that different protocols can lead to drastically different results, which can be affected by annotators’ prior preferences, highlighting the importance of aligning the protocol with the summary quality intended to be evaluated. We note that our benchmark enables a more trustworthy evaluation of automatic metrics (§6), as shown by statistical characteristics such as tighter confidence intervals and more statistically significant comparisons (§6.2). Our evaluation includes recent methods based on LLMs (Fu et al., 2023; Liu et al., 2023), and we found that they cannot outperform traditional metrics despite their successes on related benchmarks such as SummEval (Fabbri et al., 2022a).
We summarize our key findings in Tab. 1. Our contributions are the following: (1) We propose the ACU protocol for high-agreement human evaluation of summary salience. (2) We curate the RoSE benchmark, consisting of 22000 summary-level annotations and requiring over 150 hours of in-house annotation, across three summarization datasets, which can lay a solid foundation for training and evaluating automatic metrics.1 (3) We compare four human evaluation protocols for summarization
1We release our benchmark and evaluation scripts at https://github.com/Yale-LILY/ROSE.
and show how they can lead to drastically different model preferences. (4) We evaluate automatic metrics across different human evaluation protocols and call for human evaluation to be conducted with a clear evaluation target aligned with the evaluated systems or metrics, such that task-specific qualities can be evaluated without the impact of general, input-agnostic preferences of annotators. We note that the implications of our findings can become even more critical with the progress of LLMs trained with human preference feedback (Ouyang et al., 2022) and call for a more rigorous human evaluation of LLM performance.

heading: 2 Related Work
text: Human Evaluation Benchmarks Human annotations are essential to the analysis of summarization research progress. Thus, recent efforts have focused on aggregating model outputs and annotating them according to specific quality dimensions (Huang et al., 2020; Bhandari et al., 2020; Stiennon et al., 2020; Zhang and Bansal, 2021; Fabbri et al., 2022a; Gao and Wan, 2022). The most relevant work to ours is Bhandari et al. (2020), which annotates summaries according to semantic content units, motivated by the Pyramid (Nenkova and Passonneau, 2004) and LitePyramid (Shapira et al., 2019) protocols. However, this benchmark only covers a single dataset (CNNDM) without a focus on similarly-performing state-of-the-art systems, which may skew metric analysis (Tang et al., 2022a) and not fully reflect realistic scenarios (Deutsch et al., 2022). In contrast, our benchmark consists only of outputs from recently-introduced models over three datasets.
Summarization Meta-Evaluation With a human evaluation dataset, there exist many directions of meta-evaluation, or re-evaluation of the current state of evaluation, such as metric performance analyses, understanding model strengths, and human evaluation protocol comparisons.
Within metric meta-analysis, several studies have focused on the analysis of ROUGE (Lin, 2004b), and its variations (Rankel et al., 2013; Graham, 2015), across domains such as news (Lin, 2004a), meeting summarization (Liu and Liu, 2008), and scientific articles (Cohan and Goharian, 2016). Other studies analyze a broader set of metrics (Peyrard, 2019; Bhandari et al., 2020; Deutsch and Roth, 2020; Fabbri et al., 2022a; Gabriel et al., 2021; Kasai et al., 2022b), including those specific to factual consistency evaluation (Kryscinski et al., 2020; Durmus et al., 2020; Wang et al., 2020; Maynez et al., 2020; Laban et al., 20d; Fabbri et al., 2022b; Honovich et al., 2022; Tam et al., 2022).
Regarding re-evaluating model performance, a recent line of work has focused on evaluating zeroshot large language models (Goyal et al., 2022; Liang et al., 2022; Tam et al., 2022), noting their high performance compared to smaller models.
As for the further understanding of human evaluation, prior work has compared approaches to human evaluation (Hardy et al., 2019), studied annotation protocols for quality dimensions such as linguistic quality (Steen and Markert, 2021) and factual consistency (Tang et al., 2022b), and noted the effects of human annotation inconsistencies on system rankings (Owczarzak et al., 2012). The unreliability and cost of human evaluation in certain settings have been emphasized (Chaganty et al., 2018; Clark et al., 2021), with some work noting that thousands of costly data points may need to be collected in order to draw statistically significant conclusions (Wei and Jia, 2021). Our meta-analysis focuses on this latter aspect, and we further analyze potential confounding factors in evaluation such as length and protocol design, with respect to both small and large zero-shot language models.

heading: 3 Atomic Content Units for Summarization Evaluation
text: We now describe our Atomic Content Unit (ACU) annotation protocol for reference-based summary salience evaluation, including the procedure of writing ACUs based on reference summaries and matching the written ACUs with system outputs.

heading: 3.1 Preliminaries
text: In this work, we focus on a specific summarization meta-evaluation study on summary salience. Salience is a desired summary quality that requires the summary to include all and only important information of the input article. The human evaluation of summary salience can be conducted in either reference-free or reference-based manners. The former asks the annotators to assess the summary directly based on the input article (Fabbri et al., 2022a), while the latter requires the annotators to assess the information overlap between the system output and reference summary (Bhandari et al., 2020), under the assumption that the reference summary is the gold standard of summary salience.2 Given that reference-based protocols are more constrained, we focus on reference-based evaluation for our human judgment dataset collection, and we conduct a comparison of protocols in §5.

heading: 3.2 ACU Annotation Protocol
text: Inspired by the Pyramid (Nenkova and Passonneau, 2004) and LitePyramid (Shapira et al., 2019) protocols and subsequent annotation collection efforts (Bhandari et al., 2020; Zhang and Bansal, 2021), the ACU protocol is designed to reduce the subjectivity of reference-based human evaluation by simplifying the basic annotation unit – the annotators only need to decide on the presence of a single fact, extracted from one text sequence, in another text sequence, to which a binary label can be assigned with more objectivity. Specifically, the evaluation process is decomposed into two steps: (1) ACU Writing – extracting facts from one text sequence, and (2) ACU Matching – checking for the presence of the extracted facts in another sequence. We formulate the ACU protocol as a recall-based protocol, such that the first step only needs to be performed once for the reference summary, allowing for reproducibility and reuse of these units when performing matching on new system outputs. ACU Writing While the LitePyramid approach defines its basic content unit as a sentence containing a brief fact, we follow Bhandari et al. (2020) to emphasize a shorter, more fine-grained information unit. Specifically, we define the ACU protocol with the concept of atomic facts – elementary information units in the reference summaries, which no
2We note salience can be an inherently subjective quality, and the reference summary of common datasets may not always be the actual “gold standard,” discussed more in §7.
Oscar collided with Arsenal goalkeeper David Ospina in the 16th minute of the London derby. The Brazilian was substituted at half-time and Jose Mourinho said he suffered `possible concussion'. Oscar was knocked back by the goalkeeper but Michael Oliver didn't award Chelsea a penalty.
System Summary
Chelsea weren't awarded a penalty for David Ospina's clash with Oscar. Arsenal goalkeeper clattered Oscar inside the box. Brazilian was taken off at half-time, with Didier Drogba replacing him. Reference Summary
Oscar collided with Arsenal goalkeeper David Ospina in the 16th minute of the London derby. The Brazilian was substituted at half-time and Jose Mourinho said he suffered `possible concussion'. Oscar was knocked back by the goalkeeper but Michael Oliver didn't award Chelsea a penalty.
System Summary
❌
Ch lsea weren’t awarded a penalty.
David Ospina clashed with Oscar.
David Ospina clattered Oscar.
David Ospina plays for Arsenal.
David Ospina is a goalkeeper.
The clash occurred inside the box.
Oscar is Brazilian.
Oscar was taken off at half time.
Didier Drogba replaced Oscar.
Automic Content Units (ACUs)
✔ ✔ ✔ ✔ ✔
❌
❌ ✔ ✔
Reference Summary
Chelsea weren’t awarded a penalty.
David Ospina clashed with Oscar.
David Ospina clattered Oscar.
David Ospina plays for Arsenal.
David Ospina is a goalkeeper.
The clash occurred inside the box.
Oscar is Brazilian.
Oscar was taken off at half time.
Didier Drogba replaced Oscar. Oscar collided with Arsenal goalkeeper David Ospina in the 16th minute of the London derby. The Brazilian was substituted at
half-time and Jose Mourinho said he suffered ‘possible concussion’. Oscar was knocked back by the goalkeeper but Michael Oliver didn't award Chelsea a penalty.
System Summary
Automic Content Units (ACUs) Chelsea weren’t awarded a penalty for David Ospina’s clash with Oscar. Arsenal goalkeeper clattered Oscar inside the box. Brazilian was taken off at half-time, with Didier Drogba replacing him. ACU Writing
ACU Matching
longer need to be further split for the purpose of reducing ambiguity in human evaluation.3 Then, ACUs re constructed based on one atomic fact and other mini al, necessary information.
Fig. 1 shows an example of the written ACUs. To ensure annotation quality, we (the authors) write all the ACUs used in this work. We define guidelines to standardize the annotation process; for each summary sentence the annotator creates an ACU constituting the main information from the subject of the main clause (e.g., root), followed by additional ACUs for other facts while including the minimal necessary information from the root. We provide rules for dealing with quotations, extraneous adjectives, noisy summaries, and additional cases. We note that there can still be inherent subjectivity in the written ACUs among different annotators even with the provided guidelines. However, such subjectivity should be unbiased in summary comparison because all the candidate summaries are evaluated by the same set of written ACUs. ACU Matching Given ACUs written for a set of reference summaries, our protocol evaluates summarization system performance by checking the presence of the ACUs in the system-generated summaries as illustrated in Fig. 1. For this step, we recruit annotators on Amazon Mechanical Turk4 (MTurk). The annotators must pass a qualification test, and additional requirements are specified in Appendix A. Besides displaying the ACUs and the system outputs, we also provide the reference summaries to be used as context for the ACUs. Scoring Summaries with ACU ACU matching annotations can be aggregated into summary scores. We first define an un-normalized ACU score f of a candidate summary s given a set of ACUs A as:
f(s,A) = |As||A| , (1)
3We note that it can be impossible to provide a practical definition of atomic facts. Instead, we use it as a general concept for fine-grained information units.
4https://www.mturk.com/

heading: Dataset Split #Doc. #Sys. #ACU #Summ.
text: CNNDM Test 500 12 5.6k 6k CNNDM V lid 1,000 8 11.6k 8k XSum Test 500 8 2.3k 4k SamSum Test 500 8 2.3k 4k
Table 2: Statistics of the collected annotations. #Doc. is the number of input documents, #Sys. is the number of summarization systems used for collection. #ACU is the total number of written ACUs. #Summ. is the total number of summary-level annotations, which are aggregated over three annotations on the test sets, and a single annotation on the validation set of CNNDM.
where As is a subset of A that is matched with s. We note that f by default is a recall based score with respect to the reference summary r. Therefore, we also define a normalized ACU score f̃ as:
f̃α(s,A, r) = emin (0, 1− |s||r| α )f(s,A), (2)
where |s|, |r| are the length (i.e., number of words) of the candidate summary s and the reference summary r respectively, and α is a positive number controlling the strength of the normalization. This normalization is in effect a redundancy penalty, which penalizes the summaries longer than the reference and resembles the brevity penalty in BLEU (Papineni et al., 2002). In practice, we set the value of α by de-correlating f̃ with the summary length using the collected ACU annotations.

heading: 3.3 ACU Annotation Collection
text: We collect ACU annotations on three summarization datasets: CNNDM (Nallapati et al., 2016), XSum (Narayan et al., 2018), and SamSum (Gliwa et al., 2019). To reflect the latest progress in text summarization, we collect and annotate the generated summaries of pre-trained summarization systems proposed in recent years.5 Detailed informa-
5We release all of the system outputs with a unified, cased, untokenized format to facilitate future research.
tion about the summarization systems we used can be found in Appendix A.2.
Table 2 shows the statistics of the collected annotations. The annotations are collected from the test set of the above datasets, and additionally from the validation set of CNNDM to facilitate the training of automatic evaluation metrics. In total, we collect around 21.8k ACU-level annotations and around 22k summary-level annotations, aggregated over around 50k individual summary-level judgments.
To calculate inter-annotator agreement, we use Krippendorff’s alpha (Krippendorff, 2011). The aggregated summary-level agreement score of ACU matching is 0.7571, and the ACU-level agreement score is 0.7528. These agreement scores are higher than prior collections, such as RealSumm (Bhandari et al., 2020) and SummEval (Fabbri et al., 2022a), which have an average agreement score of crowd-workers 0.66 and 0.49, respectively.

heading: 4 RoSE Benchmark Analysis
text: We first analyze the robustness of our collected annotations and a case study on the system outputs.

heading: 4.1 Power Analysis
text: We analyze the statistical power of our collected human annotations to study whether it can yield stable and trustworthy results (Card et al., 2020). Statistical power is the probability that the null hypothesis of a statistical significance test is rejected given there is a real effect. For example, for a human evaluation study that compares the performance of two genuinely different systems, a statistical power of 0.80 means there is an 80% chance that a significant difference will be observed. Further details can be found in Appendix B.1.
We conduct the power analysis for pair-wise system comparisons with ACU scores (Eq. 1) focusing on two factors, the number of test examples and the observed system difference. Specifically, we run the power analysis with varying sample sizes, and group the system pairs into buckets according to their performance difference, as determined by ROUGE1 recall scores (Fig.2).6 We observe the following: (1) A high statistical power7 is difficult to reach when the system performance is similar.
6We note that these scores are proxies of the true system differences, and the power analysis is based on the assumption that the systems have significantly different performance.
7An experiment is usually considered sufficiently powered if the statistical power is over 0.80.
Notably, while the sample size of the human evaluation performed in recent work is typically around 50-100,8 such sample size can only reach a power of 0.80 when the ROUGE1 recall score difference is above 5. (2) Increasing the sample size can effectively raise the statistical power. For example, when the system performance difference is within the range of 1-2 points, the power of a 500-sample set is around 0.50 while a 100-sample set only has a power of around 0.20. The results of power analysis on three datasets with both ROUGE and ACU score differences are provided in Appendix B.2 with the same patterns, which indicates that our dataset can provide more stable summarization system evaluation thanks to its higher statistical power.

heading: 4.2 Summarization System Analysis
text: As a case study, in Tab. 3 we analyze the summary characteristics of the recent summarization systems we collected on the CNNDM test set. XSum and SamSum results are shown in Appendix A.3. Apart from the ACU scores, we note that the average summary length of different systems can greatly vary, and such differences are not always captured by the widely-used ROUGE F1. For example, the length of GSum (Dou et al., 2021) is around 40% longer than GLOBAL (Ma et al., 2021) while they have very similar ROUGE1 F1 scores. Besides, we note all systems in Tab. 3 have longer summaries than the reference summaries, whose average length is only 54.93. This can be a potential risk to users who may prefer shorter, more concise
8We provide a brief survey of the practices of human evaluation in recent text summarization research in Appendix F.
summaries. Meanwhile, the systems that generate longer summaries may be favored by users who prefer more informative summaries. Therefore, we join the previous work (Sun et al., 2019; Song et al., 2021; Gehrmann et al., 2022; Goyal et al., 2022) in advocating treating summary lengths as a separate aspect of summary quality in evaluation, as in earlier work in summarization research.9

heading: 5 Evaluating Annotation Protocols
text: Apart from ACU annotations, we collect human annotations with three different protocols to better understand their characteristics. Specifically, two reference-free protocols are investigated: Prior protocol evaluates the annotators’ preferences of summaries without the input document, while Ref-free protocol evaluates if summaries cover the salient information of the input document. We also consider one reference-based protocol, Ref-based, which evaluates the content similarity between the generated and reference summaries. Appendix D.1 provides detailed instructions for each protocol.

heading: 5.1 Annotation Collection
text: We collected three annotations per summary on a 100-example subset of the above CNNDM test set using the same pool of workers from our ACU qualification. Except for ACU, all of the summaries from different systems are evaluated within a single task with a score from 1 (worst) to 5 (best), similar
9For example, the DUC evaluation campaigns set a prespecified maximum summary length, or summary budget.
to the EASL protocol (Sakaguchi and Van Durme, 2018). We collect (1) annotations of the 12 above systems, with an inter-annotator agreement (Krippendorff’s alpha) of 0.3455, 0.2201, 0.2741 on Prior, Ref-free, Ref-based protocols respectively; (2) annotations for summaries from GPT-3 (Brown et al., 2020),10 T0 (Sanh et al., 2022), BRIO, and BART to better understand annotation protocols with respect to recently introduced large language models applied to zero-shot summarization.

heading: 5.2 Results Analysis
text: We investigate both the summary-level and systemlevel correlations of evaluation results of different protocols to study their inherent similarity. Details of correlation calculation are in Appendix C. Results on Fine-tuned Models We show the system-level protocol correlation when evaluating the fine-tuned models in Tab. 4, and the summarylevel correlation can be found in Appendix D.2. We use the normalized ACU score (Eq. 2) because the other evaluation protocols are supposed to resemble an F1 score, while the ACU score is by definition recall-based. We have the following observations: (1) The Ref-free protocol has a strong correlation with the Prior protocol, suggesting that the latter may have a large impact on the annotator’s document-based judgments. (2) Both the Prior and Ref-free protocols have a strong correlation with summary length, showing that annotators may favor longer summaries. (3) The Ref-free protocol and the Ref-based protocol have a negative correlation while ideally they are supposed to measure similar quality aspects.
We perform power analysis on the results following the procedure in §4.1 and found that ACU protocol can yield higher statistical power than the Ref-based protocol, suggesting that the ACU protocol leads to more robust evaluation results. We also found that the reference-free Prior and Ref-free
10We use the “text-davinci-002” version of GPT-3.
protocols have higher power than the referencebased protocols. However, we note that they are not directly comparable because they have different underlying evaluation targets, as shown by the near-zero correlation between them. Further details are provided in Appendix D.2. Results on Large Language Models The results are shown in Tab. 5. Apart from the system outputs, we also annotate reference summaries for referencefree protocols. We found that under the Ref-free protocol, GPT-3 receives the highest score while the reference summary is the least favorite one, similar to the findings of recent work (Goyal et al., 2022; Liang et al., 2022). However, we found the same pattern with the Prior protocol, showing that the annotators have a prior preference for GPT-3. We provide an example in Appendix D.2 comparing GPT-3 and BRIO summaries under different protocols. Given the strong correlation between the Prior and Ref-free protocols, we note that there is a risk that the annotators’ decisions are affected by their prior preferences that are not genuinely related to the task requirement. As a further investigation, we conduct an annotator-based case study including 4 annotators who annotated around 20 examples in this task, in which we compare two summary-level correlations (Eq. 3) given a specific annotator: (1) the correlation between their own Ref-free protocol scores and Prior scores; (2) the correlation between their Ref-free scores and the Ref-free scores averaged over the other annotations on each example. We found that the average value of the former is 0.404 while the latter is only 0.188, suggesting that the annotators’ own Prior score is a better prediction of their Ref-free score than the Ref-free score of other annotators.

heading: 6 Evaluating Automatic Metrics
text: We analyze several representative automatic metrics, with additional results in Appendix E on 50
automatic metric variants. We focus the metric evaluation on ACU annotations because of two insights from §5: (1) Reference-based metrics should be evaluated with reference-based human evaluation. (2) ACU protocol provides higher statistical power than the summary-level Ref-based protocol.

heading: 6.1 Metric Evaluation with ACU Annotations
text: We use the correlations between automatic metric scores and ACU annotation scores of system outputs to analyze and compare automatic metric performance. The following metrics are evaluated:
(1) lexical overlap based metrics, ROUGE (Lin, 2004b), METEOR (Lavie and Agarwal, 2007), CHRF (Popović, 2015); (2) pre-trained language model based metrics, BERTScore (Zhang et al., 2020c), BARTScore (Yuan et al., 2021); (3) question-answering based metrics, SummaQA (Scialom et al., 2019), QAEval (Deutsch et al., 2021a); (4) Lite3Pyramid (Zhang and Bansal, 2021), which automates the LitePyramid evaluation process; (5) evaluation methods based on large language models, GPTScore (Fu et al., 2023) and G-Eval (Liu et al., 2023), with two variants that are based on GPT-3.511 (G-Eval-3.5) and GPT-412 (OpenAI, 2023) (G-Eval-4) respectively. We note that for LLM-based evaluation we require the metric to calculate the recall score. For G-
11OpenAI’s gpt-3.5-turbo-0301: https://platform. openai.com/docs/models/gpt-3-5.
12OpenAI’s gpt-4-0314: https://platform.openai. com/docs/models/gpt-4.
Eval-3.5 we report two variants that are based on greedy decoding (G-Eval-3.5) and sampling (GEval-3.5-S) respectively, Details of the LLM-based evaluation are in Appendix E.2.
Tab. 6 shows the results, with additional results of more metrics in Appendix E.3. We note: (1) Several automatic metrics from the different families of methods (e.g., ROUGE, BARTScore) are all able to achieve a relatively high correlation with the ACU scores, especially at the system level. (2) Metric performance varies across different datasets. In particular, metrics tend to have stronger correlations on the SamSum dataset and weaker correlations on the XSum dataset. We hypothesize that one reason is that the reference summaries of the XSum dataset contain more complex structures. (3) Despite their successes (Fu et al., 2023; Liu et al., 2023) in other human evaluation benchmarks such as SummEval, LLM-based automatic evaluation cannot outperform traditional methods such as ROUGE on RoSE. Moreover, their low summarylevel correlation with ACU scores suggests that their predicted scores may not be well-calibrated.
Following Deutsch et al. (2022), we further investigate metric performance when evaluating system pairs with varying performance differences. Specifically, we group the system pairs based on the difference of their ACU scores into different buckets and calculate the modified Kendall’s correlation (Deutsch et al., 2022) on each bucket. The system pairs in each bucket are provided in Appendix E.4. Tab. 7 shows that the automatic metrics generally perform worse when they are used to evaluate similar-performing systems.

heading: 6.2 Analysis of Metric Evaluation
text: We analyze the metric evaluation with respect to the statistical characteristics and the impact of different human evaluation protocols on metric evaluation. Confidence Interval We select several representative automatic metrics and calculate the confidence intervals of their system-level correlations with the ACU scores using bootstrapping. Similar to Deutsch et al. (2021b), we find that the confidence intervals are large. However, we found that having a larger sample size can effectively reduce the confidence interval, which further shows the importance of increasing the statistical power of the human evaluation dataset as discussed in §4.1. We provide further details in Appendix E.5. Power Analysis of Metric Comparison We conduct a power analysis of pair-wise metric comparison with around 200 pairs, which corresponds to the chance of a statistical significance result being found. More details can be found in Appendix E.6. The results are in Fig.3, showing similar patterns as in the power analysis of summarization system comparison (§4.1): (1) Significant results are difficult to find when the metric performance is similar; (2) Increasing the sample size can effectively increase the chance of finding significant results. Correlations under Different Human Evaluation Protocols We analyze the metric correlations under different human evaluation protocols (§5). The results are shown in Tab. 8, with more results in Appendix E.7. We note: (1) Metric performance differs greatly under different protocols, likely because the protocols can have weak correlations with each other (§5.2). (2) The reference-based
automatic metrics generally perform better under reference-based evaluation protocols, but can have negative correlations with reference-free protocols.

heading: 7 Conclusion and Implications
text: We introduce RoSE, a benchmark whose underlying protocol and scale allow for more robust summarization evaluation across three datasets. With our benchmark, we re-evaluate the current state of human evaluation and its implications for both summarization system and automatic metric development, and we suggest the following: (1) Alignment in metric evaluation. To evaluate automatic metrics, it is important to use an appropriate human evaluation protocol that captures the intended quality dimension to be measured. For example, reference-based automatic metrics should be evaluated by reference-based human evaluation, which disentangles metric performance from the impact of reference summaries. (2) Alignment in system evaluation. We advocate for targeted evaluation, which clearly defines the intended evaluation quality. Specifically, text summarization, as a conditional generation task, should be defined by both the source and target texts along with pre-specified, desired characteristics. Clearly specifying characteristics to be measured can lead to more reliable and objective evaluation results. This will be even more important for LLMs pretrained with human preference feedback for disentangling annotators’ prior preferences for LLMs with the task-specific summary quality. (3) Alignment between NLP datasets and tasks. Human judgments for summary quality can be diverse and affected by various factors such as summary lengths, and reference summaries are not al-
ways favored. Therefore, existing summarization datasets (e.g. CNNDM) should only be used for the appropriate tasks. For example, they can be used to define a summarization task with specific requirements (e.g. maximum summary lengths), and be important for studying reference-based metrics.

heading: 8 Limitations
text: Biases may be present in the data annotator as well as in the data the models were pretrained on. Furthermore, we only include English-language data in our benchmark and analysis. Recent work has noted that language models may be susceptible to learning such data biases (Lucy and Bamman, 2021), thus we request that the users be aware of potential issues in downstream use cases.
As described in Appendix D.1, we take measures to ensure a high quality benchmark. There will inevitably be noise in the dataset collection process, either in the ACU writing or matching step, and high agreement of annotations does not necessarily coincide with correctness. However, we believe that the steps taken to spot check ACU writing and filter workers for ACU matching allow us to curate a high-quality benchmark. Furthermore, we encourage the community to analyze and improve RoSE in the spirit of evolving, living benchmarks (Gehrmann et al., 2021).
For reference-based evaluation, questions about reference quality arise naturally. We also note that the original Pyramid protocol was designed for multi-reference evaluation and weighting of semantic content units, while we do not weight ACUs during aggregation. As discussed above, we argue that our benchmark and analysis are still valuable given the purpose of studying conditional generation and evaluating automatic metrics for semantic overlap in targeted evaluation. We view the collection of high-quality reference summaries as a valuable, orthogonal direction to this work, and we plan to explore ACU weighting in future work.

heading: Acknowledgements
text: We thank the anonymous reviewers for their constructive comments. We are grateful to Arman Cohan for insightful discussions and suggestions, Daniel Deutsch for the initial discussions, Richard Yuanzhe Pang for sharing system outputs, and Philippe Laban for valuable comments.

heading: A Benchmark Data Collection

heading: A.1 Detailed Settings
text: We discuss the detailed settings of ACU collection in §3.2. To ensure the consistency of written ACUs among different annotators, we require each annotator to be familiar with the annotation protocol and proofread each other’s annotations to resolve any differences in initial annotations. After establishing a consistent understanding of the task, we have each reference summary annotated by one annotator. We note that there are multiple valid ways of writing the same atomic fact. In preliminary protocol analysis, we had multiple annotators write ACUs for the same reference summaries and did not find large differences in downstream interannotator agreement for ACU matching. The average time to write ACUs of one summary ranges from 2 to 5 minutes, and the overall annotation time for ACU writing is around 150 hours.
We use the following qualifications, in addition to a qualification test, to recruit MTurk workers with good track records: HIT approval rate greater than or equal to 98%, number of HITs approved greater than or equal to 10000, and located in either the United Kingdom or the United States. Workers were compensated between $0.15 and $0.55 per summary-level ACU HITs, with HITs bucketed according to the number of ACUs to be matched. For protocol comparison HITs, workers were compensated between $1 and $3. All HITs were carefully calibrated to equal a $12/hour pay rate.
The datasets we used for the collection are CNNDM, XSum and SamSum. The data release licenses are the Apache License for CNNDM and XSum, and CC BY-NC-ND 4.0 for SamSum. Our collected benchmark will be released under the 3- Clause BSD license.

heading: A.2 Summarization Models
text: We list the summarization models for ACUs annotations on CNNDM, XSum, and SamSum in §3.3.

heading: CNNDM Systems:
text: BART (Lewis et al., 2020b) introduce a denoising autoencoder for pretraining sequence to sequence tasks which is applicable to both natural language understanding and generation tasks. Pegasus (Zhang et al., 2020b) introduce a model pretrained with a novel objective function designed for summarization by which important sentences are removed from an input document and then generated from the remaining sentences. MatchSum (Zhong et al., 2020) propose a summary-level extractive system using semantic match between the extracted summary and the source document. CTRLSum (He et al., 2020) introduce a method for controllable summarization based on keyword or descriptive prompt control tokens. CLIFF (Cao and Wang, 2021) propose to use contrastive learning to improve factual consistency. We use the CLIFF output that uses an underlying BART model. GOLD (Pang and He, 2021) frames text generation as an offline reinforcement learning problem, using importance weighting and assigning weights to examples that receive a higher probability from the generation model. GSum (Dou et al., 2021) is a framework for incorporating forms of summarization guidance.
SimCLS (Liu and Liu, 2021) is a two-stage summarization model where candidates from BART are reranking by a RoBERTa (Liu et al., 2019) scoring model trained using contrastive learning. FROST (Narayan et al., 20d) propose to do content planning in both pretraining and finetuning summarization models with plans in the form of entity chains. GLOBAL (Ma et al., 2021) propose a variation of beam search that takes into account the global attention distribution. BRIO (Liu et al., 2022) proposes to train a summarization model both as a token-level generator and an evaluator of sequence candidates through contrastive reranking. BRIO-Ext (Liu et al., 2022) uses BRIO’s reranker on candidate extractive summaries from MatchSum. The following models were included in protocolcomparison annotations. T0 (Sanh et al., 2022) introduces a prompt-based model that is fine-tuned on multiple tasks, including summarization. GPT-3 (Brown et al., 2020) is the davinci-002 model trained on human demonstrations and model outputs highly rated by humans.13
XSum Systems For XSum we reuse several of the above models with their XSum-trained checkpoints as well as several variations from the above paper due to the scarcity of widely-available, easilyreproducible XSum outputs. BART (Lewis et al., 2020b) Pegasus (Zhang et al., 2020b) CLIFF (Cao and Wang, 2021) CLIFF-Pegasus (Cao and Wang, 2021) is the CLIFF algorithm applied with Pegasus as the underlying model. FROST (Narayan et al., 20d) BRIO (Liu et al., 2022) BRIO-ranking (Liu et al., 2022) is the paper’s reranking model. BART-beam-patience (Kasai et al., 2022a)

heading: SamSum Systems
text: We use system outputs from Gao and Wan (2022). BART (Lewis et al., 2020b) Pegasus (Zhang et al., 2020b)
13https://beta.openai.com/docs/ model-index-for-researchers
UniLM (Dong et al., 2019) is a model pretrained on unidirection, bidirection, and sequenceto-sequence language modeling tasks.
Ctrl-DiaSumm (Liu and Chen, 2021) propose controlled generation using named entity plans.
PLM-BART (Feng et al., 2021) use DialogGPT (Zhang et al., 2020d) to annotate input dialogues before finetuning.
CODS (Wu et al., 2021) propose a two-stage generation model that first generates a sketch that is then used as a signal to the second-stage summarizer.
MV-BART (Chen and Yang, 2020) propose a multiview encoder and a decoder that attends to these conversation views.
S-BART (Chen and Yang, 2020) encodes utterances as well as action and discourse graphs and introduces a decoder that attends to these different levels of granularity.

heading: A.3 ACU Scores of Summarization Models
text: We report the ACU scores of the summarization systems we annotated on the XSum and SamSum datasets (§4.2) in Tab. 9 and Tab. 10, respectively. The results on CNNDM can be found in Tab. 3. For the normalized ACU score (Eq. 2), we set the normalization strength α to 2, 5, 0.5, on CNNDM, XSum, SamSum, respectively, by a grid search for de-correlating the summary length and the normalized score at the summary level.

heading: B Power Analysis

heading: B.1 Detailed Settings
text: We describe the algorithm for the power analysis in §4.1 in Alg.1. While prior work (Card et al., 2020; Wei and Jia, 2021) uses parametric methods to estimate statistical power, we conduct the power analysis with the bootstrapping test (Tibshirani and Efron, 1993) as recent work (Deutsch et al., 2021b) has shown that the assumptions of the parametric methods do not always hold for human evaluation of text summarization. The process involves (1) iteratively sampling a set of examples with a certain sample size from an existing dataset, (2) running the significance test on the sampled set, and (3) estimating the power by averaging across the trials.
The essence of the test is to have a series of simulated datasets sampled from the existing dataset and run the significance test on the sampled sets. Here the existing dataset X consists of humanannotated scores of system outputs. We use paired bootstrapping for the significance test. The power analysis is conducted over all the system pairs.

heading: B.2 Powers of ACU Annotations
text: Fig.4, Fig.5, and Fig.6 show the power analysis results on CNNDM, XSum and SamSum respectively in §4.1, where the system pairs are grouped by their performance difference in either ACU or ROUGE1 recall scores. Similar to our findings on CNNDM in §4.1, we observe that increasing the sample size can effectively raise the statistical power.
Algorithm 1 Power Analysis Input: n (Sample Size) Input: X (Existing Dataset) Input: m (Trial Number) Output: p (Statistical Power) p← 0 for i = 0 to m do
X̂ ← Sampling n examples from X with replacement
p̃← Running the significance test on X̂ if p̃ < 0.05 then
p← p+ 1 end if
end for p← p/m return p

heading: C Calculating Correlations
text: We use correlations to analyze the inherent similarity between different human evaluation protocols, and the performance of automatic metrics, which is evaluated based on the correlations between the metric-calculated summary scores and the humanannotated summary scores. Specifically, given m system outputs on each of the n data samples and two different evaluation methods (e.g., human evaluation and an automatic metric) resulting in two n-row, m-column score matrices X and Y , the summary-level correlation is an average of samplewise correlations:
rsum(X,Y ) =
∑ i C(Xi, Yi)
n , (3)
where Xi, Yi are the evaluation results on the i-th data sample and C is a function calculating a correlation coefficient (e.g., the Pearson correlation coefficient). In contrast, the system-level correlation is calculated on the aggregated system scores:
rsys(X,Y ) = C(X̄, Ȳ ), (4) where X̄ and Ȳ contain m entries which are the system scores from the two evaluation methods averaged across n data samples, e.g., X̄0 = ∑ iXi,0/n.

heading: D Protocol Comparison

heading: D.1 Data Collection Details
text: The 100 examples chosen for annotation in §5 are a subset of the CNNDM ACU test set, and as here we aim to analyze trends among protocols as opposed to observing statistically significant differences among systems, we believe 100 examples suffice for this collection.

heading: Protocol w/ Doc w/ Ref Fine-grained
text: We summarize and compare different protocols in Tab. 11. We provide the following instructions to annotators for non-ACU annotations. We will release the full interface and instructions. Prior: We ask the annotator to imagine each of the candidate summaries to be evaluated as a summary of a longer news article and answer the following question: how good do you think this summary is? Ref-free: The rating measures how well the summary captures the key points of the news article. Consider whether all and only the important aspects are contained in the summary. Ref-based: The rating measures how similar two summaries are. The similarity depends on if the summaries contain similar information, not if they use the same words.

heading: D.2 Results Analysis
text: We present the result analysis of §5.2 here.
Summary Level Correlation We show the summary-level Pearson’s Correlation Coefficients among different protocols in Tab. 12.
Power Analysis The power analysis results on the Prior, Ref-free, Ref-based, and ACU protocols are shown in Fig. 7.
Case Study We show a case study in Tab. 13 comparing the summaries generated by BRIO and GPT-3. GPT-3 scores higher on Prior and Reffree (3.33/3.33 for BRIO and 3.66/4.00 for GPT-3). However, the BRIO summary scores 0.77 on unnormalized ACU annotations while GPT-3 scores 0.33. Also, Ref-based annotations favor BRIO over GPT-3 (3.66 vs. 3.33).

heading: E Metric Analysis

heading: E.1 Metrics
text: We provide additional metric details as well as results for other metrics in §6. Note that for ROUGE,
we use the Python implementation. 14 BLEU (Papineni et al., 2002) is a corpus-level precision-focused metric that calculates n-gram overlap and includes a brevity penalty. CIDEr (Vedantam et al., 2015) computes {1-4}- gram co-occurrences, down-weighting common ngrams and calculating cosine similarity between
14https://pypi.org/project/ROUGE-score/
the n-grams of the candidate and reference texts. Statistics (Grusky et al., 2018) reports summary statistics such as the length, novel and repeated n-grams in the summary, the compression ratio between the summary and article, and measures of the level of extraction. Coverage is the percentage of words that are part of an extractive fragment and density is the average length of the extractive fragment each summary word belongs to. MoverScore (Zhao et al., 2019) measures semantic distance with Word Mover’s Distance (Kusner et al., 2015) on pooled BERT n-gram embeddings. SUPERT (Gao et al., 2020) measures the semantic similarity of summaries with pseudo-reference summaries created by extracting salient sentences from the source documents. BLANC (Vasilyev et al., 2020) measures the performance gains of a pre-trained language model on language understanding tasks on the input document when given access to a document summary.
QAEval (Deutsch et al., 2021a) reports both an F1 and exact match (em) score. We do not report the learned answer overlap metric. SummaQA (Scialom et al., 2019) reports an F1 score and model confidence. We plan to report QuestEval (Scialom et al., 2021) in a future version. Lite3Pyramid includes four variations of the metric depending on the entailment model (two vs three-class entailment model) and how the output is used (as a probability vs a 0/1 label). CTC (Deng et al., 2021) proposes metrics for Compression, transduction, and creation tasks as variations of textual alignment. Relevance is scored as the average bi-directional alignment between generated and reference summaries. SimCSE (Gao et al., 2021) apply contrastive learning to learn improved sentence representations, which can then be used to compare generated and reference summary similarity. UniEval (Zhong et al., 2022) frames text evaluation as the answer to yes or no questions, in our case whether the summary is relevant or not, and constructs pseudo-data to fine-tune language models for this setting.

heading: E.2 Metrics based on Large Language Models
text: In §6.1 we evaluate two different LLM-based automatic evaluation methods.
GPTScore (Fu et al., 2023) formulates the text evaluation as the text-filling task and takes the token probability predicted by the LLMs as the quality score. We use the following prompt for calculating the recall score of the system outputs:
Answer the question based on the following reference summary and candidate summary.
Question: Can all of the information in the reference summary be found in the candidate summary? (a). Yes. (b). No.
Reference Summary: {{Reference}}
Candidate Summary: {{Candidate}}
Answer: Yes
The LLM-predicted probability of the last token, “Yes”, is used as the recall score. We use the OpenAI’s text-davinci-003 as the LLM.
G-Eval (Liu et al., 2023) introduces a similar task as GPTscore, but has the LLM to predict a numerical score directly instead of using the LLMpredicted probability. We use the following prompt for the task:
You will receive a reference summary and a candidate summary. Your task is to compare these two summaries and assess the extent to which the candidate summary covers the information presented in the reference summary.
Please indicate your agreement with the following statement: “All of the information in the reference summary can be found in the candidate summary.”
Use the following 5-point scale when determining your response:
1. Strongly Disagree
2. Disagree
3. Neither Agree nor Disagree
4. Agree
5. Strongly Agree
Input:
Reference Summary:
{{Reference}}
Candidate Summary:
{{Candidate}}
Evaluation Form (scores ONLY):
- Agreement (1-5):
We note that we set the sampling temperature to 0 to ensure more deterministic behavior for GEval-3.5 and G-Eval-4. We also experiment with a sampling strategy with GPT-3.5 (G-Eval-3.5-S), where we sample 5 outputs with a temperate 1 and take the average score as the final prediction.

heading: E.3 Metric Correlation with ACU Scores
text: We collect in total 50 different automatic metrics (including different variations of the same metric), and evaluate their performance using our collected ACU benchmark on CNNDM, XSum and SamSum datasets with three different correlation coefficients (§6.1). Tab. 15 reports the system-level correlation with the un-normalized ACU score (Eq. 1). Tab. 16 reports the summary-level correlation with the unnormalized ACU score (Eq. 1). Tab. 17 reports the system-level correlation with the normalized ACU score (Eq. 2). Tab. 18 reports the summary-level correlation with the normalized ACU score (Eq. 2).

heading: E.4 System Pairs for Fine-grained Metric Evaluation
text: For metric elevation in §6.1, we provide the system pairs in the six different buckets grouped by their performance differences below. Bucket 1: CLIFF V.S. FROST, CTRLSUM V.S. GSUM, BART V.S. CLIFF, GOLD V.S. FROST, BART V.S. FROST, CLIFF V.S. GOLD, BRIO V.S. GSUM, GOLD V.S. PEGASUS, BRIO V.S. CTRLSUM, BART V.S. GOLD, BRIO-EXT V.S. MATCHSUM. Bucket 2: FROST V.S. PEGASUS, CLIFF V.S. PEGASUS, PEGASUS V.S. GLOB, BRIO-EXT V.S. SIMCLS, BART V.S. PEGASUS, BRIO V.S. MATCHSUM, BART V.S. SIMCLS, GOLD V.S.

heading: GLOB, CLIFF V.S. SIMCLS, MATCHSUM V.S.

heading: GSUM, SIMCLS V.S. FROST.
text: Bucket 3: MATCHSUM V.S. SIMCLS, FROST V.S. GLOB, MATCHSUM V.S. CTRLSUM, CLIFF V.S. GLOB, BRIO V.S. BRIO-EXT, GOLD V.S. SIMCLS, BART V.S. GLOB, BRIO-EXT V.S. GSUM, BRIO-EXT V.S. CTRLSUM, BART V.S. BRIO-EXT, SIMCLS V.S. PEGASUS. Bucket 4: CLIFF V.S. BRIO-EXT, BRIO-EXT V.S. FROST, BRIO V.S. SIMCLS, GOLD V.S. BRIO-EXT, BART V.S. MATCHSUM, CLIFF V.S. MATCHSUM, SIMCLS V.S. GSUM, MATCHSUM V.S. FROST, SIMCLS V.S. GLOB, SIMCLS V.S. CTRLSUM, BRIO-EXT V.S. PEGASUS. Bucket 5: GOLD V.S. MATCHSUM, MATCHSUM V.S. PEGASUS, BART V.S. BRIO, BRIOEXT V.S. GLOB, BRIO V.S. CLIFF, BRIO V.S. FROST, BART V.S. GSUM, BART V.S. CTRLSUM, BRIO V.S. GOLD, CLIFF V.S. GSUM, FROST V.S. GSUM. Bucket 6: CLIFF V.S. CTRLSUM, MATCHSUM V.S. GLOB, CTRLSUM V.S. FROST, GOLD V.S. GSUM, BRIO V.S. PEGASUS, GOLD V.S. CTRLSUM, PEGASUS V.S. GSUM, CTRLSUM V.S. PEGASUS, BRIO V.S. GLOB, GLOB V.S. GSUM, CTRLSUM V.S. GLOB.

heading: E.5 Confidence Interval
text: We select several automatic metrics and calculate the confidence intervals of their system-level correlations with the ACU scores (§6.2). The results are in Fig. 8. Similar to Deutsch et al. (2021b), we found that the confidence intervals are large. However, having a larger sample size can effectively reduce the confidence interval. Specifically, we use re-sampling to generate a series of synthetic sample
sets with several different sizes and calculate the confidence interval by averaging over the sampled sets with the same size. As shown in Fig. 9, larger sample sizes lead to more stable results.

heading: E.6 Power Analysis of Metric Comparison
text: We use Alg.1 to conduct a power analysis of metric comparison based on their Kendall’s correlations with ACU scores (§6.2). We choose 20 metrics for comparison, resulting in 190 metric pairs in total, which are (1) BARTScore-r-parabank, (2) BERTScore-r-deberta, (3) BERTScore-r-roberta, (4) BLANC, (5) CHRF, (6) CTC, (7) Meteor, (8) Lite2Pyramid-p2c, (9) QAEval-em, (10) QAEvalf1, (11) ROUGE1, (12) ROUGE1r, (13) ROUGE2, (14) ROUGE2r, (15) ROUGEL, (16) ROUGELr, (17) SimCSE, (18) SummaQA, (19) SummaQAprob, (20) SUPERT. We note that we use the permutation test instead of the paired bootstrapping test to calculate the statistical significance for metric comparison, since Deutsch et al. (2021b) found that the permutation test works better for detecting significant results in metric comparison.

heading: E.7 Metric Correlation with Different Human Evaluation Protocols
text: We present the correlations between automatic metrics and different human evaluation protocols in Tab. 19 as discussed in §6.2.

heading: F Human Evaluation Practices in Recent Text Summarization Research
text: We provide a brief survey for the human evaluation practices of 55 selected papers on text summariza-
tion published at NAACL15, ACL16, and EMNLP17 from 2022. We follow the design of a similar study in Gehrmann et al. (2022) as described below. The results are shown in Tab. 14. Performed Human Evaluation: Report “yes”, if a human evaluation of any kind is done. We report that 71% of analyzed papers did human evaluation. Significance Test: Report “yes”, if a significance test is done on the human annotation results. Of the 39 papers that conducted human evaluation, a total of 27 papers reported the result of a significance test (68%), which is much higher compared to the 25% reported in the previous survey (Gehrmann et al., 2022). Power Analysis: Report “yes”, if a power analysis of any kind is mentioned. Of the 39 papers that conducted human evaluation, none of the papers did power analysis, the same as the result provided in the previous survey of Gehrmann et al. (2022). Inter-annotator Agreement: Report “yes”, if any kind of agreement test is conducted to evaluate the quality of human annotation themselves. Overall, we report a total of 12 papers (28%) that did agreement tests and documented specific agreement values. 9 out of the 12 papers recorded the specific agreement test, with Krippendorff’s alpha as the most commonly used measurement. Participants (crowd-worker, expert, etc.) Report “yes”, if at least the number of human evaluators, document sample size, annotators per document, or their demographics is mentioned. We show the sample size of the conducted human evaluation
15https://aclanthology.org/events/naacl-2022/ 16https://aclanthology.org/events/acl-2022/ 17https://preview.aclanthology.org/
emnlp-22-ingestion/volumes/2022.emnlp-main/
study in Fig. 10, and note that around 93% of them are less or equal to 200. Released Human Evaluation Data: Report “yes”, if the authors release the human evaluation data.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3-6.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We did not discuss this in the paper but our use case is research-based and consistent with the underlying licenses.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use standard benchmark datasets that have been widely used so we expect the risk of their containing offensive or personal information is relatively low.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 and Appendix A.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 6.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. We didn’t have experimental results with new models.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. We didn’t have experimental results with new models.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. We didn’t have experimental results with new models.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.2.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3, 5.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? In the appendix and supplementary materials.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3 and Appendix A.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? The data we collected contains no personal information or free-form text, therefore we consider the risk of releasing our data relatively low. The protocol for data collection follows standard MTurk procedures.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Our paper has been reviewed internally for ethics concerns.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix A.

heading: Future Work
text: , 2021) in a future version. Lite3Pyramid includes four variations of the metric depending on the entailment model (two vs three-class entailment model) and how the output is used (as a probability vs a 0/1 label). CTC (Deng et al. , 2021) proposes metrics for Compression, transduction, and creation tasks as variations of textual alignment. Relevance is scored as the average bi-directional alignment between generated and reference summaries. SimCSE (Gao et al. , 2021) apply contrastive learning to learn improved sentence representations, which can then be used to compare generated and reference summary similarity. UniEval (Zhong et al. , 2022) frames text evaluation as the answer to yes or no questions, in our case whether the summary is relevant or not, and constructs pseudo-data to fine-tune language models for this setting