Venue: EMNLP 2023

# Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation

Yixin Liu \({}^{1}\) Alexander R. Fabbri \({}^{2}\) Yilun Zhao\({}^{1}\) Pengfei Liu\({}^{3}\) Shafiq Joty\({}^{2}\) Chien-Sheng Wu\({}^{2}\) Caiming Xiong\({}^{2}\) Dragomir Radev\({}^{1}\)

\({}^{1}\)Yale University, \({}^{2}\)Salesforce AI Research, \({}^{3}\)Shanghai Jiao Tong University

{yixin.liu, dragomir.radev}@ylea.edu

###### Abstract

Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics. In this work, we develop strong-performing automatic metrics for reference-based summarization evaluation, based on a two-stage evaluation pipeline that first extracts basic information units from one text sequence and then checks the extracted units in another sequence. The metrics we developed include two-stage metrics that can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. We make the developed tools publicly available at [https://github.com/Yale-LLLY/AutoACU](https://github.com/Yale-LLLY/AutoACU).

## 1 Introduction

Automatic evaluation is an integral part of scaling natural language generation (NLG) system development and evaluation. While neural models have seen great success in NLG systems, their adoption in automatic metric development has been much slower, and classic metrics such as ROUGE Lin (2004) and BLEU Papineni et al. (2002) are still used more often than neural ones Sellam et al. (2020); Zhang* et al. (2020); Yuan et al. (2021). Compared to neural systems, we argue that there are unique requirements for neural metrics to be adopted: (1) _interpretability_ - the metric scores should be interpretable and provide intuitive insights into system performance and system output quality. (2) _evaluation efficiency_ - ideally, the automatic metrics should only introduce a small computation overhead, since it should be possible to use them on the fly for system development and fine-tuning. In this work, we aim to design neural metrics that are more aligned with these requirements, which we believe will facilitate adoption.

Our method focuses on a two-step decomposition of text sequence comparison following related work Bhandari et al. (2020); Zhang and Bansal (2021); Liu et al. (2023) - first dissecting the information in one text sequence into multiple simple facts and then checking the presence of these facts in another text sequence. Specifically, our automatic evaluation pipeline mirrors the human evaluation protocol of Liu et al. (2023), which uses atomic content units, or ACUs, as the simple facts for comparing text sequences. We believe such a **two-stage automatic evaluation** can be more _interpretable_ and _transparent_. Specifically, at the ACU level, the evaluation result indicates the presence or absence of an extracted information unit; at the summary level, the aggregated ACU score represents the percentage of information overlap from one text sequence to another. In contrast, it can be difficult to interpret the results of certain neural metrics such as BERTScore Zhang* et al. (2020) or BARTScore Yuan et al. (2021). For example, BARTScore assigns the normalized log-likelihood as the text similarity score, which is non-positive and non-linear, making it difficult to understand the system output quality based on its metric score.

Despite their advantages, such two-stage metrics Deutsch et al. (2021); Zhang and Bansal (2021); Fabbri et al. (2022) can be much slower to run, even slower than the evaluated systems. Therefore, apart from the two-stage evaluation method, we also propose a more _efficient_ one-stage metric that directly predicts the aggregated summary-level scores by training on the recently proposed RoSE Liu et al. (2023) benchmark. The one-stage metric retains the summary-level interpretability as in the two-stage evaluation, striking a balance between efficiency and interpretability. We also explore using the two-stage evaluation as pre-training for the one-stage metric, which further improves performance.

Our contributions can be summarised as: (1) A fine-grained two-stage automatic metric for reference-based summarization evaluation, which provides high interpretability; (2) An efficient one

stage automatic metric, which offers a balance between interpretability and evaluation efficiency; (3) Both types of metrics we developed achieve state-of-the-art performance on summarization evaluation, and we make them publicly available and release them as an easy-to-use Python package.1

Footnote 1: We provide the metric training scripts and a Python package for trained metrics at [https://github.com/Yale-LILY/AutoACU](https://github.com/Yale-LILY/AutoACU).

## 2 Preliminaries

Information Similarity for Summarization EvaluationThe most common and important evaluation method of summarization systems is assessing the _similarity_ between system-generated summaries and reference summaries. Since there are no widely accepted definitions of such similarity among the related work Nenkova and Passonneau (2004); Lin (2004); Bhandari et al. (2020); Fabbri et al. (2021); Zhang and Bansal (2021), we refer it as the _information similarity_ in this work - two completely similar text sequences should convey exactly the same information to the users, following the suggestion of Deutsch and Roth (2021).

Automatic Metrics of Information SimilarityTraditional automatic metrics of information similarity compare the lexical overlap of two text sequences, such as ROUGE Lin (2004), BLEU Papineni et al. (2002). In contrast, a family of neural automatic metrics (e.g., BERTScore Zhang* et al. (2020), MoverScore Zhao et al. (2019), BARTScore Yuan et al. (2021)) focus more on semantic similarity by leveraging the pre-trained language models. Neural metrics can also be pretrained on pseudo training signals Sellam et al. (2020); Zhong et al. (2022) or related corpus Yuan et al. (2021); Gao et al. (2021), or be supervisedly fine-tuned Rei et al. (2020). Apart from the one-stage metrics, related work on two-stage metrics proposes methods of decomposing the evaluation process into finer-grained sub-tasks, such as the QA-based QAEval Deutsch et al. (2021) metric, and the Lite\({}^{3}\)Pyramid Zhang and Bansal (2021) metric, which automates the evaluation process of LitePyramid Shapira et al. (2019) protocol.

Evaluation of Automatic MetricsTo evaluate information similarity metrics for text summarization, a few human evaluation benchmarks Bhandari et al. (2020); Fabbri et al. (2021); Zhang and Bansal (2021); Liu et al. (2023) have been collected, which contain system-generated summaries and their human evaluation scores. Automatic metric performance is measured by the correlation between the automatic metric scores and human evaluation scores of the system-generated summaries. For text summarization metrics, such correlations can be calculated at the _system_ level and the _summary_ level. More specifically, given \(n\) input articles and \(m\) summarization systems, the human evaluation and an automatic metric result in two \(n\)-row, \(m\)-column score matrices \(H\), \(M\) respectively. The _summary_-level correlation is an average of sample-wise correlations:

\[r_{\mathrm{sum}}(H,M)=\frac{\sum_{i}\mathcal{C}(H_{i},M_{i})}{ n}, \tag{1}\]

where \(H_{i}\), \(M_{i}\) are the evaluation results on the \(i\)-th data sample and \(\mathcal{C}\) is a function calculating a correlation coefficient (e.g., the Pearson's correlation coefficient). In contrast, the _system_-level correlation is calculated on the aggregated system scores:

\[r_{\mathrm{sys}}(H,M)=\mathcal{C}(\bar{H},\bar{M}), \tag{2}\]

where \(\bar{H}\) and \(\bar{M}\) contain \(m\) entries which are the average system scores across \(n\) data samples, e.g., \(\bar{H}_{0}=\sum_{i}H_{i,0}/n\).

## 3 Methods

We first describe our approach of a two-stage decomposition of automatic information similarity evaluation - (1) extracting fine-grained content units from one text sequence; (2) checking the existence of the extracted units in another text sequence. Then we introduce methods of training a one-stage automatic metric for information similarity and using the two-stage decomposition for pre-training.

### Two-Stage Evaluation

In Fig. 1, we provide an example of the two-stage automatic summarization evaluation process of calculating the _recall_ score of a system-generated summary using the reference summary.

**Content Unit Extraction** A (long) text sequence can contain more than one fact, or simple information unit. Therefore, we follow Liu et al. (2023) by using _Atomic Content Units_ (ACUs) to refer to the basic information units. We formulate automatic ACU extraction as a sequence-to-sequence (Seq2Seq) problem Sutskever et al. (2014): \(A\gets g(S)\), where \(S\) is the input text sequence, \(A\) is a concatenation of a set of ACUs \(\mathcal{A}\) generated as a sequence, and \(g\) is a Seq2Seq model.

 



**Content Unit Checking** Having extracted a set of ACUs \(\tilde{\mathcal{A}}\) from one text sequence \(S_{1}\), we use a Natural Langauge Inference (NLI) Gururangan et al. (2018) model \(f\) to check if the information in an extracted ACU \(a\) is conveyed by another text sequence \(S_{2}\):

\[l_{a}=f(S_{2},a)=\begin{cases}1&\text{if $a$ is entailed by $S_{2}$},\\ 0&\text{otherwise},\end{cases} \tag{3}\]

where \(l_{a}\) is the label of \(a\) assigned by the model \(f\). In addition to this standard NLI setting of viewing \(S_{2}\) as the premise and \(a\) as the hypothesis, we also explore adding \(S_{1}\) as part of the model input serving as context:

\[l_{a}=f^{*}(S_{2},a|S_{1})=\begin{cases}1&\text{if $a$ is entailed by $S_{2}$},\\ 0&\text{otherwise},\end{cases} \tag{4}\]

We use BERT Devlin et al. (2019) as the NLI model architecture and follow its input format for the standard setting (Eq. 3), which is a concatenation of \(S_{2}\) and \(a\). For the extended setting (Eq. 4), we define the input as a concatenation of \(S_{1}\), \(S_{2}\), \(a\).

Based on these two stages, we can define a _recall_ information similarity score of \(S_{2}\) w.r.t. \(S_{1}\):

\[\mathcal{R}(S_{2}|S_{1})=\frac{\sum_{a\in A}l_{a}}{|\tilde{\mathcal{A}}|}. \tag{5}\]

### One-Stage Metric and Its Pre-Training

To improve evaluation efficiency, we propose using our two-stage approach to generate pre-training data for a more lightweight, one-stage metric. Specifically, we use BERT as the backbone for a scoring model \(h\) to approximate the two-stage recall score (Eq. 5):

\[\mathcal{R}_{h}(S_{2}|S_{1})\leftarrow\mathcal{R}(S_{2}|S_{1}). \tag{6}\]

The model is trained with the mean squared error between the target score and the predicted score. The input format is "[CLS]\(S_{2}\)[SEP]\(S_{1}\)[SEP]" following Devlin et al. (2019), and a single linear layer is introduced to map the hidden representation of "[SEP]" into the predicted numer score. We can also define an F1 score as

\[\mathcal{F}_{h}(S_{2}|S_{1})=\frac{2\mathcal{R}_{h}(S_{2}|S_{1})\mathcal{R}_{ h}(S_{1}|S_{2})}{\mathcal{R}_{h}(S_{2}|S_{1})+\mathcal{R}_{h}(S_{1}|S_{2})}. \tag{7}\]

**Pre-training Corpora** As shown by Sellam et al. (2020), the robustness of automatic metrics can be improved by pre-training with synthetic pre-training data. We further extend this approach following the finding of related work Liu and Liu (2021); Liu et al. (2022), which shows that pre-trained summarization models such as BART can generate diverse and high-quality candidate summaries and summarization models can benefit from contrastive learning with the generated candidates. In a similar spirit, we construct the pre-training corpora on the existing summarization datasets such as CNN/DailyMail Nallapati et al. (2016), and for each data example we generate multiple candidate summaries using a fine-tuned summarization model. The generated summaries are then scored by the two-stage evaluation method (Eq. 5), which are used for the pre-training of one-stage model \(h\).

## 4 Experiments

### Experimental Settings

#### Datasets

We mainly use a recently-introduced summarization evaluation benchmark, RoSE Liu et al. (2023), for automatic metric development and evaluation. It contains human evaluation _recall_ scores of the system-generated summaries based on the reference summaries w.r.t. the information similarity on three summarization datasets, CNN/DailyMail (CNNDM) Nallapati et al. (2016), XSum Narayan et al. (2018), and SamSum Gliwa et al. (2019). The human evaluation is conducted following the ACU evaluation protocol Liu et al. (2023).

Figure 1: Example of two-stage automatic summarization evaluation based on the Atomic Content Unit (ACU) protocol. In the first stage, an automatic ACU extraction model dissects the information in one text sequence into ACUs. In the second stage, an automatic ACU checking (matching) model checks the presence of the extracted ACUs against another text sequence. We use the example provided by Liu et al. (2023).



2023), and the dataset also provides human-written ACUs on the reference summaries and the associated results (ACU labels) of checking the ACUs against system-generated summaries. The statistics are in Tab. 1.

Baseline MetricsWe compare our methods with related automatic metrics for text sequence comparison. Since the RoSE benchmark provides _recall_ information similarity scores, only metrics with recall scores are compared.

**ROUGE**Lin (2004) compares two text sequences by the \(n\)-grams overlap between them. We report the performance of its ROUGE-1/2/L variants.

**BERTScore**Zhang* et al. (2020) measures text sequence similarity using hidden representations computed by pre-trained language models such as BERT Devlin et al. (2019). We report its variants with RoBERTa Liu et al. (2019) (BERTScore\({}_{R}\)) and DeBERTa He et al. (2021) (BERTScore\({}_{D}\)).

**BARTScore**Yuan et al. (2021) interprets the similarity of text sequence \(x\) to \(y\) as the probability of \(x\) given \(y\) predicted by a pre-trained language model such as BART Lewis et al. (2020). We report its variants pre-trained on CNNDM (BARTSCore\({}_{C}\)) and ParaBank2 Hu et al. (2019) (BARTScore\({}_{P}\)).

**QAEval**Deutsch et al. (2021) measures information similarity by question answering accuracy. We report both its exact match score (QAEval\({}_{EM}\)) and F1 score (QAEval\({}_{F1}\)).

**Lite\({}^{3}\)Pyramid**Zhang and Bansal (2021) introduces a similar approach as our two-stage evaluation but uses a semantic role labeling He et al. (2017) model to extract content units. We report its variants that use the predicted label (Lite\({}^{3}\)Pyramid\({}_{L}\)) and probability (Lite\({}^{3}\)Pyramid\({}_{P}\)) of the two-class NLI model.

Implementation DetailsFor the _two-stage_ evaluation, the ACU extraction model is based on a T0 Sanh et al. (2022) model2 fine-tuned on the human-written ACUs provided in RoSE. As for the NLI model for ACU checking (Eq. 3&4), we use a pre-trained DeBERTa He et al. (2021) NLI model3 as the start point and further fine-tune it on the RoSE dataset with the available gold standard ACU labels. We name the two-stage method A\({}^{2}\)CU (**A**uto**ACU**), and it has three variants in total: A\({}^{2}\)CU\({}_{P}\) is with the pre-trained NLI model, A\({}^{2}\)CU\({}_{F}\) is with the fine-tuned NLI model (Eq. 3), A\({}^{2}\)CU\({}_{FC}\) is with the fine-tuned NLI model taking the source text as part of the input.

Footnote 2: huggingface.co/bigscience/T0_3Bdeberta-xlarge-mnli.

For the pre-training of _one-stage_ metric, we use system-generated summaries generated by a pre-trained BART model on the CNNDM dataset. For each data example, 12 summaries are scored by the two-stage evaluation method and used to pre-train the one-stage metric. After the pre-training, the one-stage metric can be further fine-tuned on the gold-standard scores. We name the one-stage metric as A\({}^{3}\)CU (**A**ccelerated**A**uto**ACU**), and it has three variants: A\({}^{3}\)CU\({}_{F}\) is directly fine-tuned on RoSE, A\({}^{3}\)CU\({}_{P}\) is pre-trained with A\({}^{2}\)CU only, A\({}^{3}\)CU\({}_{PF}\) is first pre-trained then fine-tuned. We note that the training on the RoSE dataset is performed on the validation split of CNNDM. Regard

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline
**Dataset** & **Split** & **\#Doc.** & **\#Sys.** & **\#ACU** & **\#Summ.** \\ \hline CNNDM & Test & 500 & 12 & 5.6k & 6k \\ CNNDM & Valid & 1,000 & 8 & 11.6k & 8k \\ XSum & Test & 500 & 8 & 2.3k & 4k \\ SamSum & Test & 500 & 8 & 2.3k & 4k \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the RoSE dataset as reported in Liu et al. (2023). **#ACU** and **#Summ.** are the number of annotations at ACU and summary levels.

\begin{table}
\begin{tabular}{l l l l l l} \hline  & \multicolumn{2}{c}{**CNNM**} & \multicolumn{2}{c}{**XSum**} & \multicolumn{2}{c}{**SamSum**} \\  & **Sys. Sum.** & **Sys.** & **Sum.** & **Sys.** & **Sum.** \\ \cline{2-6} ROUGE1 & .788 & .468 & .714 & .293 & .929 & .439 \\ ROUGE2 & .758 & .453 & .643 & .266 & **1.00** & .395 \\ BOUGEL & .879 & .454 & .643 & .258 & .929 & .415 \\ BERTScore\({}_{R}\) & .515 & .448 & .571 & .277 & .857 & .417 \\ BERTScore\({}_{D}\) & .424 & .424 & .571 & .262 & .857 & .409 \\ BARTScore\({}_{C}\) & .727 & .435 & .643 & .260 & .929 & .438 \\ BARTScore\({}_{P}\) & .727 & .453 & .714 & .282 & .929 & .430 \\ QAEval\({}_{EM}\) & .515 & .296 & .357 & .149 & .857 & .352 \\ QAEval\({}_{I}\) & .849 & .358 & .429 & .198 & .929 & .384 \\ Lite\({}^{3}\)Pyramid\({}_{L}\) & .849 & .466 & .643 & .207 & **1.00** & .494 \\ Lite\({}^{3}\)Pyramid\({}_{P}\) & .849 & .452 & .714 & .245 & **1.00** & .467 \\ \hline A\({}^{2}\)CU\({}_{P}\) & .879 & .521\({}^{\dagger}\) & **786** & .336\({}^{\dagger}\) & **1.00** & .532\({}^{\dagger}\) \\ A\({}^{2}\)CU\({}_{F}\) & .818 & .557\({}^{\dagger}\) & **786** & **.348\({}^{\dagger}\)** & **1.00** & **.556\({}^{\dagger}\)** \\ A\({}^{2}\)CU\({}_{FC}\) & .879 & .555\({}^{\dagger}\) & .214 & .202 & .571 & .275 \\ A\({}^{3}\)CU\({}_{F}\) & **.909** & .493\({}^{\dagger}\) & **.786** & .299 & .929 & .445 \\ A\({}^{3}\)CU\({}_{P}\) & .879 & .558\({}^{\dagger}\) & **.786** & .307 & .929 & .472 \\ A\({}^{3}\)CU\({}_{PF}\) & .879 & **.564\({}^{\dagger}\)** & **.786** & .319\({}^{\dagger}\) & .929 & .474 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The Kendall’s correlation between the automatic metric scores and human evaluation scores on the RoSE dataset. The correlation is calculated at both the system level and the summary level. We use the _recall_ score of the automatic metrics. A\({}^{2}\)CU is our two-stage evaluation method, A\({}^{3}\)CU is our one-stage metric. †: significantly (\(p<0.05\)) better than the best baseline. The baseline results are from Liu et al. (2023).

 

ing _metric efficiency_, A\({}^{3}\)CU takes only 6% the inference time of A\({}^{2}\)CU, demonstrating its superior efficiency. More details are in Appendix A.

### Results

We report the results in Tab. 2. Kendall's correlation coefficients are used to evaluate the metric performance at both the system and summary levels, which shows the following: (1) Both our two-stage and one-stage metrics can outperform the baseline methods across three datasets. (2) The improvement of our metrics is more significant at the summary level than at the system level. (3) Compared with the one-stage metrics, our two-stage metrics generalize better on XSum and SamSum datasets. (4) The pre-training is effective for the one-stage metric to achieve strong performance since A\({}^{3}\)CU\({}_{F}\) can outperform A\({}^{3}\)CU\({}_{F}\) at the summary level without fine-tuning on the RoSE dataset.

### Analysis

Performance Analysis of A\({}^{2}\)CuWe compare A\({}^{2}\)CU performance with SemiA\({}^{2}\)CU, a _semi_automatic metric that follows the same scoring mechanism but uses the reference ACUs as the NLI model input (Eq. 3). Results in Tab. 3 show that using the reference ACUs yields better performance, suggesting that better generated ACU quality may lead to further improvement. To analyze this quality, we calculate its similarity with reference ACUs using ROUGE scores between them in a greedy-matching manner. Specifically, given a set of generated ACUs \(\tilde{\mathcal{A}}\) and reference ACUs \(\mathcal{A}\), we calculate the example-level _recall_ score as:

\[r(\tilde{\mathcal{A}}|\mathcal{A})=\frac{\sum_{\tilde{a}\in\mathcal{A}}\max_{ a\in\mathcal{A}}ROUGE(a,\tilde{a})}{|\tilde{\mathcal{A}}|}. \tag{8}\]

Similarly, we can define the precision score and F1 score. The results on three dataset splits are reported in Tab. 4, showing a high degree of similarity between the reference and generated ACUs.

Comparative Study of **A\({}^{3}\)CU** We investigate the impact of two design decisions for the one-stage metric, the model size, and the supervision signal used in pre-training. To this end, we show metric performance with two model sizes and the metric performance when it is pre-trained to predict the average score of ROUGE-1/2/L. The results in Tab. 5 suggest that larger model sizes and more suitable pre-training can improve metric performance.

Evaluation on Other Related BenchmarksWe provide additional evaluation results of our F1-based automatic metrics (e.g., Eq. 7) in Appendix B. Apart from RoSE, we evaluate the metrics on two related benchmarks, the STS (Semantic Textual Similarity) benchmark Cer et al. (2017) and WMT19 Metrics Shared Task daRR benchmark Ma et al. (2019), which provides a more comprehensive investigation of the generalization ability of our proposed metrics.

## 5 Conclusions

We develop high-performing reference-based summarization automatic metrics, including two-stage metrics providing fine-grained interpretability and one-stage metrics for a balance between efficiency and interpretability. Furthermore, we show that the two-stage metric can be used to effectively pre-train the one-stage metric, helping to mitigate the data scarcity in developing automatic metrics.

## Acknowledgements

We are grateful to the anonymous reviewers for their constructive comments. We thank Arman Cohan for insightful discussions and suggestions.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & **Precision** & **Recall** & **F1** \\ \hline CNNDM & 83.63 & 79.84 & 81.69 \\ XSum & 80.11 & 79.14 & 79.62 \\ SamSUM & 87.01 & 87.14 & 87.08 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Quality analysis of A\({}^{2}\)CU-generated ACUs compared with reference ACUs using ROUGE-1 scores.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**CNNDM**} & \multicolumn{2}{c}{**XSum**} & \multicolumn{2}{c}{**SamSum**} \\  & **Sys. Sum.** & **Sys.** & **Summ.** & **Sys.** & **Summ.** \\ SemiA\({}^{2}\)CU & .849 & .647 & .786 & .450 & 1.00 & .621 \\ A\({}^{2}\)CU\({}_{F}\) & .818 & .557 & .786 & .348 & 1.00 & .556 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison between A\({}^{2}\)CU\({}_{F}\) and the semi-automatic metric (SemiA\({}^{2}\)CU) using reference ACUs. Kendall’s correlation coefficients are reported.

 



## Limitations

Our metrics are developed and evaluated on English corpora only, and it remains unclear whether the metrics can achieve consistent performance in other languages. The two-stage metrics we proposed are based on relatively large language models and it can be time-consuming and computationally expensive to use them, especially on large corpora.

## References

* [1]M. Bhandari, P. Narayan Gour, A. Ashfaq, P. Liu, and G. Neubig (2020) Reevaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 9347-9359. External Links: Link Cited by: SS6.
* D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia (2017) SemEval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp. 1-14. External Links: ISBN 978-3-319-3819-3, Document, Link Cited by: SS1, SS2.
* D. Deutsch and D. Roth (2021) Understanding the extent to which content quality metrics measure the information quality of summaries. In Proceedings of the 25th Conference on Computational Natural Language Learning, Online, pp. 300-309. External Links: Link, Document Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link, Document Cited by: SS1, SS2.
* A. Fabbri, C. Wu, W. Liu, and C. Xiong (2022) QAFactEval: improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 2587-2601. External Links: Link, Document Cited by: SS1, SS2.
* A. R. Fabbri, W. Kryscinski, B. McCann, C. Xiong, R. Socher, and D. Radev (2021) SummEval: re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics9, pp. 391-409. External Links: Link, Document Cited by: SS1, SS2.
* T. Gao, X. Yao, and D. Chen (2021) SimCSE: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 6894-6910. External Links: Link, Document Cited by: SS1, SS2.
* B. Gliwa, I. Mochol, M. Biesek, and A. Pawer (2019) SAMSum corpus: a human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, Hong Kong, China, pp. 70-79. External Links: Link, Document Cited by: SS1, SS2.
* S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith (2018) Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, pp. 107-112. External Links: Link, Document Cited by: SS1, SS2.
* L. He, K. Lee, M. Lewis, and L. Zettlemoyer (2017) Deep semantic role labeling: what works and what's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 473-483. External Links: Link, Document Cited by: SS1, SS2.
* P. He, X. Liu, J. Gao, and W. Chen (2021) Deberta: decoding-enhancedbert with disentangled attention. In International Conference on Learning Representations, Cited by: SS1, SS2.
* J. H. Hu, A. Singh, N. Holzenberger, M. Post, and B. Van Durme (2019) Large-scale, diverse, paraphrastic bitets via sampling and clustering. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Hong Kong, China, pp. 44-54. External Links: Link, Document Cited by: SS1, SS2.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazzvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020) BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link, Document Cited by: SS1, SS2.
* Chin-Yew Lin (2004) ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link, Document Cited by: SS1, SS2.



Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv_, abs/1907.11692.
* Liu et al. (2023) Yixin Liu, Alexander R. Fabbri, Pengfieiu Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir R. Radev. 2023. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics_.
* Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021. SimCLS: A simple framework for contrastive learning of abstractive summarization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 1065-1072, Online. Association for Computational Linguistics.
* Liu et al. (2022) Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2890-2903, Dublin, Ireland. Association for Computational Linguistics.
* Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 62-90, Florence, Italy. Association for Computational Linguistics.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
* Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
* Nenkova and Passonneau (2004) Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijang Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Reimers et al. (2016) Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic textual similarity. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 87-96, Osaka, Japan. The COLING 2016 Organizing Committee.
* Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Ajafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Elizaz Szczechla, Taewooon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trisahala Neeraj, Jos Rozen, Abheesh Sharma, Andrea Santilli, Tibault Fevry, Jason Alan Fries, Ryan Teeham, Teven Le Scao, Stellis Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In _International Conference on Learning Representations_.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Shapira et al. (2019) Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramachanth Pasunuru, Mohit Bansal, Yael Amsterdam, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 682-687, Minneapolis, Minnesota. Association for Computational Linguistics.
* Volume 2_, NIPS'14, page 3104-3112, Cambridge, MA, USA. MIT Press.
 

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In _Advances in Neural Information Processing Systems_.
* Zhang and Bansal (2021) Shiyue Zhang and Mohit Bansal. 2021. Finding a balanced degree of automation for summary evaluation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6617-6632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth move distance. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 563-578, Hong Kong, China. Association for Computational Linguistics.
* Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2023-2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

## Appendix A Experiment Details

For the two-stage evaluation, the ACU extraction model is a T0 model containing 3 billion parameters, and the ACU checking model is a pre-trained DeBERTa model with 750 million parameters. Our one-stage metric is based on BERT, and its two variants have 340 and 110 million parameters respectively. We conduct experiments on NVIDIA RTX A6000 GPUs, and the experiments take around 30 GPU hours to finish.

Inference EfficiencyIn Tab. 6 we compare the inference time of A\({}^{3}\)CU and A\({}^{2}\)CU. A\({}^{3}\)CU only takes around 6% of the inference time of A\({}^{2}\)CU.

## Appendix B Evaluation on Related Information Similarity Benchmark

Apart from the RoSE benchmark, we evaluate metric performances on two related benchmarks for assessing the _information similarity_ between two text sequences: the STS (Semantic Textual Similarity) benchmark Cer et al. (2017) and the human evaluation results from the WMT19 Metrics Shared Task Ma et al. (2019).

STS BenchmarkSTS benchmark4 contains English sentence pairs and the associated human-annotated semantic similarity scores Cer et al. (2017), which has a similar evaluation target as the information similarity for text summarization evaluation. With this benchmark, the semantic similarity metrics are evaluated by the correlation between the predicted similarity scores and the reference scores. Following the previous work Gao et al. (2021); Reimers et al. (2016), we report Spearman's correlation coefficients on the test split of the benchmark containing around 1.5k data examples. Since the metrics we evaluated previously are not designed specifically for the STS task, we compare another metric, SimCSE Gao et al. (2021), which achieves strong performance on this task.

Footnote 4: The data is provided at [http://ika2.si.ehu.es/stswiki/index.php/STSbenchmark](http://ika2.si.ehu.es/stswiki/index.php/STSbenchmark).

Wmt19 daRR BenchmarkThis benchmark contains human-annotated scores of system-generated translations Ma et al. (2019). We use only the to-English part of the benchmark, which results from _reference-based_ direct assessment (DA) of translation quality. The human-annotated DA scores are then transformed into relative rankings between two translations, i.e., the daRR scores.5 We follow the evaluation setting of Ma et al. (2019) by using Kendall's Tau-like correlation to evaluate the **segment-level** performance of automatic metrics. The benchmark contains the translations from seven languages to English and results in around 21k translated sentence pairs.

Footnote 5: We use the pre-processed data provided by Rei et al. (2020) at [https://github.com/Unbabel/COMET/](https://github.com/Unbabel/COMET/).

Apart from these two benchmarks, we also report the metric performance under _normalized_ ACU scores in RoSE, which are decorrelated with summary lengths to evaluate F1-based metrics Liu et al. (2023). We note that unlike in SS4.2, all the metrics compared here are F1-based. In particu

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & **A\({}^{2}\)CU-Stage1** & **A\({}^{2}\)CU-Stage2** & **A\({}^{3}\)CU** \\ \hline CNNDM & 21.3 & 14.7 & 2.34 \\ XSum & 9.4 & 1.9 & 0.56 \\ SamSUM & 9.3 & 1.9 & 0.69 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Inference time (in minutes) comparison of A\({}^{3}\)CU and A\({}^{2}\)CU. For A\({}^{2}\)CU we show the time of ACU generation (Stage 1) and matching (Stage 2).

 

lar, for A\({}^{3}\)CU we follow Eq. 7 to calculate the F1 scores. The two-stage metrics including A\({}^{2}\)CU are not reported because of their lower efficiency.

The results in Tab. 7 show that:

(1) While the compared metrics have a similar evaluation target, i.e., _information similarity_ between text sequences, their performance varies across different benchmarks and there is no single metric that can consistently outperform others. A similar finding in Ma et al. (2019) shows that Pearson's correlation between human-annotated cross-lingual STS scores and machine translation quality estimation scores is only 0.41.

(2) On the STS benchmark, A\({}^{3}\)CU outperforms other metrics except for SimCSE which is specifically designed for the STS task, which we believe results from A\({}^{3}\)CU's better interpretability thanks to its underlying evaluation process. Specifically, A\({}^{3}\)CU's computed scores directly indicate the information overlap between text sequences, which is close to the definition of the STS task.

(3) On the WMT benchmark, A\({}^{3}\)CU fails to outperform BERTScore and BARTScore. We hypothesize this is because A\({}^{3}\)CU is relatively insensitive to the minor differences in candidate translations when implicitly comparing them based on the same reference translation as we found that the system-generated translations in the WMT benchmark have higher similarity than the system-generated summaries in the RoSE benchmark. Specifically, while both RoSE and WMT19 dARR benchmarks contain reference-based human annotations of candidate output quality, we note that they have different data distributions. In detail, the system-generated translations of the same source sentence in the WMT benchmark have higher similarities than the system-generated summaries of the same source articles. We visualize this discrepancy in Fig. 2, which shows the similarity (as evaluated by ROUGE-1) between different candidate outputs (of the same example) on WMT19 dARR benchmark and the XSum test split6 of RoSE benchmark. As illustrated by the figure, candidates in WMT19 benchmark are more similar, which can lead to the performance difference when the same automatic metric is evaluated on these two benchmarks.

Footnote 6: We chose XSum split because it contains one-sentence news article summaries, which is similar to the data format of the WMT19 benchmark.

Figure 2: Comparison of candidate outputs similarity in the WMT19 dARR benchmark and the XSum test split of RoSE benchmark. The similarity of candidate output pairs of the same example is evaluated by ROUGE1.