Venue: NAACL 2022

SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling

Forrest Sheng Bao and Ge Luo and Hebi Li

Iowa State University, Ames, IA, USA forrest.bao@gmail.com, {gluuo,hebi}@iastate.edu Minghui Qiu Alibaba Group Hangzhou, Chinaminghui.qmh@alibaba-inc.com Yinfei Yang1600 Amphitheater Parkway Mountain View, CA, USA yangvin7@gmail.com Youbiao He Iowa State University Ames, IA, USA yh54@iastate.edu Cen Chen East China Normal University Shanghai, China cecilia.cenchen@gmail.com

###### Abstract

Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.

## 1 Introduction

In natural language processing, the problem of summarization studies generating a summary from a source document which is longer than the summary. De facto metrics to judge a generated summary include ROUGE Lin (2004), BLEU Papineni et al. (2002), and METEOR Banerjee and Lavie (2005). Previous work Ng and Abrecht (2015); Liu and Liu (2008); Liu et al. (2016); Shang et al. (2018) agrees on two major drawbacks of them: 1) they favor lexical similarity, falling short on semantic similarity or linguistic quality, and 2) they require a reference summary which is often expensive to obtain Zopf (2018).

Initially, the first drawback is partially alleviated by replacing lexicons with their word embeddings Ng and Abrecht (2015); Ellouze et al. (2017); Ruseti et al. (2018); Xia et al. (2019). After the birth of transformers Vaswani et al. (2017), this effort has expanded to sentence or document level, including reference-based Zhang* et al. (2020); Zhao et al. (2019), and reference-free ones Vasilyev et al. (2020); Scialom et al. (2019); Gao et al. (2020). The main difference between the two groups is whether a reference summary is needed when evaluating a machine-generated summary.

The two groups have complementary pros and cons. Reference-based metrics have a better performance, but they are impractical when summarization is used industrially, such as in customer support Liu et al. (2019), team conversation Zhang and Cranshaw (2018), and bug reporting Rastkar et al. (2014), where it is too costly to manually craft an equally massive amount of reference summaries. In contrast, without human written reference summaries, reference-free approaches generally perform poorer. Modern transformer-based reference-free approaches often rely on non-summarization tasks, such as QA Vasilyev et al. (2020); Scialom et al. (2019). Such fact-focused approach makes them excel on content/fact aspects (still worse than reference-based ones) but not on linguistic ones. The non-summarization tasks also introduce noises.

Therefore, in this paper, as a proof of concept, we explore a hybrid or middle approach to combine the best of both worlds. Using documentary pairs in existing summarization datasets, our weakly supervised approach mutates+reference summaries and pair them with documents to form training data and then use the trained model to evaluate unseen summaries in the presence of 

documents without corresponding reference summaries. In this way, we make use of human written summaries, which are very precious, in training, but we do not need them in summary evaluation. We call our approach **SueNes** , which stands for "**Sum**mary **e**valuation by **N**egative **s**ampling."

The quality of a summary is usually evaluated on two facets: content/fact aspects and linguistic qualities. Experiments later will show that a value of our approach is that we can use the same model architecture to build models that excel on different tasks by feeding training data from the same source but mutated in different strategies. For example, deleting words is the best for linguistic qualities while deleting sentences is the best for content/fact coverage.

Our approach is empirically compared against an array of existing metrics on three human summary evaluation datasets. Despite being training-based, our approach exhibits consistent results across various training domains which are all different from the test domain. It outperforms reference-free baselines with promising improvements on content/fact aspects, and further outperforms all existing metrics in gauging linguistic qualities.

In summary, our contributions or merits are:

* a simple but effective approach to reference-free+ summary quality assessment,

Footnote †: The definition of “reference-free” is that reference summaries are not needed in the evaluation stage.
* negative sampling for preparing training data from the unlabeled,
* one task/framework for multi-aspect judging,
* extensive cross-domain experiments to validate the effectiveness and domain robustness of our approach.

We hope our study can inspire more research into hybridizing reference-free and reference-based summary evaluation. Our code is at [https://github.com/forrestbao/SueNes/](https://github.com/forrestbao/SueNes/)

## 2 The Approach

### Model Architecture

A reference-free single-document summary quality assessor can be formulated as a regression function \(f(d,s)\in[0,1]\) of an input document \(d=[t_{1},t_{2},\cdots]\), and a machine-generated summary \(s=[t^{\prime}_{1},t^{\prime}_{2},\cdots]\), where \(t_{i}\)'s and \(t^{\prime}_{i}\)'s are text tokens. As a proof of concept, we explore an extremely lean implementation of \(f\): first \(d\) and \(s\) are jointly transformed into a vector representation \(\mathbf{e}=g(d,s)\), and then it is mapped to a summary quality score via a fully-connected layer, i.e., \(f(d,s)=\sigma(\mathbb{W}\mathbf{e})\).

The function \(g\) can be implemented in the BERT Devlin et al. (2019) style with an input sequence [[CLS], \(t_{1}\), \(t_{2}\), \(\cdots\), [SEP], \(t^{\prime}_{1}\), \(t^{\prime}_{2}\), \(\cdots\), [SEP]]. The output on the [CLS] token is a joint representation of both the document \(d\) and the summary \(s\).

While the human evaluation to a summary may cover multiple aspects, such as content/fact coverage and linguistics, a model of us will only yield one number. But by using different data mutation strategies, we can get models (different \(f\)'s) adept at different aspects of a summary.

### Negative Sample Generation

It is impractical to train \(f\) with existing summarization datasets, such as CNN/DailyMail Hermann et al. (2015); Nallapati et al. (2016), because they contain only high-quality, reference-class summaries written manually and thus are all of label 1. Some summary evaluation datasets, such as RealSumm Bhandari et al. (2020), Newsroom Grusky et al. (2018), and TAC2010 NIST (2010), do contain human ratings to system-generated summaries of various qualities. But they are too small, containing no more than 100 news articles or article groups each. Therefore, training against human ratings or in a supervised approach is impractical.

To work around, we propose a weakly supervised solution as depicted in Figure 1(a). Existing summarization datasets contain many document-summary pairs. For each pair \(\langle d,s\rangle\), the reference summary \(s\) is mutated into \(K\) new summaries of different extents \(s_{1},s_{2},\cdots,s_{K}\), which are then paired with the document to form new pairs

\[\langle d,s_{1}\rangle,\langle d,s_{2}\rangle,\cdots,\langle d,s_{K}\rangle,\]

which are finally assigned targets to form the training data

\[(\langle d,s_{1}\rangle,y_{1}),(\langle d,s_{2}\rangle,y_{2}),\cdots,(\langle d ,s_{K}\rangle,y_{K}).\]

As illustrated in Figure 2, the training target \(y_{k\in[1..K]}\) is the percentage of intact content. For example, if 30% of tokens in a mutated summary are not original, then the label is 0.7. In addition, the original document-summary pair \(\langle d,s\rangle\) is also used in training with a target of 1 

Mutations can happen at the token or sentence level, where tokens or sentences are randomly selected for mutation. A selected token or sentence is mutated in one of the three methods:

1. **inserting** a token/sentence from other summaries behind it,
2. **deleting** it, or
3. **replacing** it with a token/sentence from other summaries.

We do not mix different mutation levels nor mix different mutation methods when preparing the training data. Instead, our experiments study one combination of a mutation level and a mutation method, denoted as a mutation strategy, each time.

## 3 Experiments

### Test data

The ground truth of a summary's quality is human ratings to it. A model trained (Fig. 1(a)) is tested (Fig. 1(b)) against human ratings. Three test datasets are chosen below. Due to the limited number and sizes of human evaluation datasets, they are all in the news domain. The human evaluation protocols can be found in their respective references.

**TAC2010**[15] is a multi-document (ten-document) summarization task reporting both factual and linguistic aspects. We use \(\sum_{i\in[1..10]}f(d_{i},s)\) to approximate the score of the summary \(s\) composed from ten documents \(d_{1}\) to \(d_{10}\). We only use Set A of TAC2010 because Set B is not for regular summarization.

**Newsroom**[6] also covers both factual (in INFormativeness and RELEVance) and linguistic (in COherence and FLUency) aspects. For human ratings, three human annotators rate one pair of a document and machine-generated summary. The mean of their ratings on each aspect is used in our experiments.

**RealSumm**[7] focuses on only factual coverage. It covers 14 abstractive and 11 extractive summarizers published after 2018 and conducts human evaluation on the two groups separately.

Note that we do not and cannot train a model against the labels in a test set, as mentioned in SS 2.1.

Figure 1: The weakly supervised training approach in this paper and the test of a trained model.

Figure 2: Training sample generation by mutation. Mutated text in dark blocks while intact text in gray blocks. Sizes are out of scale.

 

If a test set rates on multiple aspects, we do not train one model for each aspect. Nor do we train models for individual or a collection of test sets. We compute correlation between the predictions from our model and human ratings on each aspect of each test set.

### Training data

Three widely used summarization datasets from three different domains are chosen for training: **Billsum**Kornilova and Eidelman (2019), **Scientific-Papers/arXiv**Cohan et al. (2018), and **Big-Patent**Sharma et al. (2019). Datasets from the news domain are avoided on purpose because the test data is in the news domain. This cross-domain setting allows us to examine whether a model is prone to domain differences. For each reference summary, \(K=5\) mutated summaries are generated. The percentage of intact content is measured by the number of tokens and the number of characters for token-level and sentence-level mutations, respectively.

### Baselines and upper bounds

To fairly compare, four recent metrics: **BLANC**Vasilyev et al. (2020), **SummaQA**Scialom et al. (2019), **SUPERT**Gao et al. (2020) and **LS-Score**Wu et al. (2020) , are used as baselines because like our approach, they do not need a reference summary to judge a machine-generated summary, i.e., reference-free.

Human crafted reference summaries give reference-based metrics advantages. The results of reference-based metrics are included as soft upper bounds: **ROUGE-1, ROUGE-2 and ROUGE-L**Lin (2004), **MoverScore**Zhao et al. (2019), **BertScore**Zhang* et al. (2020), **BLEU**Papineni et al. (2002), **METEOR**Banerjee and Lavie (2005), and \({\bf S}^{3}\)Peyrard et al. (2017).

### Settings

Because the baselines use BERT, we use BERT as well for a fair comparison. Specifically, BERT-base uncased (L=12, H=768) is fine-tuned, with a learning rate of 1e-5, three epochs, and a batch size of 14. The input sequence is limited to 512 tokens using the round robn trimmer. The training loss is MSE as this problem is regression.

### Results

We use the summary-level Peyrard et al. (2017) meta-evaluation strategy to report an approach's average correlation with human ratings. Summary evaluation usually covers two types of aspects, contents/facts and linguistics. They are reported separately in Tables 1 and 2. Due to space limit, only the best mutation strategy is reported for each aspect group.

**On content/fact aspects**, the best mutation strategy is sentence deletion and our best models outperform baselines on all test datasets. Our approach makes the most improvement over baselines on RealSumm, a dataset much bigger than Newsroom and more modern than TAC2010, and the least improvement on TAC2010, the oldest dataset.

**On linguistic aspects**, the best mutation strategy is word deletion. Here, even our worst model cannot be outperformed by any baseline nor upper bound. As mentioned earlier, canonical metrics are lexical-based while modern reference-based and reference-free approaches focus on facts. Through mutating reference summaries, our approach can create summaries of different linguistic qualities. Although our approach makes big improvements over baselines on TAC2010 and Newsroom's FLUency, its edge is smaller on Newsroom's COherence. A sentence-level scrambling mutation may improve our approach's performance on COHer 



### Discussions

What is the best mutation?Across datasets, deletion-based mutations are most effective. The two kinds of deletions happen to be complementarily effective for two aspect groups: sentence deletion for content/fact aspects vs. word deletion for linguistic aspects. This is an advantage of our approach that _under a uniformed framework, different summary quality aspects can be gauged by designing different mutation options_.

The complementariness of sentence deletion and word deletion can be well explained as that removing a sentence from a reference summary reduces a great amount of key information while removing a word from a sentence changes it syntactically. We found that word-level mutations are less useful for content/fact aspects, probably because of the inertia of the context after words are altered.

Which training domain/dataset should I use?Due to the composition of summarizers and the limited data size in human evaluation, it is very hard to get a consistent ranking of metrics on different datasets (Bhandari et al., 2020). For example, in Table 1, Billsumm outperforms all baselines and its peers on TAC2010 but not the case on Newsroom and RealSumm.

Still, the impact of training domain seems manageable. The average absolute deviations across the training datasets/domains are given at the bottom of Tables 1 and 2. They mostly below 3.5%. A qualitative analysis shows that the variation seems more due to the characteristics of the text than the domain. Legislative bills (Billsum) have lots of short, hierarchical clauses and thus differ from common English greatly. Scientific papers have many equations and cross-references. There are also many occurrences of LaTeX or MathML in the dataset arXiv. On top of that, all our experiments use different training and test domains. Hence we would say that the impact of domain variation is very small.

## 4 Conclusion

In this paper, we propose a weakly supervised approach to summary quality evaluation. A few mutation methods are introduced to make use of the massive, precious human written summaries in summarization datasets. In cross-domain experiments, our approach achieves better performance than baselines, especially on linguistic aspects. We hope this proof-of-concept study can inspire more reference-free summary evaluation.

## Acknowledgments

 Bao, Luo, Li, and He's work in this paper is partially supported by National Science Foundation (NSF) grants No. MCB-1821828 and No. CNS-1817089. The authors would also like to thank reviewers who have given precious feedback on improving this work.

## References

* S. Banerjee and A. Lavie (2005)Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72. Cited by: SS1.
* M. Bhandari, P. Narayan Gour, A. Ashfaq, P. Liu, and G. Neubig (2020)ReEvaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* A. Cohan, F. Dernoncourt, D. S. Kim, T. Bui, S. Kim, W. Chang, and N. Goharian (2018)A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. . External Links: Document Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of

\begin{table}
\begin{tabular}{c c|c|c c} \hline \hline  & \multicolumn{2}{c|}{TAC2010} & \multicolumn{2}{c}{Newsroom} \\  & & Ling. & COH & FLU \\ \hline Our approach (mutated in word deletion) & \multirow{2}{*}{
\begin{tabular}{c} _Trusted on:_ \\ Billsum \\ arXiv \\ BigPatent \\ \end{tabular} } & **0.46\({}^{1}\)** & 0.65\({}^{2}\) & 0.65\({}^{2}\) \\  & & axiv & 0.38\({}^{3}\) & **0.67\({}^{1}\)** & **0.67\({}^{1}\)** \\  & BigPatent & 0.43\({}^{2}\) & 0.62\({}^{2}\) & 0.63\({}^{3}\) \\ \hline \multirow{4}{*}{Baselines} & BLANC-tune & 0.29 & 0.59 & 0.53 \\  & SummaV-AFI & 0.24 & 0.49 & 0.47 \\  & SummQA-CFD & 0.15 & 0.42 & 0.37 \\  & SUPERIT & 0.32 & 0.62\({}^{2}\) & 0.54 \\  & LS-Score’ & NA & 0.63 & 0.59 \\ \hline \multirow{6}{*}{Upper bounds} & R-1 & 0.26 & 0.23 & 0.22 \\  & R-2 & 0.35 & 0.09 & 0.10 \\  & R-1 & 0.18 & 0.21 & 0.20 \\  & MoveScore & 0.35 & 0.17 & 0.14 \\  & BertScreto & 0.36 & 0.27 & 0.24 \\  & BLEU & 0.35 & -0.06 & -0.04 \\  & METFOR & 0.34 & 0.17 & 0.17 \\  & SS3.5 & xsep & 0.36 & 0.19 & 0.18 \\ \hline \multicolumn{2}{c|}{Our best over baseline beat (\%)} & 41.92 & 8.41 & 25.02 \\ \multicolumn{2}{c|}{Our average absolute deviation (\%)} & 2.72 & 1.71 & 1.74 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Spearman’s correlation on **linguistic** aspects. Spinetics are ranks in each aspect(column.



deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS3.
* S. Ellouze, M. Jaoua, and L. Hadrich Belguith (2017)Machine learning approach to evaluate multilingual summaries. In Proceedings of the Multi-Ling 2017 workshop on summarization and summary evaluation across source types and genres, pp. 47-54. Cited by: SS3.
* Y. Gao, W. Zhao, and S. Eger (2020)SUPERT: towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1347-1354. External Links: Link, Document Cited by: SS3.
* M. Grusky, M. Naaman, and Y. Artzi (2018)Newsroom: a dataset of 1.3 million summaries with diverse extractive strategies. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). External Links: Link, Document Cited by: SS3.
* K. Poritz Hermann, T. Kocisky, E. Grefenstette, L. Espebolt, W. Kay, M. Suleyman, and P. Blunsom (2015)Teaching machines to read and comprehend. In NeurIPS, pp. 1693-1701. External Links: Link, Document Cited by: SS3.
* A. Kornilova and V. Eidelman (2019)BillSum: a corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, Hong Kong, China, pp. 48-56. External Links: Link, Document Cited by: SS3.
* C. Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link, Document Cited by: SS3.
* C. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau (2016)How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2122-2132. External Links: Link, Document Cited by: SS3.
* C. Liu, P. Wang, J. Xu, Z. Li, and J. Ye (2019)Automatic dialogue summary generation for customer service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '19, New York, NY, USA, pp. 1957-1965. External Links: ISBN 978-1-4503-3213-X, Link, Document Cited by: SS3.
* F. Liu and Y. Liu (2008)Correlation between rouge and human evaluation of extractive meeting summaries. In ACL, pp. 201-204. External Links: Link, Document Cited by: SS3.
* R. Nallapati, B. Zhou, C. dos Santos, C. Gulcehre, and B. Xiang (2016)Abastractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290. External Links: Link, Document Cited by: SS3.
* J. Ng and V. Abrecht (2015) Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1925-1930. External Links: Link, Document Cited by: SS3.
* NIST (2010)TAC2010 guided summarization competition. Note: [https://tac.nist.gov/2010/Summarization/Guided-Summ.2010](https://tac.nist.gov/2010/Summarization/Guided-Summ.2010) Cited by: SS3.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BEUR: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. External Links: Link, Document Cited by: SS3.
* M. Peyrard, T. Botschen, and I. Gurevych (2017)Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, Copenhagen, Denmark, pp. 74-84. External Links: Link, Document Cited by: SS3.
* S. Rastkar, G. C. Murphy, and G. Murray (2014)Automatic summarization of bug reports. IEEE Transactions on Software Engineering40 (4), pp. 366-380. External Links: ISSN 1558-1221, Link, Document Cited by: SS3.
* S. Ruseti, M. Dascalu, A. M. Johnson, D. S. McNamara, R. Balyan, K. S. McCarthy, and S. Traussan-Matu (2018)Scoring summaries using recurrent neural networks. In International Conference on Intelligent Tutoring Systems, pp. 191-201. External Links: Link, Document Cited by: SS3.
* T. Scialom, S. Lamprier, B. Piwowarski, and J. Staiano (2019)Ansvers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3246-3256. External Links: Link, Document Cited by: SS3.
* G. Shang, W. Ding, Z. Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorte (2018)Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization. In ACL, pp. 664-674. External Links: Link, Document Cited by: SS3.

 

* Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019. BIG-PATENT: A large-scale dataset for abstractive and coherent summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2204-2213, Florence, Italy. Association for Computational Linguistics.
* Vasilyev et al. (2020) Oleg Vasilyev, Vedant Dharmidharka, and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_, pages 11-20, Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NeurIPS_, pages 5998-6008.
* Wu et al. (2020) Hanlu Wu, Tengfei Ma, Lingfei Wu, Tariro Manyumwa, and Shouling Ji. 2020. Unsupervised reference-free summary quality evaluation via contrastive learning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3612-3621, Online. Association for Computational Linguistics.
* Xia et al. (2019) Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2019. Automatic learner summary assessment for reading comprehension. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2532-2542, Minneapolis, Minnesota. Association for Computational Linguistics.
* Zhang and Cranshaw (2018) Amy X. Zhang and Justin Cranshaw. 2018. Making sense of group chat through collaborative tagging and summarization. _Proc. ACM Hum.-Comput. Interact._, 2(CSCW).
* Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In _International Conference on Learning Representations_.
* Zhao et al. (2019) Wei Zhao, Maxime Peprard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 563-578, Hong Kong, China. Association for Computational Linguistics.
* Zopf (2018) Markus Zopf. 2018. Estimating summary quality with pairwise preferences. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 1687-1696.

## Appendix A Dataset statistics

For test sets:

* **TAC2010 Guided Summarization Task Set****A** consists of 46 topics, each of which is associated with a set of 10 documents. We evaluate the metrics over summaries generated by 43 systems.
* **Newsroom** contains human-rated summaries generated by 7 systems for 60 documents.
* **RealSumm** sampled 100 documents from the CNN/DailyMail test set, and collected human ratings for summaries generated by 11 extractive systems and 14 abstractive systems.

For training sets, the numbers of pairs of documents and reference summaries in the train split are:

* **Billsum**: 18,949
* **Scientific papers/arXiv**: 203,037
* **Big-Patent**: 1,207,222 For each dataset, we use the entire (except for Big-Patent, 10% due to its huge size) train split in Google Tensorflow Datasets for training.

## Appendix B Computational environment and cost

All experiments were carried out on one RTX3090 GPU installed on a desktop computer. The training takes about a week for all three training datasets.

## Appendix C Another type of mutation

In addition to the three mutation methods mentioned already, we have another method called crosspairing.

Illustrated in Figure 3, it is inspired by the next-sentence prediction (NSP) task in original BERT training. Given a document and its reference summary, we create negative data by pairing the document with reference summararies of other documents.

Figure 3: Training sample generation via cross pairing.

 

We assign the label 0 to a mismatching document-summary pair, and the label 1 to any original pair of a document and its reference summary.

## Appendix D Complete empirical results

Due to space limit, we were only able to present the result of the best mutation method in SS3.5. Full results are given in Tables 3, 4, 5, and 6. Pearson's for LS-Score is unable to be produced due to reasons explained in the footnote on page 4.

\begin{table}
\begin{tabular}{c c r|r l|r l} \hline \hline  & Mutation & \multicolumn{2}{l|}{Training set} & \multicolumn{2}{l|}{TAC2010} & \multicolumn{2}{l}{Newswroom} \\  & \multicolumn{2}{l|}{} & \multicolumn{2}{l|}{set} & \multicolumn{2}{l|}{Linguistic} & \multicolumn{2}{l}{COH} & \multicolumn{2}{l}{FLU} \\ \hline \multirow{8}{*}{Our approach} & \multirow{4}{*}{crosspair} & Bilsum & 0.29 & 0.43 & 0.39 \\  & & ArXiv & 0.28 & 0.48 & 0.42 \\  & & BigPaiet & 0.28 & 0.48 & 0.42 \\ \cline{2-7}  & sentence- & \multirow{4}{*}{delete} & Bilsum & 0.33 & 0.59 & 0.53 \\  & & ArXiv & 0.32 & 0.53 & 0.46 \\  & & BiPatent & 0.30 & 0.62 & 0.54 \\ \cline{2-7}  & & Billsum & 0.39 & 0.45 & 0.42 \\  & & ArcXiv & 0.27 & 0.50 & 0.43 \\  & & BigPlaet & 0.38 & 0.41 & 0.31 \\ \cline{2-7}  & & Bissum & 0.31 & 0.55 & 0.53 \\  & & ArXiv & 0.16 & 0.55 & 0.48 \\  & & BigPlaet & 0.19 & 0.51 & 0.48 \\ \cline{2-7}  & & Billsum & 0.33 & 0.60 & 0.57 \\  & & ArXiv & 0.07 & 0.54 & 0.49 \\  & & BigPlaet & 0.24 & 0.54 & 0.46 \\ \cline{2-7}  & & Billsum & 0.46 & 0.65 & 0.65 \\  & & ArXiv & 0.38 & 0.67 & 0.67 \\  & & BigPlaet & 0.43 & 0.62 & 0.63 \\ \hline \multirow{8}{*}{Baselines} & BLANC-tune & 0.29 & 0.59 & 0.53 \\  & SummQA+F1 & 0.24 & 0.49 & 0.47 \\  & SummQA+CPD & 0.15 & 0.42 & 0.37 \\  & SUPERET & 0.32 & 0.62 & 0.54 \\  & LS-Score * & N/A & 0.63 & 0.59 \\ \hline \multirow{8}{*}{Upper bounds} & R-1 & 0.26 & 0.23 & 0.28 & 0.63 \\  & R-2 & 0.35 & 0.09 & 0.10 \\  & R-L & 0.18 & 0.21 & 0.20 \\  & MoversCore & 0.35 & 0.17 & 0.14 \\  & BertScore & 0.36 & 0.27 & 0.24 \\  & BLEU & 0.35 & -0.06 & -0.04 \\  & METCROR & 0.34 & 0.17 & 0.17 \\  & SI\_pyr & 0.36 & 0.19 & 0.18 \\  & S3\_resp & 0.36 & 0.17 & 0.17 \\ \hline \multicolumn{7}{c}{ Our best over baseline best (\%)} & \multicolumn{2}{l|}{19.17} & \multicolumn{2}{l}{-0.28} & \multicolumn{2}{l}{5.49} \\ \hline \multicolumn{7}{c}{ crosspair} & \multicolumn{2}{l|}{2.02} & \multicolumn{2}{l|}{2.75} & \multicolumn{2}{l}{3.00} & \multicolumn{2}{l}{1.02} \\ \multirow{2}{*}{Our average absolute} & sentence-delete & 3.32 & 2.57 & 2.21 & 3.45 & 2.82 \\  & sentence-replace & 2.99 & 3.43 & 2.57 & 9.28 & 3.92 \\  & word-inert & 4.64 & 1.87 & 1.03 & 5.01 & 0.37 \\  & word-delete & 1.96 & 1.79 & 0.82 & 2.55 & 0.44 \\  & word-replace & 7.59 & 1.11 & 1.73 & 2.60 & 1.96 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Full results for **Spearman’s correlation on **linguistic aspects**.

\begin{table}
\begin{tabular}{c c c|r l|r l} \hline \hline  & Mutation & \multicolumn{2}{l|}{Training TAC2010} & \multicolumn{2}{l|}{Newswroom} & \multicolumn{2}{l}{RealSumm} \\  & \multicolumn{2}{l|}{set} & \multicolumn{2}{l|}{Pyramid} & \multicolumn{2}{l|}{INFREL} & \multicolumn{2}{l}{Abs} & \multicolumn{2}{l}{Ext} \\ \hline \multirow{8}{*}{Our approach} & \multirow{4}{*}{crosspair} & Bilsum & 0.38 & 0.50 & 0.49 & 0.66 & -0.05 \\  & & ArXiv & 0.37 & 0.55 & 0.06 & -0.08 \\  & & BigPaiet & 0.33 & 0.56 & 0.57 & -0.06 & -0.05 \\ \cline{2-7}  & sentence- & \multirow{4}{*}{replace} & Bilsum & 0.44 & 0.47 & 0.42 & 0.04 & -0.08 \\  & & AirXiv & 0.35 & 0.58 & 0.49 & 0.19 & 0.03 \\  & & BigPaiet & 0.39 & 0.49 & 0.46 & -0.08 & -0.04 \\ \cline{2-7}  & & Billsum & 0.21 & 0.60 & 0.56 & 0.06 & -0.01 \\  & & ArXiv & 0.10 & 0.66 & 0.58 & 0.20 & -0.01 \\  & & BigPlaet & 0.20 & 0.63 & 0.59 & 0.14 & -0.02 \\ \cline{2-7}  & & Billsum & 0.27 & 0.64 & 0.61 & 0.12 & 0.02 \\  & & ArXiv & 0.23 & 0.62 & 0.59 & 0.17 & 0.01 \\  & & BigPlaet & 0.28 & 0.59 & 0.60 & 0.10 & 0.01 \\ \cline{2-7}  & & Billsum & 0.25 & 0.66 & 0.60 & 0.10 & -0.03 \\  & & arXiv & 0.08 & 0.65 & 0.57 & 0.15 & -0.02 \\  & & BigPlaet & 0.25 & 0.63 & 0.62 & 0.07 & -0.06 \\ \cline{2-7}  & sentence- & \multirow{4}{*}{sentence} & Bilsum & 0.49 & 0.70 & 0.61 & 0.26 & 0.01 \\  & & ArXiv & 0.41 & 0.69 & 0.59 & 0.34 & 0.12 \\  & & BigPlaet & 0.42 & 0.75 & 0.65 & 0.33 & 0.13 \\ \hline \multirow{8}{*}{Baselines} & BLANC-tune & 0.43 & 0.69 & 0.61 & 0.31 & 0.11 \\  & SummaQA+F1 & 0.30 & 0.57 & 0.52 & 0.22 & 0.08 \\  & SurpanaQA+CFD & 0.29 & 0.54 & 0.44 & 0.24 & 0.05 \\  & SUPERET & 0.48 & 0.68 & 0.60 & 0.25 & 0.07 \\  & LS-Score * & N/A & 0.70 & 0.64 & N/A & N/A \\ \hline \multirow{8}{*}{Upper bounds} & R-1 & 0.56 & 0.32 & 0.28 & 0.63 & 0.22 \\  & R-2 & 0.64 & 0.15 & 0.13 & 0.56 & 0.22 \\  & R-L & 0.50 & 0.30 & 0.26 & 0.60 & 0.21 \\  & MorviCore & 0.72 & 0.22 & 0.22 & 0.50 & 0.19 \\  & BestSource & 0.68 & 0.32 & 0.82 & 0.57 & 0.19 \\  & BLEU & 0.60 & -0.08 & 0.01 & 0.30 & 0.16 \\  & METCROR & 0.67 & 0.42 & 0.24 & 0.63 & 0.25 \\  & S3\_pyr & 0.73 & 0.27 & 0.25 & 0.64 & 0.

 



\begin{table}
\begin{tabular}{c c r|r l|r l} \hline \hline  & Mutation & \multicolumn{2}{l|}{Training set} & \multicolumn{1}{l|}{TAC2010} & \multicolumn{3}{l}{Newsroom} \\  & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{Linguistic} & \multicolumn{1}{l}{COH} & \multicolumn{1}{l}{FLU} \\ \hline \multirow{6}{*}{Our Approach} & \multicolumn{2}{l|}{Billsum} & 0.44 & 0.63 & 0.66 & -0.07 & -0.05 \\  & \multicolumn{1}{l|}{ArXiv} & 0.45 & 0.62 & 0.65 & -0.07 & -0.07 \\  & \multicolumn{1}{l|}{BigPatent} & 0.39 & 0.63 & 0.68 & -0.07 & -0.05 \\ \cline{2-6}  & sentence- & \multicolumn{1}{l|}{Billsum} & 0.48 & 0.64 & 0.67 & 0.04 & -0.09 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.24 & 0.56 & 0.58 & 0.07 & 0.05 \\  & \multicolumn{1}{l|}{BigPatent} & 0.41 & 0.59 & 0.61 & -0.07 & -0.04 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.34 & 0.70 & 0.72 & 0.08 & 0.00 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.30 & 0.67 & 0.69 & 0.19 & -0.01 \\  & \multicolumn{1}{l|}{BigPatent} & 0.26 & 0.64 & 0.68 & 0.14 & -0.02 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.39 & 0.76 & 0.78 & 0.12 & 0.05 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.39 & 0.68 & 0.70 & 0.18 & 0.03 \\  & \multicolumn{1}{l|}{BigPatent} & 0.38 & 0.71 & 0.74 & 0.13 & 0.01 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.35 & 0.72 & 0.76 & 0.09 & -0.04 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.29 & 0.67 & 0.70 & 0.12 & 0.00 \\  & \multicolumn{1}{l|}{BigPatent} & 0.29 & 0.66 & 0.71 & 0.08 & -0.04 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.55 & 0.75 & 0.74 & 0.26 & 0.06 \\  & \multicolumn{1}{l|}{delete} & \multicolumn{1}{l|}{ArXiv} & 0.47 & 0.69 & 0.61 & 0.34 & 0.11 \\  & \multicolumn{1}{l|}{BigPatent} & 0.50 & 0.79 & 0.72 & 0.35 & 0.16 \\ \hline \multirow{6}{*}{Baselines} & Blanc-tune & 0.51 & 0.73 & 0.68 & 0.33 & 0.13 \\  & summaQA-F1 & 0.34 & 0.59 & 0.55 & 0.21 & 0.09 \\  & SummaQA-CFD & 0.33 & 0.60 & 0.52 & 0.25 & 0.06 \\  & Supert & 0.55 & 0.77 & 0.77 & 0.27 & 0.09 \\ \hline \multirow{6}{*}{Upper bounds} & R-1 & 0.55 & 0.26 & 0.25 & 0.66 & 0.26 \\  & R-2 & 0.69 & 0.03 & 0.03 & 0.59 & 0.24 \\  & R-L & 0.48 & 0.14 & 0.13 & 0.62 & 0.25 \\  & MoverScore & 0.68 & 0.06 & 0.09 & 0.51 & 0.20 \\  & BertScore & 0.65 & 0.29 & 0.28 & 0.61 & 0.24 \\  & BLEU & 0.62 & 0.14 & -0.10 & 0.32 & 0.15 \\  & METEOR & 0.71 & 0.08 & 0.09 & 0.67 & 0.28 \\  & S3\_pyr & 0.76 & 0.11 & 0.10 & 0.67 & 0.28 \\  & S3\_resp & 0.76 & 0.04 & 0.04 & 0.65 & 0.28 \\ \hline \multicolumn{6}{c}{Our best over baseline best (%)} & 0.15 & 2.75 & 1.37 & 7.12 & 28.53 \\ \hline \multirow{6}{*}{Upper bounds} & crosspair & 2.41 & 0.42 & 1.02 & 0.00 & 0.97 \\  & sentence-delete & 2.85 & 3.53 & 5.27 & 3.68 & 3.68 \\  & sentence-replace & 9.43 & 2.74 & 3.43 & 5.65 & 5.04 \\  & word-insert & 2.74 & 1.92 & 1.75 & 3.80 & 0.57 \\ deviation (\%) & word-delete & 0.42 & 2.78 & 2.97 & 2.35 & 1.25 \\  & word-replace & 2.85 & 2.49 & 2.57 & 1.74 & 1.46 \\ \hline \end{tabular}
\end{table}
Table 6: Full results for **Pearson’s correlation on linguistic aspects**.

\begin{table}
\begin{tabular}{c c c|r l|r l} \hline \hline  & Mutation & \multicolumn{2}{l|}{Training set} & \multicolumn{2}{l|}{TAC2010} & \multicolumn{2}{l}{Newsroom} & \multicolumn{2}{l}{RealSumm} \\  & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{} & \multicolumn{1}{l|}{Abs Ext} \\ \hline \multirow{6}{*}{Our approach} & \multicolumn{2}{l|}{Billsum} & 0.44 & 0.63 & 0.66 & -0.07 & -0.05 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.45 & 0.62 & 0.65 & -0.07 & -0.07 \\  & \multicolumn{1}{l|}{BigPatent} & 0.39 & 0.63 & 0.68 & -0.07 & -0.05 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.48 & 0.64 & 0.67 & 0.04 & -0.09 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.24 & 0.56 & 0.58 & 0.07 & 0.05 \\  & \multicolumn{1}{l|}{BigPatent} & 0.41 & 0.59 & 0.61 & -0.07 & -0.04 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.34 & 0.70 & 0.72 & 0.08 & 0.00 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.30 & 0.67 & 0.69 & 0.19 & -0.01 \\  & \multicolumn{1}{l|}{BigPatent} & 0.26 & 0.64 & 0.68 & 0.14 & -0.02 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.39 & 0.76 & 0.78 & 0.12 & 0.05 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.39 & 0.68 & 0.70 & 0.18 & 0.03 \\  & \multicolumn{1}{l|}{BigPatent} & 0.38 & 0.71 & 0.74 & 0.13 & 0.01 \\ \cline{2-6}  & \multicolumn{1}{l|}{Billsum} & 0.35 & 0.72 & 0.76 & 0.09 & -0.04 \\  & \multicolumn{1}{l|}{ArtXiv} & 0.29 & 0.67 & 0.70 & 0.12 & 0.00 \\  & \multicolumn{1}{l|}{BigPatent} & 0.29 & 0.66 & 0.71 & 0.08 & -0.04 \\ \cline{2-6}  & \multicolumn{1}{l|}{sentence-} & \multicolumn{1}{l|}{Bitsum} & 0.55 & 0.75 & 0.74 & 0.26 & 0.06 \\  & \multicolumn{1}{l|}{delete} & \multicol