SlowGuess/NAACL_2022_OCR huggingface dataset:
summaries. In ACL, pages 201-204. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290.
Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925-1930.
NIST. 2010. TAC2010 guided summarization competition. https://tac.nist.gov/2010/ Summarization/Guided-Summ.2010. guidelines.html. Accessed: 2021-08-16.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84, Copenhagen, Denmark. Association for Computational Linguistics.
S. Rastkar, G. C. Murphy, and G. Murray. 2014. Automatic summarization of bug reports. IEEE Transactions on Software Engineering, 40(4):366-380.
Stefan Ruseti, Mihai Dascalu, Amy M Johnson, Danielle S McNamara, Renu Balyan, Kathryn S McCarthy, and Stefan Trausan-Matu. 2018. Scoring summaries using recurrent neural networks. In International Conference on Intelligent Tutoring Systems, pages 191-201. Springer.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3246-3256, Hong Kong, China. Association for Computational Linguistics.
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorre. 2018. Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization. In ACL, pages 664-674. Association for Computational Linguistics.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIG-PATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Hanlu Wu, Tengfei Ma, Lingfei Wu, Tariro Manyumwa, and Shouling Ji. 2020. Unsupervised reference-free summary quality evaluation via contrastive learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3612-3621, Online. Association for Computational Linguistics.
Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2019. Automatic learner summary assessment for reading comprehension. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2532-2542, Minneapolis, Minnesota. Association for Computational Linguistics.
Amy X. Zhang and Justin Cranshaw. 2018. Making sense of group chat through collaborative tagging and summarization. Proc. ACM Hum.-Comput. Interact., 2(CSCW).
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations.
Markus Zopf. 2018. Estimating summary quality with pairwise preferences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1687-1696.
For test sets:
- TAC2010 Guided Summarization Task Set A consists of 46 topics, each of which is associated with a set of 10 documents. We evaluate the metrics over summaries generated by 43 systems.
- Newsroom contains human-rated summaries generated by 7 systems for 60 documents.
- RealSumm sampled 100 documents from the CNN/DailyMail test set, and collected human ratings for summaries generated by 11 extrative systems and 14 abstractive systems.
For training sets, the numbers of pairs of documents and reference summaries in the train split are:
Billsum: 18,949
- Scientific papers/arXiv: 203,037
Big-Patent: 1,207,222
For each dataset, we use the entire (except for BigPatent,  $10\%$  due to its huge size) train split in Google Tensorflow Datasets for training.
B Computational environment and cost
All experiments were carried out on one RTX3090 GPU installed on a desktop computer. The training takes about a week for all three training datasets.
C Another type of mutation
In addition to the three mutation methods mentioned already, we have another method called crosspairing.
Figure 3: Training sample generation via cross pairing.
Illustrated in Figure 3, it is inspired by the next-sentence prediction (NSP) task in original BERT training. Given a document and its reference summary, we create negative data by pairing the document with reference summaries of other documents.
2456 7
We assign the label 0 to a mismatching document-summary pair, and the label 1 to any original pair of a document and its reference summary.
D Complete empirical results
Due to space limit, we were only able to present the result of the best mutation method in § 3.5. Full results are given in Tables 3, 4, 5, and 6. Pearson's for LS-Score is unable to be produced due to reasons explained in the footnote on page 4.
Table 3: Full results for Spearman's correlation on content/fact aspects.
Mutation	Training set	TAC2010 Pyramid	Newsroom INF	REL	RealSumm Abs	Ext
Our approach	crosspair	Billsum	0.38	0.50	0.49	-0.06	-0.05
ArXiv	0.37	0.57	0.55	-0.06	-0.08
BigPatent	0.33	0.56	0.57	-0.06	-0.05
sentence-replace	Billsum	0.44	0.47	0.42	0.04	-0.08
ArXiv	0.35	0.55	0.49	0.19	0.03
BigPatent	0.39	0.49	0.46	-0.08	-0.04
word-insert	Billsum	0.21	0.60	0.56	0.06	-0.01
ArXiv	0.10	0.66	0.58	0.20	-0.01
BigPatent	0.20	0.63	0.59	0.14	-0.02
word-delete	Billsum	0.27	0.64	0.61	0.12	0.02
ArXiv	0.23	0.62	0.59	0.17	0.01
BigPatent	0.28	0.59	0.60	0.10	0.01
word-replace	Billsum	0.25	0.66	0.60	0.10	-0.03
ArXiv	0.08	0.65	0.57	0.15	-0.02
BigPatent	0.25	0.63	0.62	0.07	-0.06
sentence-delete	Billsum	0.49	0.70	0.61	0.26	0.01
ArXiv	0.41	0.69	0.59	0.34	0.12
BigPatent	0.42	0.75	0.65	0.33	0.13
Baselines	BLANC-tune	0.43	0.69	0.61	0.31	0.11
SummaQA-F1	0.30	0.57	0.52	0.22	0.08
SummaQA-CFD	0.29	0.54	0.44	0.24	0.05
SUPERT	0.48	0.69	0.60	0.25	0.07
LS-Score *	N/A	0.70	0.64	N/A	N/A
Upper bounds	R-1	0.56	0.32	0.28	0.63	0.22
R-2	0.64	0.15	0.13	0.56	0.22
R-L	0.50	0.30	0.26	0.60	0.21
MoverScore	0.72	0.22	0.22	0.50	0.19
BertScore	0.68	0.32	0.28	0.57	0.19
BLEU	0.60	-0.08	-0.01	0.30	0.16
METEOR	0.67	0.24	0.24	0.63	0.25
S3_pyr	0.73	0.27	0.25	0.64	0.24
S3_resp	0.73	0.25	0.22	0.63	0.24
Our best over baseline best (%)	-8.47	-4.63	2.14	-35.93	-76.38
Our average absolute deviation (%)	crosspair	2.02	2.75	3.00	0.00	1.02
sentence-delete	3.32	2.57	2.21	3.45	5.28
sentence-replace	2.99	3.34	2.57	9.28	3.92
word-insert	4.64	1.87	1.03	5.01	0.37
word-delete	1.96	1.79	0.82	2.55	0.44
word-place	7.59	1.11	1.73	2.60	1.96
Table 4: Full results for Spearman's correlation on linguistic aspects.
Mutation	Training set	TAC2010 Linguistic	Newsroom COH	FLU
Our approach	crossover	Billsum	0.29	0.43	0.39
ArXiv	0.28	0.48	0.42
BigPatent	0.28	0.48	0.42
sentence-delete	Billsum	0.33	0.59	0.53
ArXiv	0.32	0.53	0.46
BigPatent	0.30	0.62	0.54
sentence replace	Billsum	0.39	0.45	0.42
ArXiv	0.27	0.50	0.43
BigPatent	0.38	0.41	0.31
word-insert	Billsum	0.31	0.55	0.53
ArXiv	0.16	0.55	0.48
BigPatent	0.19	0.51	0.48
word-replace	Billsum	0.33	0.60	0.57
ArXiv	0.07	0.54	0.49
BigPatent	0.24	0.54	0.46
word-delete	Billsum	0.46	0.65	0.65
ArXiv	0.38	0.67	0.67
BigPatent	0.43	0.62	0.63
Baselines	BLANC-tune	0.29	0.59	0.53
SummaQA-F1	0.24	0.49	0.47
SummaQA-CFD	0.15	0.42	0.37
SUPERT	0.32	0.62	0.54
LS-Score *	N/A	0.63	0.59
Upper bounds	R-1	0.26	0.23	0.22
R-2	0.35	0.09	0.10
R-L	0.18	0.21	0.20
MoverScore	0.35	0.17	0.14
BertScore	0.36	0.27	0.24
BLEU	0.35	-0.06	-0.04
METEOR	0.34	0.17	0.17
S3_pyr	0.36	0.19	0.18
S3_resp	0.36	0.17	0.17
Our best over baseline best (%)	19.17	-0.28	5.49
Our average absolute deviation (%)	crosspair	0.29	2.00	1.50
sentence-delete	1.15	3.10	3.17
sentence-replace	4.97	3.05	5.05
word-insert	6.01	1.62	2.38
word-delete	2.72	1.71	1.74
word-replace	9.28	2.56	4.23
2457 8
Table 5: Full results for Pearson's correlation on content/fact aspects.
Mutation	Training set	TAC2010 Pyramid	Newsroom INF	REL	RealSum Abs	Ext
Our approach	crosspair	Billsum	0.44	0.63	0.66	-0.07	-0.05
ArXiv	0.45	0.62	0.65	-0.07	-0.07
BigPatent	0.39	0.63	0.68	-0.07	-0.05
sentence-replace	Billsum	0.48	0.64	0.67	0.04	-0.09
ArXiv	0.24	0.56	0.58	0.07	0.05
BigPatent	0.41	0.59	0.61	-0.07	-0.04
word-insert	Billsum	0.34	0.70	0.72	0.08	0.00
ArXiv	0.30	0.67	0.69	0.19	-0.01
BigPatent	0.26	0.64	0.68	0.14	-0.02
word-delete	Billsum	0.39	0.76	0.78	0.12	0.05
ArXiv	0.39	0.68	0.70	0.18	0.03
BigPatent	0.38	0.71	0.74	0.13	0.01
word-replace	Billsum	0.35	0.72	0.76	0.09	-0.04
ArXiv	0.29	0.67	0.70	0.12	0.00
BigPatent	0.29	0.66	0.71	0.08	-0.04
sentence-delete	Billsum	0.55	0.75	0.74	0.26	0.06
ArXiv	0.47	0.69	0.61	0.34	0.11
BigPatent	0.50	0.79	0.72	0.35	0.16
Baselines	Blanc-tune	0.51	0.73	0.68	0.33	0.13
summaQA-F1	0.34	0.59	0.55	0.21	0.09
SummaQA-CFD	0.33	0.60	0.52	0.25	0.06
Supert	0.55	0.77	0.77	0.27	0.09
Upper bounds	R-1	0.55	0.26	0.25	0.66	0.26
R-2	0.69	0.03	0.03	0.59	0.24
R-L	0.48	0.14	0.13	0.62	0.25
MoverScore	0.68	0.06	0.09	0.51	0.20
BertScore	0.65	0.29	0.28	0.61	0.24
BLEU	0.62	-0.14	-0.10	0.32	0.15
METEOR	0.71	0.08	0.09	0.67	0.28
S3_pyr	0.76	0.11	0.10	0.67	0.28
S3_resp	0.76	0.04	0.04	0.65	0.28
Our best over baseline best (%)	0.15	2.75	1.37	7.12	28.53
Our average absolute deviation (%)	crosspair sentence-delete sentence replace word-insert word-delete word replace	2.41 2.85 9.43 2.74 0.42 2.85	0.42 3.53 2.74 1.92 2.78 2.49	1.02 5.27 3.43 1.75 2.97 2.57	0.00 3.68 5.65 3.80 2.35 1.74	0.97 3.68 5.04 0.57 1.25 1.46
Table 6: Full results for Pearson's correlation on linguistic aspects.
Mutation	Training set	TAC2010 Linguistic	Newsroom COH	FLU
Our Approach	crosspair	Billsum	0.39	0.52	0.46
ArXiv	0.39	0.50	0.44
BigPatent	0.40	0.51	0.44
sentence-delete	Billsum	0.48	0.61	0.55
ArXiv	0.39	0.56	0.50
BigPatent	0.43	0.65	0.57
sentence replace	Billsum	0.43	0.52	0.44
ArXiv	0.21	0.48	0.42
BigPatent	0.39	0.45	0.38
word-insert	Billsum	0.45	0.60	0.56
ArXiv	0.35	0.56	0.52
BigPatent	0.32	0.52	0.46
word-replace	Billsum	0.47	0.61	0.58
ArXiv	0.35	0.56	0.53
BigPatent	0.33	0.53	0.48
word-delete	Billsum	0.56	0.69	0.67
ArXiv	0.51	0.67	0.66
BigPatent	0.49	0.66	0.64
Baselines	Blanc-tune	0.42	0.62	0.59
summaQA-F1	0.29	0.51	0.47
SummaQA-CFD	0.21	0.48	0.43
Supert	0.46	0.65	0.58
Upper bounds	R-1	0.27	0.17	0.14
R-2	0.40	-0.02	-0.02
R-L	0.18	0.07	0.06
MoverScore	0.43	0.02	0.00
BertScore	0.50	0.21	0.17
BLEU	0.36	-0.14	-0.12
METEOR	0.46	0.03	0.02
S3_pyr	0.45	0.04	0.03
S3_resp	0.44	-0.01	-0.02
Our best over baseline best (%)	21.28	6.71	13.50
Our average absolute deviation (%)	crosspair	0.43	0.64	0.93
sentence-delete	3.01	3.20	2.65
sentence-replace	8.89	2.51	2.39
word-insert	5.29	2.86	3.35
word-delete	2.56	1.27	0.98
word-replace	6.02	2.88	3.25