SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 2: Single system reranking on CNNDM. Base denotes the base system. Supervised denotes the Refactor directly trained on the base systems' outputs. Pre-trained denotes the pretrained Refactor. Fine-tuned denotes the fine-tuned model. R-1, R-2 and R-L denote ROUGE-1, ROUGE-2 and ROUGE-L. †: significantly better than the base system (p < 0.01).
on 4 GTX-1080-Ti GPUs while fine-tuning takes around 20 hours.
5.5 Exp-I: Single System Reranking
We use BART and GSum for this experiment, and use beam search to generate the candidate summaries where the beam size is set to 4.
The results are listed in Tab. 2, which shows that (1) Refactor can boost the base system's performance by a significant margin, (2) the fine-tuned Refactor outperforms supervised Refactor directly trained on the base system's outputs, showing the effectiveness of the two-step training. Notably, we observe the fine-tuned Refactor can boost BART's performance from 44.26 to 45.15 on ROUGE-1, indicating that the top-1 output selected by beam search is not always the best one, and Refactor can effectively utilize the complementarity introduced by considering all the beam search results.
5.6 Exp-II: Multiple Systems Stacking
Summary-level For summary-level combination, we explore two-system combination (BART & pre-trained Refactor) and three-system combination (BART, GSum & pre-trained Refactor). The results are shown in Tab. 3.
Sentence-level For sentence-level combination, we use BART and pre-trained Refactor as the base
<sup>7</sup>http://www.cs.cornell.edu/people/tj/svm_light/svm_rank.html  
<sup>8</sup>We use the 'bert-base-uncased' version with 110M parameters.
Setting	Method	R-1	R-2	R-L
Base	BART	44.26	21.12	41.16
Refactor	44.13	20.51	40.29
GSum	45.93	22.30	42.68
Two	Min	40.40	17.64	37.12
Max	47.99	23.99	44.33
Random	44.25	20.87	40.78
BERTScore	43.95	20.45	40.23
RankSVM	44.66	21.32	41.44
Supervised†	44.75	21.40	41.47
Pre-trained‡	44.66	21.19	41.15
Fine-tuned†	45.04	21.61	41.72
Three	Min	39.51	17.01	36.35
Max	49.94	25.59	46.30
Random	44.82	21.35	41.44
BERTScore	44.10	20.64	40.42
RankSVM	45.72	22.13	42.58
Supervised	45.80	22.25	42.68
Pre-trained	45.27	21.74	41.93
Fine-tuned†	46.12	22.46	42.92
Table 3: Summary level combination on CNNDM. Two denotes two-system combination (BART and pre-trained Refactor). Three denotes three-system combination (BART, pre-trained Refactor and GSum). R-1, R-2 and R-L denote ROUGE-1, ROUGE-2 and ROUGE-L. †: significantly better than the best single system  $(p < 0.01)$ .
System	R-1	R-2	R-L
BART	44.26	21.12	41.16
Refactor	44.13	20.51	40.29
Min	31.51	10.83	28.87
Max	50.91	26.07	46.97
Random	41.66	18.77	38.27
BERTScore	43.55	20.14	39.84
RankSVM	43.18	19.91	39.51
Supervised †	44.96	21.50	41.43
Pre-trained†	44.88	21.13	41.16
Fine-tuned†	44.93	21.48	41.42
systems. The sentences of each system's output are merged together to form the candidate sentence set, and all combinations of three sentences in the candidate set are generated as candidate summaries. To prune the candidates, we use tri-gram blocking to filter out candidates of which there exists an identical tri-gram in two sentences. The average number of candidates in the test set is 15.8. The results are shown in Tab. 4.
We have the following observations: (1) the pretrained Refactor can already outperform the base systems, and (2) fine-tuning can further improve the performance. Meanwhile, we notice there are two exceptions: (i) For sentence-level combina
Table 4: Sentence level combination on CNNDM. R-1, R-2 and R-L denote ROUGE-1, ROUGE-2 and ROUGE-L. †: significantly better than the best single system  $(p < 0.01)$ .
bin	#sys	Max	Min	Rand	Best	Ours
39-40	3	45.28	34.30	39.88	39.98	40.45
41-42	8	50.14	32.65	41.44	41.89	43.20
42-43	3	47.37	36.79	42.10	42.27	43.38
43-44	2	47.60	39.63	43.58	43.97	44.07
44-45	3	50.29	38.66	44.58	44.68	45.29
Table 5: Multiple system combination. bin denotes the bin range. #sys denotes the number of systems. Ours denotes the pre-trained Refactor model. Best denotes the candidate system with best performance.
tion, supervised Refactor has similar performance as fine-tuned Refactor. We hypothesis that this is because here the number of candidates in the fine-tuning data is relatively large, therefore directly training on the fine-tuning data is sufficient enough. (ii) The pre-trained Refactor cannot outperform GSum model in the three-system combination setting in Tab. 3. The reason might be that GSum has much stronger performance than the other two systems, which intuitively makes the expected gain from system combination lower than other settings.
5.7 Exp-III: Generalization on 19 Top-performing Systems
To evaluate the Refactor's generalization ability, we explore another setting where the pre-trained Refactor is directly used to select the outputs of multiple systems without fine-tuning.
To this end, we collect 19 top-performing summarization systems on CNNDM dataset. Here, we investigate if our Refactor can boost the performance of candidate systems with similar performance. In addition, we also aim to investigate how the range width of different systems' performance affects Refactor's performance. Therefore, we group the candidate systems into equal-width bins based on their average ROUGE-1 scores, and evaluate our Refactor on each bin separately.
In Tab. 5 we report the average ROUGE-1 scores of the oracles, Refactor, and the best candidate system in each bin whose width is 1. Refactor consistently outperforms the best candidate system, showing its generalization ability.
Next, in Fig. 3 we plot the change of Refactor's performance with different bin widths. We define the success rate of Refactor with a given bin width to be the number of bins where Refactor outperforms the single best base system normalized by the total number of bins. We observe that Refactor is more likely to improve the performance of base systems when the system-level performance of the
Method	XSum	PubMed	WikiHow
R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Base	47.12	24.46	39.04	43.42	15.32	39.21	41.98	18.09	40.53
Min	42.45	20.50	35.19	39.60	13.57	35.53	40.55	17.40	39.18
Max	51.51	28.04	42.70	45.23	16.72	40.67	43.00	18.44	41.44
Random	46.98	24.08	38.88	42.39	15.12	38.08	41.77	17.92	40.33
BERTScore	47.13	24.04	38.89	43.64	15.40	39.41	41.77	17.93	40.29
RankSVM	46.85	24.31	39.09	43.63	15.34	39.46	42.00	18.08	40.57
Pre-trained	47.45	24.55	39.41	43.58	15.36	39.38	41.97	18.03	40.52
Fine-tuned	47.32	24.31	39.22	43.72	15.41	39.51	42.12	18.13	40.66
Table 6: Single system reranking on other datasets. Pre-trained denotes the pre-trained Refactor model. Fine-tuned denotes the fine-tuned model. R-1, R-2 and R-L denote ROUGE-1, ROUGE-2 and ROUGE-L separately.
Figure 3: The Refactor's success rates with different bin widths.  $\mathbf{W}$  denotes the bin widths measured by ROUGE-1.  $\mathbf{R}$  denotes the success rate of the Refactor outperforming the single best base system.
base systems is similar. Intuitively, if one base system is significantly better than the other systems, it is more difficult for Refactor to use other systems to complement the best base system.
5.8 Exp-IV: Effectiveness on More Popular Datasets
Next, we move on to other text summarization datasets to evaluate our proposed method's strength beyond CNNDM dataset. Some of the datasets used here are not as well-studied as CNNDM dataset, so there are less top-performing systems on these datasets. Therefore, here we focus on the experiments of the single system setting.
Setup Regarding the pre-trained Refactor, we use an extractive oracle to select document sentences and use the combinations of these sentences as candidates. In addition, since on Xsum the abstractive systems outperform extractive systems by a large margin, we use a pre-trained BART model with Diverse Beam Search (Vijayakumar et al., 2018) to generate 16 candidates per sample
for pre-training. Regarding system re-ranking, we use BART as the base system to generate the candidate summaries except on Xsum dataset, where we use PEGASUS since it achieves better performance. Similar to §5.5, we use the outputs of beam search as the candidates. We select the first 4 outputs as the candidates.
The results in Tab. 6 show that Refactor is able to bring stable improvement over the base systems. The average summary length of these datasets varies from 23.3 (XSum) to 210.3 (Pubmed). Therefore, the results here demonstrate the Refactor can be applied to datasets with different characteristics. On XSum dataset, the pre-trained Refactor outperforms the fine-tuned Refactor. This may result from the additional pre-training data we introduced using BART, which is effective enough to train the Refactor for reranking PEGASUS output.
5.9 Fine-grained Analysis
We perform a fine-grained evaluation of Refactor to understand where improvement mainly comes.
Setup We choose the summary-level system combination setting on CNNDM test set in §5.6 as a case study, where the base systems are: BART and pre-trained Refactor, and then we use a fine-tuned Refactor to combine them. Specifically, we first (i) define  $\delta(C_{\mathrm{BART}}, C_{\mathrm{Pretrain}})$  as the performance (i.e., ROUGE) gap on the candidate summary  $C$ . (ii) then partition test samples into different buckets  $S_1, \dots, S_n$  according to the performance gap  $\delta$ . (iii) calculate selection accuracy for each bucket, which represents how accurately the Refactor can
As introduced in §4.4, Refactor could be used as either a base system or a system combinator.
Figure 4: Fine-tuned Refactor's selection accuracy on CNNDM with different difficulties. The X-axis is the difference of ROUGE score of BART and pre-trained Refactor outputs.
identify the best one from two candidate summaries.
The results are shown in Fig. 4. We observe that the selection accuracy is increasing as the gap  $\delta$  becoming larger, indicating that Refactor performs better on the candidate summaries with diverse performance. Combining the results we get in §5.7, we conclude that Refactor has the largest potential gain when the base systems effectively complement each other - They have similar system-level performance but diverse summary-level performance. For example, each base system may perform significantly better than others on a subset of data with different characteristics but could not outperform others across the whole dataset.
6 Implications and Future Directions
We present a general framework for utilizing the complementarity of modern text summarization systems by formulating text summarization as a two-stage learning problem. Our proposed model, Refactor, can be used either as a base system or a meta system, effectively mitigating the learning gaps introduced in the two-stage learning. Experimental results show that Refactor is able to boost the performance of the base systems, and achieves the state-of-the-art performance on CNNDM and XSum datasets. We believe this work opens up a new direction for improving the performance of text summarization systems apart from an iterative process of searching for better model architectures - The gain of performance could be made by fully investigating and utilizing the complementarity of different systems with various architectures, problem formulations, decoding strategies, etc.
We thank Professor Graham Neubig and anonymous reviewers for valuable feedback and helpful suggestions. This work was supported in part by a grant under the Northrop Grumman SOTE-RIA project and the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
Michael Collins and Terry Koo. 2005. Discriminative reranking for natural language parsing. Computational Linguistics, 31(1):25-70.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1).
Bich-Ngoc Do and Ines Rehbein. 2020. Neural reranking for dependency parsing: An evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4123-4133, Online. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13063-13075.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739-3748, Brussels, Belgium. Association for Computational Linguistics.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020. Gsum: A general framework for guided neural abstractive summarization. arXiv preprint arXiv:2010.08014.
Kevin Duh, Katsuhito Sudoh, Xianchao Wu, Hajime Tsukada, and Masaaki Nagata. 2011. Generalized minimum bayes risk system combination. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1356-1360.
Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1347-1354, Online. Association for Computational Linguistics.
Ruifang Ge and Raymond J. Mooney. 2006. Discriminative reranking for semantic parsing. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 263-270, Sydney, Australia. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Tobias Glasmachers. 2017. Limits of end-to-end learning. arXiv preprint arXiv:1704.08305.
Jesús González-Rubio, Alfons Juan, and Francisco Casacuberta. 2011. Minimum Bayes-risk system combination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1268-1277, Portland, Oregon, USA. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Kai Hong, Mitchell Marcus, and Ani Nenkova. 2015. System combination for multi-document summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Lisbon, Portugal. Association for Computational Linguistics.
Nabil Hossain, Marjan Ghazvininejad, and Luke Zettlemoyer. 2020. Simple and effective retrieve-edit-terank text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2532-2538, Online. Association for Computational Linguistics.
Liang Huang. 2008. Forest reranking: Discriminative parsing with non-local features. In Proceedings of ACL-08: HLT, pages 586-594, Columbus, Ohio. Association for Computational Linguistics.
Xuancheng Huang, Jiacheng Zhang, Zhixing Tan, Derek F. Wong, Huanbo Luan, Jingfang Xu, Maosong Sun, and Yang Liu. 2020. Modeling voting for system combination in machine translation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3694-3701. ijcai.org.
Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142.
Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818-1828, Brussels, Belgium. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mahnaz Koupae and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chen Li, Yang Liu, and Lin Zhao. 2015. Improving update summarization via supervised ILP and sentence reranking. In Proceedings of the 2015 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pages 1317-1322, Denver, Colorado. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Tomoya Mizumoto and Yuji Matsumoto. 2016. Discriminative reranking for grammatical error correction with statistical machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1133-1138, San Diego, California. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Căglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018a. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 177-184.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Kai Ming Ting and Ian H. Witten. 1997. Stacked generalizations: When does it work? In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, IJCAI '97, Nagoya, Japan, August 23-29, 1997, 2 Volumes, pages 866-873. Morgan Kaufmann.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence.
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous graph neural networks for extractive document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6209-6219, Online. Association for Computational Linguistics.
Taro Watanabe and Eiichiro Sumita. 2011. Machine translation system combination by confusion forest. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1249-1257, Portland,
Oregon, USA. Association for Computational Linguistics.
Andrew M Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan-Andrei Iliescu, Mikel Lujan, and Gavin Brown. 2019. To ensemble or not ensemble: When does end-to-end training fail. In Computer Vision and Pattern Recognition (CVPR).
Pengcheng Yin and Graham Neubig. 2019. *Reranking for neural semantic parsing*. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4553–4559, Florence, Italy. Association for Computational Linguistics.