SlowGuess/ACL_2020_OCR huggingface dataset:
We implement a guard rail to penalize the model when frame-filling patterns are observed. During training, we keep track of the last 100 summaries produced by the model. We then aggregate the frequency of words for each word position in the 100 summaries. If any word appears more than  $50\%$  of the time at a specific word position, we raise the "No-frame-filling" penalty. In the example given above, the word "talks" appeared in the second word position in more than  $50\%$  of the summaries, as well as the word "about" in the fifth position.
These rule-based training guard rails are simple and effective. In our finalized trained models, very few summaries exhibit penalized behavior:  $2\%$  for no-repetition,  $5\%$  for finish-your-sentence, and  $2.5\%$  for no-frame-filling.
We present results for Summary Loop models trained in the news domain under three different length constraints: 10, 24, and 46 words, matching the distributions of the Headline, Newsroom
Method	R-1	R-2	R-L	Coverage Score	Fluency Score	Brevity (avg words)
Baselines
Human-written Summaries	100	100	100	0.392	0.612	58.5
X Lead-3 baseline	40.3	17.7	36.6	0.421	0.656	84.0
Supervised Methods
Pointer Generator (See et al., 2017)	36.4	15.7	33.4	0.342	0.547	55.6
PG + Coverage (See et al., 2017)	39.5	17.3	36.4	0.377	0.508	61.7
Bottom-Up (Gehrmann et al., 2018)	41.2	18.7	38.3	0.378	0.538	73.9
PEGASUSBASE (Zhang et al., 2019a)	41.8	18.8	38.9	-	-	-
PEGASUSLARGE (Zhang et al., 2019a)	44.1	21.3	40.9	-	-	-
Unsupervised Methods
X TextRank (Mihalcea and Tarau, 2004)	35.2	12.9	28.7	0.370	0.612	49.62
GPT2 Zero-Shot (Radford et al., 2019)	29.3	8.3	26.6	-	-	-
Summary Loop 45	37.7	14.8	34.7	0.404	0.627	47.0
Table 2: ROUGE Results (F-1) on the non-anonymized CNN/DM test-set for supervised and unsupervised methods. Extractive methods indicated with X. Our ROUGE scores have a  $95\%$  confidence interval of at most  $\pm 0.30$ . Coverage, Fluency and Brevity (average number of words) included for systems where summaries are available, using Coverage and Fluency models from our work.
Supervised Methods	R-1	R-2	R-L
X Lead-3 baseline	32.0	21.1	29.6
PG + Coverage	27.5	13.3	23.5
Unsupervised Methods	R-1	R-2	R-L
X TextRank	24.5	10.1	20.1
Summary Loop 24	27.0	9.6	26.4
Table 3: ROUGE Results on the released test set of Newsroom. X indicate extractive methods. Summary Loop outperforms other unsupervised method, is competitive with supervised Pointer-Generator.
(Grusky et al., 2018) and CNN/DM (Nallapati et al., 2016) datasets. We compare our summaries using the standard ROUGE metric, and by analyzing summaries for the errors made, the technique used and the level of abstraction. Finally, we show the Summary Loop can be complemented with supervision, reducing the amount of data needed to achieve comparable ROUGE results.
5.1 News ROUGE Scores
Table 2 and Table 3 present ROUGE results on the CNN/DM and Newsroom datasets respectively. In both cases, Summary Loop outperforms other unsupervised methods, and is competitive with supervised methods despite not being exposed to any example summaries. On CNN/DM, Summary Loop performs in between the Pointer Generator and Bottom Up architecture in terms of ROUGE-1. On the Newsroom, Summary Loop is within 0.6 ROUGE
1 points of the Pointer-Generator with Coverage and surpasses it by 2 ROUGE-L points.
Recent breakthroughs in pretrained Transformer models have shown that using larger models in Summarization can lead to large improvements. For instance, a "large" version of the PEGASUS model (Zhang et al., 2019a) outperforms the "base" version by 2.3 ROUGE-1 points. Because Summary Loop experiments were performed using "base" models, we expect that using larger Transformer models could lead to similar gains.
Table 2 confirms that human-written summaries obtain amongst the highest Fluency and Coverage scores. Human-written summaries are only outperformed by Summary Loop summaries, and the Lede-3 baseline. However, the Summary Loop summaries are obtained by directly optimizing for Fluency and Coverage, and Lede-3 baseline summaries achieve their higher Coverage at the expense of being much longer (i.e. 84 words on average compared to 58 in human-written summaries).
5.2 Technique and Error Analysis
We perform a manual analysis of 200 randomly-selected summaries on the test set of CNN/DM from the Pointer-Generator with Coverage (PGC), Bottom-Up (BU) and the unsupervised Summary Loop (SL). We annotated each summary with two types of errors: Inaccurate (information in summary contradicts document), Ungrammatical (one sentence or more is not properly constructed), and
Error Made	PGC	BU	SL
Inaccurate (%)	11	31	24
Ungrammatical (%)	7	15	18
Technique Used (Success/Total)	PGC (S/T)	BU (S/T)	SL (S/T)
Sent. Compression	86 / 110	96 / 177	118 / 194
Sent. Merging	13 / 27	29 / 65	71 / 121
Novel Sentence	0 / 1	4 / 18	33 / 70
Entity Manipulation	7 / 10	15 / 27	27 / 40
Total Technique	106 / 148	144 / 287	249 / 425
Table 4: Error and Technique analysis on 200 randomly selected summaries on the CNN/DM test-set for the Point-Gen with Cov. (PGC), Bottom-Up (BU) and unsupervised Summary Loop (SL). For each summarization technique, we report two numbers: the number of successful occurrences in summaries with no error, and the total number of occurrences in the 200 summaries.
four summarization techniques: Sentence Compression (summary sentence is a document sentence with words removed), Sentence Merging (2 or more document sentences are merged into a summary sentence), Novel Sentence (original sentence in the summary), and Entity Manipulation (a named entity is modified or simplified, e.g. changing a full name to a last name). We present Summary Loop examples illustrating each error and technique in Figures A3 - A8.
The analysis was performed by the first author of the paper, labeling article/summary pairs without knowledge of model origin. A summary can manifest any number of summarization Techniques, or none. Labeling is binary: if a summary exhibits more than one or instances of a Technique, it receives a 1, otherwise it receives a 0. Results of the analysis are summarized in Table 4.
SL uses significantly more summarization techniques (425) than PGC (148) and BU (287) summaries. Beyond raw counts, SL is more successful at applying summarization techniques (59% success) than BU (50% success), but less successful than PGC (72%). Note however that PGC takes little risk: 19% of the summaries go beyond sentence compression, and 39% are extractive, using none of the summarization techniques.
5.3 Level of Abstraction
All methods generating summaries one word at a time have potential for abstraction. In Figure 4 we analyze human and system written summaries for abstraction level. We measure a summary's level of abstraction by looking at the length of spans
Span	Gold	Summary Loop	Bottom Up	PG + Cov
Novel	9.8%	0.6%	1.3%	1.0%
Length 1	23.6%	6.0%	2.9%	1.4%
Length 2	20.8%	11.4%	4.5%	1.1%
Length 3-5	24.7%	26.4%	13.0%	3.6%
Length 6-10	12.0%	29.7%	21.1%	9.3%
Length 11+	9.1%	25.9%	57.2%	83.6%
Avg. Length	4.2	7.8	14.8	25.2
Figure 4: Histogram and average copied span lengths for abstractive summaries. A summary is composed of novel words and word spans of various lengths copied from the document. Summary Loop summaries copy shorter spans than prior automatic systems, but do not reach abstraction levels of human-written summaries.
Initialization Method	R-1	R-2	R-L	Test Loss
28k samples from CNN/DM (10%)
Random Initialization	7.0	0.9	8.8	6.05
GPT2	37.1	15.9	31.9	2.21
Summary Loop S10	38.7	16.2	35.1	2.07
All of CNN/DN (100%)
Random Weights	20.4	4.1	19.1	4.22
GPT2	38.4	17.2	35.0	2.02
Summary Loop S100	41.0	18.1	37.3	1.89
Table 5: ROUGE Results on the CNN/DM test-set for supervised generative Transformers. Initializing with the unsupervised Summary Loop outperforms random and GPT2 initializations.
copied from the document. Summary Loop is the most abstractive automated method, although less so than human written summaries. SL cuts nearly in half the length of copied spans compared to other automated methods.
5.4 Supervision is not the enemy
If summaries are available, we show that they can complement the unsupervised Summary Loop. We run supervised experiments on CNN/DM using a generative Transformer architecture and varying the initialization. We compare initializing with (1) random weights, (2) the original GPT2 weights, and (3) the Summary Loop weights of target length 45. We train each model with teacher forcing, comparing using the entire CNN/DM training set to just  $10\%$  of it. The results are summarized in Table 5.
First, initializing with the Summary Loop leads to higher ROUGE score both in the  $10\%$  and full dataset setting. As expected, results improve when using the entirety of the data, and the Summary Loop initialized model trained with the entirety of CNN/DM obtains a ROUGE-1 F1-score of 41.0,
within the confidence interval of the supervised Bottom Up (Gehrmann et al., 2018) architecture. This is a strong result as the Transformer we use is a generic language model, and is not specialized for summarization.
Second, initializing with Summary Loop and training with  $10\%$  of CNN/DM yields comparable ROUGE scores to initializing with GPT2 and using the entire CNN/DM, showing that Summary Loop can be useful when fewer summaries are available.
Customizing summaries. In Figure 1, we illustrate the effect of the length constraint by summarizing the same document under three different length constraints. Each model adapts to its word budget. However, length is only one way to customize summaries. One might want to summarize based on point of view, chronology, theme, etc.
Fluency vs. Grammaticality. By choosing to represent the validity of summaries with a Language model, we encourage fluent summaries (i.e., with likely sequences of words) but not necessarily grammatical ones. Extending the scoring to include grammaticality, either by using a parsing model, or leveraging the Corpus of Linguistic Acceptability (Warstadt et al., 2019) could prove useful.
Summarization in the wild. Because our method is unsupervised, it can be applied to new domains and languages. In this work, we benefited from pretrained BERT and GPT2 models in English, which do not yet exist publicly for other languages. Once they become available in other languages, the Summary Loop can be ported over.
Abstraction dangers. Recent work around measuring factuality in generated text, using Natural Language Inference (Guo et al., 2018) or rule-based fact extraction (Zhang et al., 2019b) becomes increasingly important with summaries that are more abstractive. This work can be naturally included into the Summary Loop, with a fact-checker model generating an accuracy score.
In this work we present a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. When tested on common news summarization datasets, our method significantly outperforms previous unsupervised methods, and gets within the range of competitive supervised
methods. Our models attain levels of abstraction closer to human-written summaries, although with more abstraction, more potential for factual inaccuracies arise.
We would like to thank Forrest Huang, David Chan, Roshan Rao, Katie Stasaski and the ACL reviewers for their helpful comments. This work was supported by the first author's internship at Bloomberg, and a Bloomberg Data Science grant. We also gratefully acknowledge support received from an Amazon Web Services Machine Learning Research Award and an NVIDIA Corporation GPU grant.
Kristjan Arumae and Fei Liu. 2018. Reinforced extractive summarization with question-focused rewards. In Proceedings of ACL 2018, Student Research Workshop, pages 105-111.
Federico Barrios, Federico López, Luis Argerich, and Rosita Wachenchauzer. 2015. Variations of the similarity function of textrank for automated summarization. In Argentine Symposium on Artificial Intelligence (ASAI 2015)-JAIIO 44 (Rosario, 2015).
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686.
Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, XianLing Mao, and Heyan Huang. 2019. Cross-lingual natural language generation via pre-training. arXiv preprint arXiv:1909.10481.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4052-4059.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3938-3948.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719.
Min Gui, Junfeng Tian, Rui Wang, and Zhenglu Yang. 2019. Attention optimization for abstractive document summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1222-1228.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687-697.
Philippe Laban and Marti A Hearst. 2017. newslens: building and visualizing long-ranging news stories. In Proceedings of the Events and Stories in the News Workshop, pages 1-9.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ca glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL 2016, page 280.
Nikola I Nikolov and Richard HR Hahnloser. 2019. Abstractive document summarization without parallel data. arXiv preprint arXiv:1907.12951.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multireward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of ICLR.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789.
Juan Enrique Ramos. 2003. Using tfidf to determine word relevance in document queries.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3237-3247.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1073-1083.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Wenbo Wang, Yang Gao, He-Yan Huang, and Yuxiang Zhou. 2019. Concept pointer network for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3067-3076.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. 2019. Bottlesum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3743-3752.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777.
Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2019b. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. arXiv preprint arXiv:1911.02541.
A Masking Procedure Details
The masking procedure follows these steps:
1. We randomly sample 5,000 documents in the domain being summarized (e.g. News) as a training corpus,
2. The training corpus is tokenized using the tokenizer of the Coverage model. In our case, we tokenize with the Word Piece model of the BERT Base model (Devlin et al., 2019),
3. We train a tfidf transformation model using the tokenized training corpus using default parameters of scikit-learn's tfidf implementation (Pedregosa et al., 2011),
4. Given a document to be masked, we use the trained tfidf model to produce a tfidf for the document,
5. The words present in the document are ranked in decreasing order of tfidf score, and the  $k$  words with highest tfidf form the masking set,
6. All occurrences of the words in the masking set are replaced by a mask in the document, creating the masked document.
B Fluency Examples
Table A1 provides examples from the Headline dataset of sampled headlines and their corresponding Fluency Score. The Fluency Score, a normalized language model log-perplexity, ranges from 0 to 1. Even though all these headlines are written by a human, the Fluency scores vary, with the higher-scoring headlines using more standard grammatical constructs. Note that the use of complex entity names does not prevent the model from obtaining a high Fluency score.
Example Headline	Fluency Score
Henry's Monaco recruit giant Brazil-ian Naldo for relegation scrap	0.16
Tesla shares dive after price cut, production numbers	0.41
French police arrest gilets jaunes protests leader Eric Drouet	0.59
Carlos Ghosn will appear in public for the first time since his arrest	0.75
Table A1: Example selected headlines and their Fluency score. The headlines were picked from a corpus of human-written news headlines. The average Fluency in the corpus is 0.479.
C Model Size and Initialization
Figure A1 shows the model size and initialization model used for each of the Summarizer, Coverage and Fluency models.
Summarizer Architecture
GPT2-base: 12-layer, 768-hidden, 12-heads
Summarizer Initialization
GPT2 base model from Radford et al. (2019)
Coverage Architecture
BERT-base: 12-layer, 768-hidden, 12-heads
Coverage Initialization
Pretrained model obtained in Section 3.4.2
Fluency Architecture
Fluency Initialization
GPT2 base model from (Radford et al., 2019), finetuned with Language modeling on news text.
Figure A1: The model size choice as well as initialization method for the Summarizer, Coverage and Fluency models in the Summary Loop. Each model leverages a pretrained Transformer.
D Training Plots
Figure A2 presents the plots of key variables we obtain during the training of the length 10 Summary Loop model. The training occurred over 10 days using a single Titan X GPU. During a first phase which occurs in the first 2 days of training, the model learns to copy content from the news article, which helps it achieve high Fluency and Coverage. In a second phase starting around the second day, the Summarizer learns to gain Coverage which maintaining Fluency mostly constant, which makes the overall Summary Score rise. The Summarizer model quickly learns to use its word budget, and after 10 days of training, the model uses an average of 9.7 words in its summaries.
E Example Annotated Summaries
Figures A3, A4, A5, A6, A7, and A8 show example documents and the generated Summary Loop summary from the error and technique analysis of Section 5.2. Each summary manifests a summarization technique or error observed.
5146
(a) Fluency Score
(b) Coverage Score
(c) Summary Score
(d) Average number of words in summary
Figure A2: Plots of key variables during the training of the length 10 Summary Loop: (a) is a plot of the average Fluency Score, (b) is a plot of the average normalized Coverage Score, (c) is a plot of the average Summary Score (taking guard-rails into account), and (d) is a plot of the average number of words in summaries produced.
Sentence Compression Example
Document: He has long struggled to convince voters that he is a suitable choice for prime minister. Now Ed Miliband has hired a leadership coaching firm that helps people overcome anxiety and find their "inner voice". The consultants drafted in by the Labour leader claim to work with politicians to build "leadership skills" using "neuroscience" and "business psychology". Ed Miliband, pictured, has hired a US guru who can help him convince himself that he can be Prime Minister. [...]
Summary: Ed Miliband has hired a US guru who can help politicians on their leadership skills using neuroscience. Mr Miliband has hired the firm that can help politicians to build their leadership skills. The consultants drafted in by the Labour leader claim to work with politicians.
Figure A3: Summary Loop summary from the Error and Technique analysis (Section 5.2) illustrating the Sentence Compression technique. The blue boldface highlight is an example of sentence compression.
Sentence Merging Example