SlowGuess/NAACL_2021_OCR huggingface dataset:
Finding a suitable length for the summary is one of the most important open problems in automatic summarization (Shapira et al., 2018; Sun et al., 2019). A summary should be shorter than the original, but long enough to include the most important information. Length normalization seeks to rescale the log-likelihood score of a summary, denoted by  $S(\mathbf{x},\mathbf{y}) = \log p_{\theta}(\mathbf{y}|\mathbf{x})$ , by its length  $|\mathbf{y}|$ , with an exponent  $p$  (Eq. (3)). It is used by some neural abstractive summarizers (See et al., 2017; Lewis et al., 2020). However, the method does not consider the density of information in the source text and it may still generate ultra-short summaries.
$$
\mathcal {S} _ {l n} (\mathbf {x}, \mathbf {y}) = \mathcal {S} (\mathbf {x}, \mathbf {y}) / | \mathbf {y} | ^ {p} \tag {3}
$$
Instead, we attempt to estimate the appropriate length of the summary given a source text, denoted by  $\mathcal{L}_{\mathrm{pred}}$ , and reward a system summary if it stays close to the estimated length (Huang et al., 2017). Concretely, we assign a per-word reward to the summary, represented by  $r\min(|\mathbf{y}|, \mathcal{L}_{\mathrm{pred}})$  (Eq. (4)). A system summary continues to be rewarded until it
System	R-1	R-2	R-L
Ivt2k-1sent (Nallapati et al., 2016)	32.67	15.59	30.64
SEASS (Zhou et al., 2017)	36.15	17.54	33.63
DRGD (Li et al., 2017)	36.27	17.57	33.62
Pointer-Gen (See et al., 2017)	34.19	16.92	31.81
R3Sum (Cao et al., 2018)	37.04	19.03	34.46
EntailGen (Guo et al., 2018)	35.98	17.76	33.63
BiSET (Wang et al., 2019)	38.45	19.53	36.04
MASS (Song et al., 2019)	38.73	19.71	35.96
UniLM (Dong et al., 2019a)	38.90	20.05	36.00
PEGASUS (Zhang et al., 2020a)	39.12	19.86	36.24
Ours (AVERAGE)	35.51	16.33	32.75
Ours (BEST QUALITY)	36.71	17.27	33.63
Ours (BEST SUMMARY LENGTH)	39.27	20.40	36.76
Table 5: Results on the Gigaword test set evaluated by ROUGE (Lin, 2004).<sup>2</sup>
reaches the predicted length  $(|\mathbf{y}| \leq \mathcal{L}_{\mathrm{pred}})$ . Beyond that, increasing the length of the summary does not lead to additional rewards. We obtain the predicted length  $\mathcal{L}_{\mathrm{pred}}$  using a baseline abstractive summarizer, which takes the source text as input and greedily decodes a summary in a left-to-right manner until an end-of-sequence symbol is predicted;  $\mathcal{L}_{\mathrm{pred}}$  is the length of the decoding sequence.  $r$  is a coefficient to scale the reward and it is tuned on the validation data. Finally, the reward-augmented log-likelihood  $S_{rwd}(\mathbf{x}, \mathbf{y})$  is used as a scoring function to rank all summary hypotheses of varying lengths.
$$
\mathcal {S} _ {r w d} (\mathbf {x}, \mathbf {y}) = \mathcal {S} (\mathbf {x}, \mathbf {y}) + r \min  (| \mathbf {y} |, \mathcal {L} _ {\text {p r e d}}) \tag {4}
$$
Datasets We perform extensive experiments on Gigaword (Parker, 2011) and Newsroom (Grusky et al., 2018) datasets. The goal is to generate an abstractive summary from a lengthy source sentence. For each article, we pair its first sentence with the title to form a summarization instance. Both datasets contain large collections of news articles. Gigaword (1995-2010) contains 3,810,674 / 10,000 / 1,951 instances, respectively, in the train, validation and test splits. Newsroom (1998-2017) contains 199,341 / 21,530 / 21,377 instances, respectively. We conduct experiments on both datasets to demonstrate the generality of our two-staged strategy. Our method generates a diverse set of summaries from a source sentence in stage one, then score and select admissible summaries in stage two.
The system summaries are evaluated using both automatic metrics (ROUGE; Lin, 2004) and human evaluation of information coverage, grammaticality
2Our experiments are performed on the original Gigaword dataset (Parker, 2011) without anonymization. The data provided by Rush et al. (2015) replaced all digit characters with # and replaced word types seen less than 5 times with UNK.
Gigaword	Summary Length (L)	Best Quality	Best Length
7	8	9	10	11	12	13	14	15	16	Avg.
R-1 F1 (%)	32.01	35.42	37.05	37.95	38.05	37.79	37.27	36.66	35.75	35.13	36.31	36.71	39.27
R-2 F1 (%)	13.47	15.68	17.39	18.31	18.24	18.22	17.85	17.19	16.63	16.00	16.90	17.27	20.40
R-L F1 (%)	29.76	32.85	34.46	35.31	35.10	34.87	34.21	33.53	32.71	32.02	33.48	33.63	36.76
Newsroom	Summary Length (L)	Best Quality	Best Length
7	8	9	10	11	12	13	14	15	16	Avg.
R-1 F1 (%)	40.99	43.38	44.94	46.06	46.57	46.77	46.53	46.25	45.76	45.21	45.25	45.77	46.60
R-2 F1 (%)	19.15	20.99	22.11	23.02	23.47	23.59	23.38	23.15	22.79	22.33	22.40	22.58	23.85
R-L F1 (%)	38.24	40.34	41.56	42.36	42.69	42.68	42.31	41.88	41.29	40.63	41.40	41.48	43.07
Table 6: Results on Gigaword and Newsroom datasets where the generator produces summaries of varying lengths.
and faithfulness to the original text. We introduce a new dataset where a source sentence is associated with multiple summaries, and admissible ones are labelled by human annotators (§5.1). The dataset will serve as a useful testbed for future summarization research, where multiple reference summaries is key to improve the reliability of evaluation results (Louis and Nenkova, 2013). This paper focuses on generating abstractive summaries from single source sentences. However, we expect the insights gained from this study to inform the design of future summarizers of different kinds.
Experimental Setup Our generator is initialized with RoBERTa-Base (Liu et al., 2019) due to its high performance on generation-related tasks. We use Byte Pair Encoding (Sennrich et al., 2016) with a vocabulary of 50,265 tokens. The model contains 12 Transformer blocks (Vaswani et al., 2017), with a hidden size of 768 and 12 attention heads, for a total of 110M parameters. We fine-tune the model on the train split of Gigaword and Newsroom, respectively, before applying it to the test sets. The model is fine-tuned for 20 epochs. Each epoch contains  $24\mathrm{k} / 1.5\mathrm{k}$  batches and our batch size is 128. The model uses  $10\mathrm{k} / 1\mathrm{k}$  warm-up steps, respectively, for Gigaword and Newsroom. We use the AdamW (Loshchilov and Hutter, 2017) optimizer with an initial learning rate of 1e-4. The momentum parameters are set to 0.9 and 0.999. On a deep learning workstation equipped with 2x Titan RTX GPUs, our model takes 64 and 5.5 hours to fine-tune on Gigaword and Newsroom. At test time, our beam size is  $K = 20$ . The model produces summaries ranging from  $L = 7$  to 16 tokens for a given source sentence.
Our selector for best overall quality is trained using 1.8M instances automatically constructed from the train split of Gigaword. The set is balanced with an equal number of positive and negative instances. 226k instances are created with the type of Search and Replace, and 400k instances are cre
ated using each of the four remaining corruption types. The reward coefficient  $r$  is set to 2.0 across all experiments.
Automatic Evaluation In Table 6, we present results on Gigaword and Newsroom test sets evaluated by ROUGE (Lin, 2004). We report R-1, R-2 and R-L  $F_{1}$ -scores that respectively measure the overlap of unigrams, bigrams, and longest common subsequences between system and reference summaries. For each summarization instance, our generator produces multiple alternative summaries, ranging from  $L = 7$  to 16 tokens. E.g., "Daiwa Bank" corresponds to four tokens, 'Dai', 'wa', 'Bank' plus an ending period. Our BEST-QUALITY and BEST-LENGTH selectors each identifies a single best summary from the set of alternative summaries for each summarization instance.
We observe that the BEST-LENGTH selector has achieved the highest scores. It performs better than using any single target length for all summaries. Among summaries of different lengths, the highest R-2  $F_{1}$ -scores are obtained when the target summary length is set to 11 and 12 tokens, respectively, for Gigaword and Newsroom. This is close to the median length of reference summaries, which are 12 and 13 tokens for these datasets. Our findings show that, the target summary length can make a non-negligible impact on automatic evaluation results. It is best for system summaries to be long enough to include the most important information to achieve satisfying results.
In Table 5, we report results on the Gigaword test split that contains 1,951 instances. Our approach is compared against strong neural abstractive systems, including PEGASUS (Zhang et al., 2020a), UniLM (Dong et al., 2019a) and MASS (Song et al., 2019). These systems draw on large-scale unsupervised pretraining to improve the quality of summaries, yielding some of the best reported results. In comparison, our BEST-LENGTH selector either
Candidate Summary	Contains the main idea?	Is true-to- original?	Is grammatical?
(1) Izetbegovic blasts Karadzic	☐ Yes ☑ No	☐ Yes ☐ No	☐ Yes ☐ No
(2) Karadzic accused of swaying US Congress	☐ Yes ☐ No	☐ Yes ☐ No	☐ Yes ☐ No
(3) Karadzic seeks to sway US Congress	☐ Yes ☐ No	☐ Yes ☐ No	☐ Yes ☐ No
(4) Karadzic seeks to sway Congress	☐ Yes ☐ No	☐ Yes ☐ No	☐ Yes ☐ No
(5) Karadzic misleading US Congress	☐ Yes ☑ No	☐ Yes ☑ No	☐ Yes ☑ No
(6) Monday's international soccer scores	☐ Yes ☑ No	☐ Yes ☑ No	☐ Yes ☑ No
Source Text: Bosnian President Alija Izetbegovic on Monday accused Bosnian Serb leader Radovan Karadzic of seeking to sway the US Congress against approving US troops to help enforce peace in the former Yugoslavia.
Figure 2: Effectiveness of position-aware beam search (§3.1). A larger beam tends to give better results.
surpasses or performs comparably to these systems. The summaries selected by it achieve the highest R-2  $F_{1}$ -score of  $20.4\%$ . We further choose the summary that yields the highest score for each instance, creating an oracle set of summaries, which yield a R-2  $F_{1}$ -score of  $33.4\%$ . The results indicate that, with better summary selectors, there is a great potential that we can further boost summarization performance.
In Figure 2, we investigate the effectiveness of our position-aware beam search (§3.1). The beam size  $K$  is set to  $\{1,5,10,15,20\}$ . We report the average R-2  $\mathrm{F_1}$ -score across summaries of all lengths. Results show that our position-aware beam search is effective at decoding summaries and works robustly across a range of beam sizes. A larger beam  $(K = 20)$  tends to give better results.
Human Evaluation We are interested in a holistic evaluation of the multiple alternative summaries produced by the generator. To accomplish this, we develop a new dataset containing 500 summarization instances randomly sampled from the Gigaword test set. Our generator produces 7 alternative summaries for each instance, which have varying lengths that range from  $L = 7$  to 13 tokens. We recruit human evaluators to judge the quality of each summary given its source text.<sup>3</sup>
Table 7: Example annotation interface. A human annotator is instructed to read over the summaries before seeing the source text to effectively recognize any hallucinated content that is not found in the source text. A native English speaker creates annotations for multiple instances, which are shared with all annotators to provide guidance.
Content	Truthful	Grammatical	Overall
Average	80.7	82.6	96.5	74.2
Best Length	82.8	86.0	97.4	77.8
Best Quality	93.0	90.8	97.0	88.2
Table 8: Results of human assessment. BEST-QUALITY summaries have a higher likelihood of being admissible according to the criteria, suggesting the effectiveness of the method.
Our annotation interface is presented in Table 7. A human annotator is instructed to read over all summaries before seeing the source text. It allows him/her to effectively recognize any hallucinated content that is not found in the source text. The annotator is asked to answer three yes-no questions. They include (a) has the summary successfully convey the main points of the source text? (b) is the summary truthful to the meaning of the original? (c) is the summary grammatical? A native speaker creates gold-standard annotations for multiple instances, they are shared with all annotators to provide guidance. Our annotators are recruited using Appen (appen.com). It is a crowdsourcing platform similar to Amazon Mechanical Turk (mturk.com), but provides great quality control mechanisms to ensure high-quality work.
We recruit 5 annotators to judge the quality of each summary. A summary is deemed admissible under a criterion if the majority answer is yes. We observe that,  $74.2\%$  of summaries produced by our generator are admissible under all three criteria. The results suggest that our generator is able to produce multiple, equally valid summaries for a given source text. We additionally examine the percentage of admissible summaries under each criterion, results are shown in Table 8. Grammaticality has the best performance  $(96.5\%)$ , followed by truthfulness  $(82.6\%)$  and content coverage  $(80.7\%)$ . There appears to be room for improvement for the latter two aspects. Moreover, the summaries chosen by our BEST-QUALITY selector demonstrate a high ad
Our annotated dataset is available on Github at https://github.com/ucfnlp/varying-length-summ
missible rate— $93\%$ ,  $90.8\%$  and  $97\%$ —respectively for the three criteria, suggesting the effectiveness of the selector. Further, we observe a discrepancy between ROUGE and human judgments (Fabbri et al., 2020) as summaries yielding highest ROUGE scores are not always deemed admissible by human evaluators. We hope this dataset provides a testbed for future summarizers to be judged on their ability to produce multiple summaries per instance rather than a single summary.
In Table 3, we show example system summaries and the order in which summary tokens are produced. E.g.,  $\{2,5\}$  indicate the two tokens "Bo-J" (Bank of Japan) are generated the 2nd and 5th place in the summary. We find that our generator can effectively decide what content should be included in the summary given the limited space available, yielding summaries with varying levels of details. Important spans such as "calls for calm" tend to be generated first, less vital ones later. Our findings corroborate the hypothesis that a masked language model may enable generation in a flexible word order (Liao et al., 2020). Further, we observe that the order in which tokens are generated is related to their dependencies ("call→for"), which supports the findings of Clark et al. (2019).
We investigate a new approach to neural abstractive summarization that focuses on producing multiple summary hypotheses with varying lengths and levels of details. Our selectors are designed to identify summaries that have the optimal length and the best overall quality. The approach obtains state-of-the-art results on summarization benchmarks and opens up a potential new avenue for customizing summary selectors to suit users' needs.
Future work includes extending this research to long documents. Our confidence-driven generator and the selectors could potentially be extended to operate on spans of text (Joshi et al., 2020) rather than individual tokens, thus allowing for efficient generation of multiple summary hypotheses and identification of admissible summaries and/or summary segments.
We are grateful to the reviewers for their insightful comments, which have helped us improve the paper. This research was supported in part by the National Science Foundation grant IIS-1909603.
Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020. Few-shot learning for opinion summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4119-4135, Online. Association for Computational Linguistics.
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 152-161, Melbourne, Australia. Association for Computational Linguistics.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017. Faithful to the original: Fact aware neural abstractive summarization. CoRR, abs/1711.04434.
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1662-1675, New Orleans, Louisiana. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Yufei Chen, Sheng Huang, Fang Wang, Junjie Cao, Weiwei Sun, and Xiaojun Wan. 2018. Neural maximum subgraph parsing for cross-domain semantic dependency analysis. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 562-572, Brussels, Belgium. Association for Computational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Gonçalo M. Correia and André F. T. Martins. 2019. A simple and effective approach to automatic post-editing with transfer learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3050-3056, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019a. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13063-13075. Curran Associates, Inc.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019b. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393-3402, Florence, Italy. Association for Computational Linguistics.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320-9331, Online. Association for Computational Linguistics.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45-54, Melbourne, Australia. Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 11181-11191. Curran Associates, Inc.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687-697, Melbourne, Australia. Association for Computational Linguistics.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Controllable text generation. CoRR, abs/1703.00955.
Liang Huang, Kai Zhao, and Mingbo Ma. 2017. When to finish? optimal beam search for neural text generation (modulo beam size). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2134-2139, Copenhagen, Denmark. Association for Computational Linguistics.