Abstractive Summarizers are Excellent Extractive Summarizers
Year: 2023

Abstract:
In this paper, we explore the efficacy of modeling extractive summarization with an abstractive summarization system. We propose three novel inference algorithms for sequence-tosequence models, evaluate them on established summarization benchmarks, and show that recent advancements in abstractive designs have enabled them to compete directly with extractive systems with custom extractive architectures. We show for the first time that a single model can simultaneously produce both stateof-the-art abstractive and extractive summaries, introducing a unified paradigm for summarization systems. Our results question fundamental concepts of extractive systems and pave the way for a new paradigm generative modeling for extractive summarization.1

heading: 1 Introduction
text: Extractive summarization selects a set of salient sentences from the original document(s) and composes them into a summary. Compared to abstractive summaries, made up of words or phrases that do not appear in the input document, extractive summaries are less flexible but avoid inconsistencies and hallucinations. The pipeline for building an extractive summarizer typically consists of two separate stages: sentence labeling and extractive modeling. Since few summarization datasets come with gold labels indicating which document sentences are summary-worthy, the first step is to create oracle sentence labels (Nallapati et al., 2017). The task is commonly modeled with a sequence labeling architecture (Cheng and Lapata, 2016) where a salience score is estimated for each document sentence, and top-ranked sentences are selected for summary inclusion. Recent work has expanded extractive modeling to higher-order sentence selection to account for complex label
1We distribute the code to replicate the results presented in the paper at https://github.com/danielvarab/GenX.
dependencies, via extracting sentences stepwise (Narayan et al., 2020), or reranking a small set of summary candidates (Zhong et al., 2020; An et al., 2022).
In this work, we revisit these fundamental concepts in extractive summarization. Specifically, we highlight that heuristically-derived sentence labels can be highly suboptimal (Narayan et al., 2018b; Xu and Lapata, 2022b), and that customized neural architectures for extractive modeling prevent taking advantage of independent improvements. We recognize that generative modeling with a neural encoder-decoder architecture (Bahdanau et al., 2015; Sutskever et al., 2014), the de facto choice for abstractive summarization (Nallapati et al., 2017; Zhang et al., 2020; Lewis et al., 2020), constitutes a promising direction for extractive summarization. In particular, such models learn directly from abstractive references and do, therefore, not require sentence labeling, while also embodying the extractive capabilities previously enabled by specialized neural architectures. Existing literature
330
has established varied and many connections between abstractive and extractive modeling such as copy mechanism (See et al., 2017), content selection (Kedzie et al., 2018; Gehrmann et al., 2018), and generation guidance (Dou et al., 2021). These connections, however, are mostly abstract-centric which are identified or constructed to improve abstractive summarization. In contrast, there are few studies from an extract-centric point of view.
In this work, we propose a new summarization paradigm that unifies extractive and abstractive summarization with generative modeling, without compromising abstractive performance. To this end, we treat extractive summarization as an inference-time task and explore methods for adapting a pre-trained abstractive system for extractive summarization without further optimization. We hypothesize that an abstractive system can be used as a summary evaluator for not only abstracts but extracts as well. A model optimized on abstractve references should be able to provide an accurate quality estimation for an extractive candidate summary when conditioned on the input document. A straightforward approach to validate this assumption is to search for the best document extract with an abstractive model for candidate evaluation. However, performing an exhaustive search over a combinatorial space of all eligible summary candidates is computationally intractable. To tackle this challenge, we propose GenX, Generative eXtractive summarization, which introduces a set of inference algorithms (shown in Figure 1) to reduce the search complexity via various approximations of the entire search space, at either sentence- or summary-level.
Experiments show that GenX achieves competitive or superior performance compared to custom systems developed for extractive summarization on the CNN/DM benchmark without compromising its ability to generate abstracts. Particularly, for one-stage summarization the proposed method shows superior results to custom extractive stateof-the-art systems. GenX also exhibits high robustness in zero-shot transfer: on XSum, its zero-shot performance surprisingly surpasses its fully supervised counterpart. We further conduct an extensive analysis of GenX’s properties, providing potential directions for future research on generative modeling for extractive summarization.

heading: 2 Generative Modeling for Extracts
text: Given a generative model θ trained on summarization data comprising documents and abstractive references, at inference time, for an input document D and a summary sequence Y , we estimate the length-normalized log probability of Y , following the standard practice in neural text generation (Cho et al., 2014):
pθ(Y |D) = 1
|Y |
|Y |∑
t=1
log pθ(Yt|D,Y<t) (1)
As θ is optimized at the token level, we evaluate both complete and partial summaries with pθ(Y |D).
The candidate summary space for a document D = {si}ni=1 of n sentences is combinatorial, consisting of |C(D)| = C ( m n ) candidate summaries of length m. To sidestep the computational intractability, we introduce three inference algorithms that reduce the search complexity via approximations. The first two (ranking and reranking) construct a candidate summary set, using either a discriminative or generative model (see Figure 1(a)), while the last approach searches directly over the partial summary candidate space (see Figure 1(b)).
Generative Ranking We employ a pre-trained generative model at both sentence- and summarylevel for hierarchical ranking. Specifically, we input each document sentence s into a generator and evaluate its summary-worthiness independently via its likelihood. We then rank all document sentences, and any subset of size m of the top-k sentences is considered as a candidate summary c. The sequence-to-sequence generator then evaluates and ranks all candidate summaries, and the highestranked one is selected as the extractive hypothesis:
y = argmax c⊆top-k pθ(s|D)
pθ (⊕(c)|D) (2)
where ⊕ concatenates the selected document sentences in c, ordered by their rank.
Generative Reranking Instead of using the same generative model for both sentence and summary evaluation, we assume access to an existing discriminative model pϕ(s|D) for sentence evaluation and ranking. Following Zhong et al. (2020), we adopt BERTSUMEXT (Liu and Lapata, 2019) to score each document sentence and then build candidate summaries as the combinations of top-scoring
sentences. In this case, the role of generative modeling is a summary-level reranker pθ(⊕(c)|D). Generative Search Instead of ranking, we consider constructing a summary by searching directly over the sentence space, i.e., without first composing candidate summaries from the input document. We propose a novel search algorithm that autoregressively selects a sentence until a stopping criterion is satisfied. Specifically, at each search step t, we evaluate and select a sentence as:
yt = argmax s∈D
pθ(y<t ⊕ s|D) (3)
where ⊕ concatenates the selected sentences y<t and a candidate sentence s. The selected sentence yt is then concatenated with y<t to form the selection history for the next step, as shown in Figure 1(c). We follow common practice in non-autoregressive extractive summarization (Liu and Lapata, 2019; Zhong et al., 2020) and assume a fixed number of sentences in the summary hypothesis, leading to a fixed number of search steps. Narayan et al. (2020) introduced a stepwise model which employs a special stop-token where the search stops when the token is generated. To explore this we additionally experiment with a dynamic stopping criterion where search over sentences continues until the end of the sequence token, EOS , provides a higher summary likelihood
than adding an additional sentence:
s.t. max s∈D pθ(y<t⊕ s|D) > pθ(y<t⊕ EOS|D). (4)

heading: 3 Experimental Setup
text: We perform supervised experiments on CNN/DM (Hermann et al., 2015) and zero-shot experiments on XSum (Narayan et al., 2018a). We evaluate summaries with ROUGE (Lin and Hovy, 2003). Details for our experimental settings and datasets can be found in Appendix A.
As there is no established baseline for extractive summarization with generative modeling, we construct Posthoc Rank, a posthoc method for direct comparison with GenX. The baseline first generates an abstract using the abstractive model. Then, the generated abstract is used to query document sentences and m sentences are retrieved with BM25 as the summary while applying tri-gram blocking.

heading: 4 Results
text: Supervised Summarization Table 1 shows the results of various systems trained and evaluated on CNN/DM. The first block presents the performance of the Lead-3 baseline which considers the first 3 sentences in a document as the summary and an Oracle baseline which serves as an upper bound.
The second block reports the performance of one-stage summarization systems. Stepwise ECTSum (Narayan et al., 2020) is a state-of-the-art autoregressive system that learns to score partial summaries by selecting which sentence is a summary sentence iteratively. Different from GenX, it is a highly-customized extractive architecture optimized with extractive oracle summaries. As can be seen, GenX performs on par with Stepwise ETCSum, and outperforms BERTSumExt (Liu and Lapata, 2019) and RoBERTaSumExt (Narayan et al., 2020). Two popular extractive systems based on sequence labeling.
The third block presents the results of two-stage systems. TRB denotes an additional stage for sentence selection with Trigram Blocking, an effective method for reducing redundancy. MatchSum (Zhong et al., 2020) is a state-of-the-art extractive system that takes top-ranked sentences from BERTSumExt and then re-ranks the summary candidates composed by them with a model based on a Siamese-BERT architecture. As can be seen, GenX models improve over the one-stage BERTSumExt and RoBERTaSumExt, i.e., with or without BERTSumExt as a sentence-level ranker. Its reranking variant also outperforms BERTSumExt+TRB and RoBERTaSumExt+TRB, showing that generative summary-level evaluation is more effective than heuristically-derived selection criteria. Note, the performance of GenX still falls short of state-ofthe-art MatchSum. This is all achieved while the design allows the base generative model to retain its ability to produce abstractive summaries. This is not applicable to any existing extractive systems except Posthoc Rank, which shows significantly inferior performance.
Zero-Shot Summarization We also examine the generalization capability of extractive systems in a zero-shot setting.2 As shown in Table 2, GenX generalizes to a different dataset robustly, outperforming strong one- and two-stage systems. It is generally perceived that a model’s zero-shot performance is inferior to the supervised performance. Surprisingly, GenX performs substantially better in the zero-shot setting than its supervised counterpart. One potential reason for this is that despite the discrepancy between training and inference, CNN/DM is a more extractive dataset than XSum (Liu and Lapata, 2019), and therefore contains more extract-specific knowledge. Compared to existing systems, GenX is more capable of transferring the extractive ability learned from CNN/DM to XSum. This shows that treating extractive summarization as an inference task can significantly
2We do not include zero-shot results of Stepwise ETCSum as there are no publicly available code or models.
reduce the risk of overfitting to one specific dataset, shedding light on a new direction for knowledge transferring in zero-shot summarization.

heading: 5 Ablation Study
text: We further assessed GenX with an ablation study. Replacing BRIO (trained with MLE and Contrastive Loss) with Bart (trained with MLE) leads to the largest performance drop. With the augmentation of contrastive learning, the abstractive system is competent in the dual role of both a generation and evaluation model, emphasizing the importance of calibrating a generative model on its summarylevel probability, even for its extractive inference.
The dynamic stopping mechanism introduced in Equation (4) performs on par with fixed-step search, showing that learning directly from abstracts is a promising way to teach models when to stop for summary extraction. GenX is also shown to be able to search for extractive summaries of less redundancy as its performance can not be further improved by incorporating Trigram Blocking.
6 Efficiency
We have shown that abstractive systems are capable extractive summarizers, however, it is important to highlight that the proposed method exhibits different computational requirements than that of contemporary extractive designs. Unlike extractive designs that compute a single score for a candidate sentence or summary (via a classification token), abstractive systems produce scores for all individual tokens in a candidate summary3. Computing these extra tokens causes approaches such as ranking and reranking with GenX more computationally demanding. However, when combined with search GenX stands as an efficient solution to searching through an otherwise intractable candidate summary space. This is enabled by an abstractive system’s ability to sequentially score text (see Equation 1) and boils down to the complexity of beam search. This is a clear improvement in computational efficiency over systems like MatchSum which only supports scoring complete summaries and must exhaustively recompute different permutations in the candidate summary spaces. To make this strategy computationally tractable these models resort to heavy pruning which limits the
3For the sake of generality we ignore computational costs related to encoding as this varies across models but emphasize that it can have sizable practical implications.
expressiveness that high-order modeling otherwise enables.

heading: 7 Related Work
text: There is a plethora of work on controlling different aspects of summarization, from content (Xu and Lapata, 2022a; Ahuja et al., 2022) to formats (Zhong et al., 2022). In this work, we offer efficient and effective control over the summary type (extract versus abstract) during inference. Recent work also investigates how to treat discriminative tasks such as information extraction and retrieval with generative modeling and its effectiveness for entities (De Cao et al., 2020) and string identifiers (Bevilacqua et al., 2022). Others have suggested delegating extractive inference to the encoder of a generative model (An et al., 2022). Despite the resemblances, extractive summarization with generative modeling remains under-explored and stands as a promising research direction with the surge of innovations in large language models.

heading: 8 Conclusion
text: In this paper, we explored the possibility of modeling extractive summarization with an abstractive system. We proposed three novel inference algorithms which allow an abstractive model to perform the extractive task. Our results showed that not only is extractive summarization feasible, but recent systems are directly competitive with contemporary extractive systems. This work shows that extractive and abstractive paradigms can be unified through a sequence-to-sequence design, removing the need for oracle summary labels and custom extractive model architectures.

heading: 9 Limitations
text: One potential way to improve the extractive performance of a generative system is to explicitly model the likelihood of extracts during training. Driven by this intuition, we investigate creating a mixture of extractive and abstractive candidates for contrastive learning in BRIO. Specifically, we obtain extractive candidates with beam labeling proposed in Xu and Lapata (2022b), while the abstractive ones are from the original BRIO training data. Nevertheless, as we can see, this mixing method hurts both BRIO’s extractive and abstractive performance. However, it is noteworthy that extractive summary is important in a wider context, as shown in Section 4: reference summaries
in CNN/DM are highly extractive and optimizing a model on these summaries therefore may have provided it with the task instruction needed for extractive summarization, albeit implicitly. We leave the study of a more effective extract-aware learning strategy for future study.
Furthermore, we emphasize that the conclusions drawn in this paper are based on results produced on English datasets from the news domain. Even though these datasets are established benchmark datasets for summarization it is imaginable that other domains and languages may have produced different evidence. Despite this, the results remain insightful as the results show that extractive summarization is in fact feasible with modern abstractive systems. In future research, we look forward to shedding light on the possibilities and limitations of the proposed methods in a broader context.

heading: B License Information
text: The datasets used in this work, CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018a), are both released under the MIT License.
C System Output
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 9.
7 A2. Did you discuss any potential risks of your work? Section 9.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3-4 and Appendix A-B.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix B.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We used existing benchmarks as they are for fair comparisons.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.
C 3 Did you run computational experiments? Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the study of a more effective extract-aware learning strategy for future study