CTRLsum: Towards Generic Controllable Text Summarization

1 INTRODUCTION . Neural summarization systems aim to compress a document into a short paragraph or sentence while preserving key information . There are largely two categories of summarization systems : extractive summarization that extracts important portions of a document ( Cheng & Lapata , 2016 ; Nallapati et al. , 2017 ; Narayan et al. , 2018 ) , and abstractive summarization that freely generates novel sentences ( Rush et al. , 2015 ; See et al. , 2017 ; Paulus et al. , 2018 ) which can produce coherent and fluent summaries more flexibly . In this paper we focus on abstractive summarization . Typically abstractive summarization methods take a document as input and yield a generic summary to cover certain information identified by the model . However , content of interest is user-dependent . Summaries should select information with respect to preferences of a user . For example , Figure 1 shows an NBA basketball news article , and the reference summary describes several match results . However , fans of certain basketball stars in these teams such as Lebron James or Stephen Curry might only be interested in the matches they played and would like to know the player ’ s scores as well . Motivated by this , we focus on controllable summarization which allows the users to manipulate the summaries from the model . We propose CTRLsum , a framework to control summaries through control tokens in the form of a set of keywords or descriptive prompts . At training time , the model learns to predict summaries conditioned on both the source document and keywords that serve as external guidance . During inference , keywords and optional prompts , which are the target prefix to constrain decoding , are combined as control tokens to convey user preferences as shown in Figure 1 . Keywords and prompts are complementary . Prompts do not perform well in many cases such as entity or length controlled summarization as our preliminary experiments imply , but keywords can achieve those goals in a flexible way , for example , by using entity as keywords or varying the number of keywords to control entities and length respectively . However , keywords struggle in more open-ended scenarios like summarizing a list of contributions of scientific papers , while constraining the decoding with prompt “ the main contributions of this paper are : ( 1 ) ” is possibly sufficient to achieve the goal . 1Code and model checkpoints will be public after the review period . CTRLsum is trained using only keywords as additional input which can be easily identified from training summaries . It requires neither extra human annotations nor pre-defining control aspects for training , yet is quite flexible to achieve a broad scope of text manipulation as we will show in this paper . In contrast , prior work primarily rely on pre-defined “ control codes ” ( Fan et al. , 2018 ; Liu et al. , 2018 ; Keskar et al. , 2019 ) , thus need to collect annotations for training and can not generalize to unseen control aspects easily at test time . We use pretrained BART ( Lewis et al. , 2019 ) as the underlying architecture and perform experiments on three datasets in three distinct domains : CNN/Dailymail news articles ( Hermann et al. , 2015 ) , arXiv scientific papers ( Cohan et al. , 2018 ) , and BIGPATENT patent documents ( Sharma et al. , 2019 ) . We quantitatively evaluate CTRLsum on five control aspects : ( 1 ) entity-centric ( §4.2 ) and ( 2 ) length-controllable summarization ( §4.3 ) , ( 3 ) summarizing the contributions of scientific papers , ( 4 ) summarizing the purpose of an invention ( §4.4 ) , and ( 5 ) summarizing answers to given questions in a zero-shot reading comprehension setting ( §4.5 ) . Notably , our approach also achieves comparable or superior performance to the strong BART summarization model on all datasets in a standard , uncontrolled setting ( §4.6 ) , leading to state-of-the-art results on the CNN/Dailymail dataset . 2 CTRLSUM . 2.1 OVERVIEW . Unconstrained neural summarization methods are trained to learn the conditional distribution p ( y|x ) , where x and y represent the source document and summary respectively . The generated summaries depend solely on the document x without human involvement . To control the output summaries , we propose using additional control tokens z to represent user preferences and training a summarization model that predicts the conditional distribution p ( y|x , z ) . The control tokens z include keywords as extra inputs during training and inference . They can also optionally include prompts at test time to further constrain the decoding process . As shown in Figure 1 , control tokens – in the form of keywords , prompts , or a combination of both – act as an interface between users and an otherwise black-box neural model , providing a flexible way for users to explicitly control automatic summarization . Next we describe how to obtain automatic keywords for training as well as potential applications at test time . 2.2 AUTOMATIC KEYWORD EXTRACTION . In addition to extracting keywords from training data to train the model , CTRLsum also features an automatic keywords extraction mechanism at test time , which can be used to suggest automatic keywords according to user preferences , or perform uncontrolled summarization without user signals . Next we describe the keywords extraction methods at training and inference time respectively . Training . For training , we use the ground-truth summary to identify keywords in the source document . Specifically , we first greedily select sentences from the document that maximize the ROUGE scores ( Lin , 2004 ) with the reference summary . This step constrains keywords to those found in important sentences . Then , we identify all the longest sub-sequences in the extracted sentences that have matched sub-sequences in the ground-truth summary , similar to the copying word recognition method in ( Gehrmann et al. , 2018 ) . Finally , we remove duplicate words and stop words and keep the remaining tokens as keywords . Compared to other keywords extraction methods ( Riloff & Lehnert , 1994 ; Mihalcea & Tarau , 2004 ) which output only a few salient words , our extraction retains most content words found in the summary . This encourages dependence on the given keywords by building a reliable correlation between their presence in the input and the target . It in turn ensures that user-provided keywords are not ignored by the model at test time , which is catastrophic for a controllable summarization system . Inference . We formulate the keyword extraction problem at test time as a sequence labeling task . Concretely , we train a BERT-based sequence tagger ( Devlin et al. , 2018 ) on the keywords and documents from training dataset . This tagger then computes the selection probability qj for each token in the test document . Similar to training time extraction , we first select ns sentences with the highest average token selection probability . Within these sentences words with qj > are selected as keywords up to a maximum number of mmax . The three hyperparameters ns , , mmax are selected based on the uncontrolled summarization performance on validation datasets . The results are reasonably robust to different settings ( see Appendix D for details ) . 2.3 SUMMARIZATION : TRAINING DETAILS . Format . At training time we prepend the keyword sequence to the source document separated with a special token . The summarization model is then trained to maximize p ( y|x , z ) in an end-to-end fashion . The keyword sequence maintains the order of the keywords as they were in the source document , but we observe that the model often ignores this ordering as it frequently differs between source and target summary . We also separate keywords from different source sentences with the special token ( “ | ” ) . In applications where the sentence boundary is unknown , as when users propose their own keywords , the “ | ” token can be ignored as in some of our experiments . Keyword Dropout . As mentioned in §2.2 , our keyword extraction strategy retains most words from the summary found in the source document . Without regularization , the dependence on such keywords is strong enough that the model rarely generates novel words in the summary . To remedy this , we randomly drop keywords at training time so that the model learns to rely on keywords that are present in the input , while also learning to still carry over key information from the source document that is not present in the keywords . Note that keywords dropout is applied at training time only . Next we are going to introduce the five control aspects that we study in this paper as example use cases of CTRLsum . Qualitative examples of them are shown in Table 1 . 2.4 SUMMARIZATION : INFERENCE WITH KEYWORDS .. The keywords provide a generic interface to control multiple aspects of summaries , which allows the user to optionally rely on automatically extracted keywords , user provided keywords , or a combination of both . This method provides clean separation of test-time user control and the training process , including pretraining . Consequently , CTRLsum can be adapted to new use cases without changing model parameters . For example , though nothing during training specifically focuses on controlling entities or length , examples below demonstrate the general applicability of keyword control to entity and length manipulation . Entity Control . The goal of entity control is to produce summaries that focus on entities of interest . Figure 1 exemplifies summarization with respect to different players when those player names are included as keywords directly influencing the summary . Length Control . Users may have different preferences as to the length of summaries . We allow such manipulation of the summary length through a user-specified length parameter . Specifically , we first separate the training data into 5 buckets by summary length so that each bucket has the same number of examples . Then we compute the average number of keywords Kl for each bucket on the training data . At test time , a user can specify length parameter l ∈ { 0 , 1 , 2 , 3 , 4 } to include the Kl keywords with the highest selection probability computed by the sequence tagger . This is similar to ( Saito et al. , 2020a ) , which uses the number of “ guiding words ” to control summary length . 2.5 SUMMARIZATION : INFERENCE WITH KEYWORDS AND PROMPTS . Prompts are pre-defined text sequences used as the target prefix to constrain decoding . They have been utilized to perform multi-purpose text generation with a single unified model ( Radford et al. , 2019 ; Brown et al. , 2020 ) . In the CTRLsum framework , prompts are a kind of control token sequence , and we always use such tokens as both the target prefix and keywords ( ablation results on using prompts as keywords or prefix alone can be found in Appendix C ) . We find that using prompts as keywords besides prefix helps focus on prompt-related content and mitigate the over-generation issue of vanilla summarization models , as we will show in §4.4 . To the best of our knowledge , we are the first to evaluate such a prompt-based control method for summarization systems . Summarizing Contributions . Existing datasets about scientific papers such as arXiv ( Cohan et al. , 2018 ) collect paper abstracts as the summaries , which often include extra background context and lack detailed contribution descriptions for the associated paper . In many cases , readers would benefit from an explicit list of contributions in order to understand the novelty and value of the paper . For these cases , we propose using control tokens – “ the main contributions of this paper are : ( 1 ) ” . This prompt then triggers generation of a summary focused on contributions . Summarizing Invention Purpose . Patent article summaries in existing datasets such as BIGPATENT ( Sharma et al. , 2019 ) can be over-complicated , often covering core method details . Yet for a non-technical reader it would be preferred to provide a one-sentence summary that states the purpose of the invention while ignoring technical details . To apply CTRLsum in this scenario , we use the control tokens , “ the purpose of the present invention is ” . This triggers a concise summary focused on patent purpose . Question-guided summarization . Human summarization can be constrained by questions ( Kryściński et al. , 2019 ) that require answers to be found in the summary . This points to an important connection between summarization and reading comprehension that we further explore . We hypothesize that a summarization model can directly answer some questions about the article if guided properly . This suggests the possibility of subsuming reading comprehension as a form of summarization . To verify this hypothesis , we use the control tokens “ Q : question text ? A : ” to trigger reading comprehension behaviour . We note that prompts- and keywords-based control are complementary in practice – while prompts could theoretically achieve any type of control , empirically they often do not work well for many aspects and the model is very sensitive to the precise wording of the prompt . For example , we found that using prompts such as “ a summary focused on [ entity ] is : ” or “ a short summary is : ” does not work as well as explicitly using keywords for entity or length control ( details can be found in Appendix C ) .

Summary: This paper proposes a two-stage summarization system where a document is provided along with (optionally) keywords or a prompt. This supplemental information helps to guide the summarization and possibly make it more user-specific. The keywords and prompt can also be guessed automatically by a BERT-base model, which seems to improve automatic metrics on CNN/daily mail.