CorDial: Coarse-to-fine Abstractive Dialogue Summarization with Controllable Granularity

Dialogue summarization is challenging due to its multi-speaker standpoints , casual spoken language style , and limited labeled data . In this paper , we propose CORDIAL , aiming to improve the abstractive dialogue summarization quality and , at the same time , enable granularity controllability . We propose 1 ) a coarse-tofine generation strategy that generates a summary draft followed by a final summary . The summary draft , which provides weakly-supervised signals , comprises pseudo-labeled interrogative pronoun categories and noisy key phrases extracted with a constituency parser . 2 ) A simple strategy to control the granularity of the final summary . CORDIAL can automatically determine or control the number of generated summary sentences for a given dialogue by predicting and highlighting different text spans from the source text . Our model achieves state-of-the-art performance on the largest dialogue summarization corpus SAMSum . We conduct comprehensive case study and show competitive human evaluation results and controllability to human-annotated summaries . 1 INTRODUCTION . Text summarization tasks aim to distill the most critical information in a text to produce an abridged version . In particular , abstractive – as opposed to extractive – summarization requires neural generative models with a high level of semantic understanding , as the output words do not necessarily appear in the source text . It is more challenging but gives much flexibility to a summary compared to any extractive summarization models ( Zhang et al. , 2018 ) . Abstractive dialogue summarization has been discussed in the literature of AMI meeting corpus ( McCowan et al. , 2005 ) . The size and quality of labeled data are bottlenecks , as collecting summaries is costly and judgements when creating them are inherently subjective . The AMI corpus has only 141 summaries , and the largest dialogue summarization dataset SAMSum ( Gliwa et al. , 2019 ) has number of training samples only equal to 5 % of the commonly-used text summarization dataset CNN/DailyMail ( Hermann et al. , 2015 ) . In addition to ( and perhaps due to ) the shortage of labeled data , dialogue summarization has not received much attention despite the prevalence of dialogues ( text messages , emails , social media ) and the vast application potential of dialogue summarization systems . Significant research efforts have been focused on summarization of single-speaker documents such as News ( Hermann et al. , 2015 ; Nallapati et al. , 2016 ; See et al. , 2017 ) or scientific publications ( Qazvinian & Radev , 2008 ; Nikolov et al. , 2018 ) . However , summarizing a dialogue presents a unique set of challenges . A conversation always involves multiple speakers with different points of view , and its natural language style is very different from a standard writing style . For example , conversational data contains more abbreviations and typos . Information is more scattered across in a dialogue , compared to articles where usually titles or the first few sentences contain the most salient information . Recently , the ability to control text summarization in the News domain has been gradually attracting more attention ( Fan et al. , 2018 ; Liu et al. , 2018 ) , with work focusing on learning length embeddings to control summary lengths . However , the length information is only added during the decoding stage , making the encoding stage less informed , and the overall conditional generation implicit . Saito et al . ( 2020 ) instead first explicitly extract “ prototype ” text span in the desired length and then paraphrase it to the output News summary . However , the retrieve-and-rewrite process is restricted by the extraction quality , leaving its performance limited by extractive solutions ’ capabilities . In this paper , we propose CORDIAL , a coarse-to-fine abstractive dialogue summarization model equipped with granularity controllability . Unlike previous methods ( Goo & Chen , 2018 ; Pan et al. , 2018 ) which heavily rely on explicit intent annotations in datasets , we label each dialogue turn with a pre-defined interrogative pronoun category using a weakly-supervised labeling approach . The automatically labeled user intent together with its corresponding key phrase extraction provide weak supervision during summary generation . In addition , we propose a length-controllable generation method specifically for dialogue summarizaiton . We match each summary sentence “ linearly ” to its corresponding dialogue context and clip it by highlighting tokens . We then train our model to predict where to clip and generate only one sentence for each highlighted dialogue . This strategy enables CORDIAL to generate a summary at different granularity by highlighting arbitrary numbers of text spans from a dialogue and making our model more interpretable . We build our model on top of BART-xsum ( Lewis et al. , 2019 ) , which is first pre-trained with unsupervised denoising objectives , and further fine-tuned on the News summarization corpus XSUM ( Narayan et al. , 2018 ) . In the experimental results , we show that CORDIAL achieves state-ofthe-art dialogue summarization performance on several automatic metrics . The main contributions of this work 1 are : 1 ) We propose a coarse-to-fine strategy that uses artificial summary draft as weak supervision , 2 ) we introduce a text-span based conditional generation approach to control the granularity of generated dialogue summaries without human-written summaries at different detail levels , and 3 ) we conduct comprehensive case study and human evaluation to show that CORDIAL can achieve consistent and informative summary , especially for controllable summary , where existing models either can not do it or do it poorly . 2 METHODOLOGY . In this section , we first briefly cover the background of generative language pre-trained models . Then , we introduce our proposed summary draft construction and summary controllability in detail . The proposed solution is generalizable to all the generative language models . We define the conversational history input as D = { X1 , X2 , . . . , XN } , where each Xi has a sequence of words , N is the 1Our code is released at www.anonymous.com total numbers of dialogue turns , and the input may contain more than two speakers . We intend to generate M -sentence dialogue summary Y = { Y1 , . . . , YM } that is suppose to be briefer than the overall dialogue history . 2.1 GENERATIVE PRE-TRAINED LANGUAGE MODELS . Self-supervised generative language models ( Radford et al. , 2019 ; Yang et al. , 2019 ; Dong et al. , 2019 ; Lewis et al. , 2019 ) have achieved remarkable success in many NLP tasks . Instead of solely training on a specific dataset , pre-training on a large-scale text corpus such as Wikipedia and BookCorpus has shown good knowledge transferability . However , in conversational AI , previous works ( Wolf et al. , 2019b ; Wu et al. , 2020 ) argue that there is an intrinsic difference of linguistic patterns between human conversations and writing text . We would like to answer the following question : which generative language model is the best base model for dialogue summarization tasks . In this paper , we investigate four recently proposed models , DialoGPT ( Zhang et al. , 2019d ) , UniLM ( Dong et al. , 2019 ) , PEGASUS ( Zhang et al. , 2019a ) , and BART ( Lewis et al. , 2019 ) . The first one is a generative language model trained on open-domain conversational data , and the other three achieve promising results on several text summarization benchmarks . 2.2 DRAFT CONSTRUCTION . For each turn in the dialogue , we create a summary draft containing the speaker ’ s dialogue action category ( defined later ) together with the most critical key phrases . Our hypothesis is such summary drafts can provide useful weak supervision beneficial to the final summarization task . Unlike taskoriented dialogue systems , which have explicit and annotated intents ( e.g. , book flight and check account ) and actions ( e.g. , inform and request ) , dialogue summarization tasks rarely have such labels . Therefore , we define a set of interrogative pronoun categories , and then label each utterance with its category using a weakly-supervised labeling approach ( Ratner et al. , 2019 ) .2 The interrogative pronoun categories are designed to identify functional units ( intent ) of all utterances , serving as the dialogue ’ s logic . For example , in Figure 1 , Morgan asked Suzanne “ Do you feel like going to a concert next week ? ” One can expect that Suzanne will confirm her willingness in the next dialogue turn . We define such dialogue action categories as follows : WHY : asks the reason of the state mentioned in the previous turn , e.g. , “ why ” or “ why not ” ; WHAT : requests more details about the aforementioned object , the sentence usually starts with “ what ’ s ” or “ what about ” ; WHERE : asks the location of an appointment or event ; WHEN : asks the time of an appointment or event , e.g . , “ when ” or “ what time ” ; CONFIRM : asks the other speaker to establish the correctness of certain case , the sentence usually starts with patterns like “ are you ” , “ will you ” or “ has he ” ; ABSTAIN : the utterance does not belong to any of the previous categories . It occurs when speakers continue to state or comment without seeking for more information from the others . Following this , we define key phrases as essential words or phrases in the annotated summary . We first use a trained constituency parser ( Kitaev & Klein , 2018 ) to parse each dialogue turn and each summary sentence . Then , we identify the longest common sub-sequence with a threshold among dialogue and summary to be the key phrases . Note that not every dialogue turn contains key phrases and the key phrases could be noisy . Overall , we construct a summary draft by concatenating turn index , dialogue action category label , and extracted key phrases within the entire dialogue history into a string , ending with a special token , “ TL ; DR ” . Take Figure 1 as an example , the summary draft is “ 1 what 2 abstain ’ s just one of ... square garden 8 why 9 abstain TL ; DR ” . We train our model first to generate this summary draft and then generate the final summary in an autoregressive way . We use TL ; DR token to distinguish draft and final summary during inference time . 2.3 CONTROLLABILITY . The high-level intuition for our solution is that if we can control a generative model only to generate one sentence as output for a partially-highlighted input , we can control the number of output sentences by choosing how to highlight the input . We highlight each dialogue split using the special token pair < hl > and < /hl > . For example , in Figure 1 , we generate the first summary sentence for the first segment marked in blue color with the highlighting token pair , and the second 2github.com/snorkel-team/snorkel and third by green and orange markers , respectively . This way , we can not only gain the summary controllability but also make the generation more interpretable . The next challenge is , during training , we have to find a mapping between each sentence in a reference summary to its corresponding dialogue split . In other words , how do we know where to insert the highlighting tokens ? We do so by training a dialogue-turn-level binary classifier ( detailed below ) that predicts whether each turn is a cutting point . Our key observation is that sentences within a reference summary usually have a strong temporal dependency , that is , people summarize the dialogue almost “ linearly ” . We use a naive approach that simply find the dialogue split that has the highest ROUGE score to each summary sentence . The cutting point tm = argmaxt SIM ( Xcm : t , Ym ) , ( 1 ) where SIM could be any similarity functions ( we use ROUGE-1 ) available , and cm is the accumulated turn index ( c1 = 1 and cm = tm−1 ) that indicates which part of a dialogue has been covered by a summary sentence . tm is the cutting point in the dialogue history for the m-th summary sentence . Note that for a summary with M sentences , we only need M − 1 cutting points . With the pseudo labels ( tm ) provided by this heuristic , we formulate the dialogue segmentation problem into a binary classification problem . Specifically , we train a classifier C , which takes dialogue history as input and predicts whether each dialogue turn is a cutting point . We prefix each dialogue turn with a separation token ( e.g. , xsep = < s > ) and take such long sequence as input to the classifier . H = C ( [ xsep , X1 , xsep , X2 , . . . , xsep , XN ] ) ∈ RN×demb , P̂ = Sigmoid ( W1 ( H ) ) ∈ RN . ( 2 ) The classifier output H is the representations of those separation tokens , and each of them is a demb dimension vector . W1 ∈ Rdemb×1 is a trainable linear mapping . The P̂ is the predicted segment probability that is trained with binary cross-entropy loss . We use a BERT-base model ( Devlin et al. , 2018 ) as a classifier C and the i-th cutting point is triggered if P̂i > 0.5 . This prediction means that our model can automatically determine how many sentences should be generated in the final summary . If no cutting point is triggered , then we will generate a one-sentence summary . If one cutting point is triggered , we will have a two-sentence summary , and so on . Note that the segment classifier does not need to be perfect ( e.g. , 86 % accuracy in our experiment ) as the labels contain a certain noise and splitting a dialogue by one turn earlier or later usually does not make a big difference . As a result , we find that the final ROUGE score is quite similar for both “ oracle ” dialogue splits and predicted dialogue splits . Finally , we can control the number of output summary sentences by controlling the dialogue split . Specifically , we first decide the expected number of output sentences ( e.g. , K ) , and then we choose the top K− 1 indexes with highest probabilities in segmentation probability P̂ . We use these K− 1 indexes as cutting points to add highlighting tokens . For example , if we want to have a threesentence summarization , we can split the dialogue into three pieces by choosing the top two highest segment probability . Then , we input the same dialogue three times with different highlighting portions to CORDIAL and obtain one-sentence summary each . We can also generate one-sentence summary by clipping the whole dialogue with one pair of highlighting tokens at the beginning and the end of a dialogue ( we call this output CORDIAL-1 ) .

Summary: This paper addresses the problem of abstractive dialogue summarization. Its key idea is to label an interrogative pronoun category and extract key phrases from each dialogue turn as weak guide for dialogue summarization. It also proposes a length-controllable generation method for final summary. The proposed approach is evaluated on the SAMSum as one of the largest abstractive dialogue summarization benchmarks, on which it shows competitive performance over recent models.