CorDial: Coarse-to-fine Abstractive Dialogue Summarization with Controllable Granularity

Dialogue summarization is challenging due to its multi-speaker standpoints , casual spoken language style , and limited labeled data . In this paper , we propose CORDIAL , aiming to improve the abstractive dialogue summarization quality and , at the same time , enable granularity controllability . We propose 1 ) a coarse-tofine generation strategy that generates a summary draft followed by a final summary . The summary draft , which provides weakly-supervised signals , comprises pseudo-labeled interrogative pronoun categories and noisy key phrases extracted with a constituency parser . 2 ) A simple strategy to control the granularity of the final summary . CORDIAL can automatically determine or control the number of generated summary sentences for a given dialogue by predicting and highlighting different text spans from the source text . Our model achieves state-of-the-art performance on the largest dialogue summarization corpus SAMSum . We conduct comprehensive case study and show competitive human evaluation results and controllability to human-annotated summaries . 1 INTRODUCTION . Text summarization tasks aim to distill the most critical information in a text to produce an abridged version . In particular , abstractive – as opposed to extractive – summarization requires neural generative models with a high level of semantic understanding , as the output words do not necessarily appear in the source text . It is more challenging but gives much flexibility to a summary compared to any extractive summarization models ( Zhang et al. , 2018 ) . Abstractive dialogue summarization has been discussed in the literature of AMI meeting corpus ( McCowan et al. , 2005 ) . The size and quality of labeled data are bottlenecks , as collecting summaries is costly and judgements when creating them are inherently subjective . The AMI corpus has only 141 summaries , and the largest dialogue summarization dataset SAMSum ( Gliwa et al. , 2019 ) has number of training samples only equal to 5 % of the commonly-used text summarization dataset CNN/DailyMail ( Hermann et al. , 2015 ) . In addition to ( and perhaps due to ) the shortage of labeled data , dialogue summarization has not received much attention despite the prevalence of dialogues ( text messages , emails , social media ) and the vast application potential of dialogue summarization systems . Significant research efforts have been focused on summarization of single-speaker documents such as News ( Hermann et al. , 2015 ; Nallapati et al. , 2016 ; See et al. , 2017 ) or scientific publications ( Qazvinian & Radev , 2008 ; Nikolov et al. , 2018 ) . However , summarizing a dialogue presents a unique set of challenges . A conversation always involves multiple speakers with different points of view , and its natural language style is very different from a standard writing style . For example , conversational data contains more abbreviations and typos . Information is more scattered across in a dialogue , compared to articles where usually titles or the first few sentences contain the most salient information . Recently , the ability to control text summarization in the News domain has been gradually attracting more attention ( Fan et al. , 2018 ; Liu et al. , 2018 ) , with work focusing on learning length embeddings to control summary lengths . However , the length information is only added during the decoding stage , making the encoding stage less informed , and the overall conditional generation implicit . Saito et al . ( 2020 ) instead first explicitly extract “ prototype ” text span in the desired length and then paraphrase it to the output News summary . However , the retrieve-and-rewrite process is restricted by the extraction quality , leaving its performance limited by extractive solutions ’ capabilities . In this paper , we propose CORDIAL , a coarse-to-fine abstractive dialogue summarization model equipped with granularity controllability . Unlike previous methods ( Goo & Chen , 2018 ; Pan et al. , 2018 ) which heavily rely on explicit intent annotations in datasets , we label each dialogue turn with a pre-defined interrogative pronoun category using a weakly-supervised labeling approach . The automatically labeled user intent together with its corresponding key phrase extraction provide weak supervision during summary generation . In addition , we propose a length-controllable generation method specifically for dialogue summarizaiton . We match each summary sentence “ linearly ” to its corresponding dialogue context and clip it by highlighting tokens . We then train our model to predict where to clip and generate only one sentence for each highlighted dialogue . This strategy enables CORDIAL to generate a summary at different granularity by highlighting arbitrary numbers of text spans from a dialogue and making our model more interpretable . We build our model on top of BART-xsum ( Lewis et al. , 2019 ) , which is first pre-trained with unsupervised denoising objectives , and further fine-tuned on the News summarization corpus XSUM ( Narayan et al. , 2018 ) . In the experimental results , we show that CORDIAL achieves state-ofthe-art dialogue summarization performance on several automatic metrics . The main contributions of this work 1 are : 1 ) We propose a coarse-to-fine strategy that uses artificial summary draft as weak supervision , 2 ) we introduce a text-span based conditional generation approach to control the granularity of generated dialogue summaries without human-written summaries at different detail levels , and 3 ) we conduct comprehensive case study and human evaluation to show that CORDIAL can achieve consistent and informative summary , especially for controllable summary , where existing models either can not do it or do it poorly . 2 METHODOLOGY . In this section , we first briefly cover the background of generative language pre-trained models . Then , we introduce our proposed summary draft construction and summary controllability in detail . The proposed solution is generalizable to all the generative language models . We define the conversational history input as D = { X1 , X2 , . . . , XN } , where each Xi has a sequence of words , N is the 1Our code is released at www.anonymous.com total numbers of dialogue turns , and the input may contain more than two speakers . We intend to generate M -sentence dialogue summary Y = { Y1 , . . . , YM } that is suppose to be briefer than the overall dialogue history . 2.1 GENERATIVE PRE-TRAINED LANGUAGE MODELS . Self-supervised generative language models ( Radford et al. , 2019 ; Yang et al. , 2019 ; Dong et al. , 2019 ; Lewis et al. , 2019 ) have achieved remarkable success in many NLP tasks . Instead of solely training on a specific dataset , pre-training on a large-scale text corpus such as Wikipedia and BookCorpus has shown good knowledge transferability . However , in conversational AI , previous works ( Wolf et al. , 2019b ; Wu et al. , 2020 ) argue that there is an intrinsic difference of linguistic patterns between human conversations and writing text . We would like to answer the following question : which generative language model is the best base model for dialogue summarization tasks . In this paper , we investigate four recently proposed models , DialoGPT ( Zhang et al. , 2019d ) , UniLM ( Dong et al. , 2019 ) , PEGASUS ( Zhang et al. , 2019a ) , and BART ( Lewis et al. , 2019 ) . The first one is a generative language model trained on open-domain conversational data , and the other three achieve promising results on several text summarization benchmarks . 2.2 DRAFT CONSTRUCTION . For each turn in the dialogue , we create a summary draft containing the speaker ’ s dialogue action category ( defined later ) together with the most critical key phrases . Our hypothesis is such summary drafts can provide useful weak supervision beneficial to the final summarization task . Unlike taskoriented dialogue systems , which have explicit and annotated intents ( e.g. , book flight and check account ) and actions ( e.g. , inform and request ) , dialogue summarization tasks rarely have such labels . Therefore , we define a set of interrogative pronoun categories , and then label each utterance with its category using a weakly-supervised labeling approach ( Ratner et al. , 2019 ) .2 The interrogative pronoun categories are designed to identify functional units ( intent ) of all utterances , serving as the dialogue ’ s logic . For example , in Figure 1 , Morgan asked Suzanne “ Do you feel like going to a concert next week ? ” One can expect that Suzanne will confirm her willingness in the next dialogue turn . We define such dialogue action categories as follows : WHY : asks the reason of the state mentioned in the previous turn , e.g. , “ why ” or “ why not ” ; WHAT : requests more details about the aforementioned object , the sentence usually starts with “ what ’ s ” or “ what about ” ; WHERE : asks the location of an appointment or event ; WHEN : asks the time of an appointment or event , e.g . , “ when ” or “ what time ” ; CONFIRM : asks the other speaker to establish the correctness of certain case , the sentence usually starts with patterns like “ are you ” , “ will you ” or “ has he ” ; ABSTAIN : the utterance does not belong to any of the previous categories . It occurs when speakers continue to state or comment without seeking for more information from the others . Following this , we define key phrases as essential words or phrases in the annotated summary . We first use a trained constituency parser ( Kitaev & Klein , 2018 ) to parse each dialogue turn and each summary sentence . Then , we identify the longest common sub-sequence with a threshold among dialogue and summary to be the key phrases . Note that not every dialogue turn contains key phrases and the key phrases could be noisy . Overall , we construct a summary draft by concatenating turn index , dialogue action category label , and extracted key phrases within the entire dialogue history into a string , ending with a special token , “ TL ; DR ” . Take Figure 1 as an example , the summary draft is “ 1 what 2 abstain ’ s just one of ... square garden 8 why 9 abstain TL ; DR ” . We train our model first to generate this summary draft and then generate the final summary in an autoregressive way . We use TL ; DR token to distinguish draft and final summary during inference time . 2.3 CONTROLLABILITY . The high-level intuition for our solution is that if we can control a generative model only to generate one sentence as output for a partially-highlighted input , we can control the number of output sentences by choosing how to highlight the input . We highlight each dialogue split using the special token pair < hl > and < /hl > . For example , in Figure 1 , we generate the first summary sentence for the first segment marked in blue color with the highlighting token pair , and the second 2github.com/snorkel-team/snorkel and third by green and orange markers , respectively . This way , we can not only gain the summary controllability but also make the generation more interpretable . The next challenge is , during training , we have to find a mapping between each sentence in a reference summary to its corresponding dialogue split . In other words , how do we know where to insert the highlighting tokens ? We do so by training a dialogue-turn-level binary classifier ( detailed below ) that predicts whether each turn is a cutting point . Our key observation is that sentences within a reference summary usually have a strong temporal dependency , that is , people summarize the dialogue almost “ linearly ” . We use a naive approach that simply find the dialogue split that has the highest ROUGE score to each summary sentence . The cutting point tm = argmaxt SIM ( Xcm : t , Ym ) , ( 1 ) where SIM could be any similarity functions ( we use ROUGE-1 ) available , and cm is the accumulated turn index ( c1 = 1 and cm = tm−1 ) that indicates which part of a dialogue has been covered by a summary sentence . tm is the cutting point in the dialogue history for the m-th summary sentence . Note that for a summary with M sentences , we only need M − 1 cutting points . With the pseudo labels ( tm ) provided by this heuristic , we formulate the dialogue segmentation problem into a binary classification problem . Specifically , we train a classifier C , which takes dialogue history as input and predicts whether each dialogue turn is a cutting point . We prefix each dialogue turn with a separation token ( e.g. , xsep = < s > ) and take such long sequence as input to the classifier . H = C ( [ xsep , X1 , xsep , X2 , . . . , xsep , XN ] ) ∈ RN×demb , P̂ = Sigmoid ( W1 ( H ) ) ∈ RN . ( 2 ) The classifier output H is the representations of those separation tokens , and each of them is a demb dimension vector . W1 ∈ Rdemb×1 is a trainable linear mapping . The P̂ is the predicted segment probability that is trained with binary cross-entropy loss . We use a BERT-base model ( Devlin et al. , 2018 ) as a classifier C and the i-th cutting point is triggered if P̂i > 0.5 . This prediction means that our model can automatically determine how many sentences should be generated in the final summary . If no cutting point is triggered , then we will generate a one-sentence summary . If one cutting point is triggered , we will have a two-sentence summary , and so on . Note that the segment classifier does not need to be perfect ( e.g. , 86 % accuracy in our experiment ) as the labels contain a certain noise and splitting a dialogue by one turn earlier or later usually does not make a big difference . As a result , we find that the final ROUGE score is quite similar for both “ oracle ” dialogue splits and predicted dialogue splits . Finally , we can control the number of output summary sentences by controlling the dialogue split . Specifically , we first decide the expected number of output sentences ( e.g. , K ) , and then we choose the top K− 1 indexes with highest probabilities in segmentation probability P̂ . We use these K− 1 indexes as cutting points to add highlighting tokens . For example , if we want to have a threesentence summarization , we can split the dialogue into three pieces by choosing the top two highest segment probability . Then , we input the same dialogue three times with different highlighting portions to CORDIAL and obtain one-sentence summary each . We can also generate one-sentence summary by clipping the whole dialogue with one pair of highlighting tokens at the beginning and the end of a dialogue ( we call this output CORDIAL-1 ) .

Summary: This paper proposes CorDial, a new method for dialog summarization. CorDial firstly constructs a coarse draft by generating intent and key phrases for every dialog turn, and splits the dialog into chunks by inserting special boundary tokens; then the segmented original dialog and the constructed draft are feed as input to generate the final summary. CorDial employs BART-xsum as its backbone model, which is a pre-trained language model finetuned on XSUM summary dataset. Experiment result on SAMsum dataset shows CorDial achieves SotA performance under both automatic evaluation metric and human evaluations.