Abstractive Dialog Summarization with Semantic Scaffolds

1 INTRODUCTION . Summarization aims to condense a piece of text to a shorter version , retaining the critical information . On dialogs , summarization has various promising applications in the real world . For instance , the automatic doctor-patient interaction summary can save doctors ’ massive amount of time used for filling medical records . There is also a general demand for summarizing meetings in order to track project progress in the industry . Generally , multi-party conversations with interactive communication are more difficult to summarize than single-speaker documents . Hence , dialog summarization will be a potential field in summarization track . There are two types of summarization : extractive and abstractive . Extractive summarization selects sentences or phrases directly from the source text and merges them to a summary , while abstractive summarization attempts to generate novel expressions to condense information . Previous dialog summarization research mostly study extractive summarization ( Murray et al. , 2005 ; Maskey & Hirschberg , 2005 ) . Extractive methods merge selected important utterances from a dialog to form summary . Because dialogs are highly dependant on their histories , it is difficult to produce coherent discourses with a set of non-consecutive conversation turns . Therefore , extractive summarization is not the best approach to summarize dialogs . However , most modern abstractive methods focus on single-speaker documents rather than dialogs due to the lack of dialog summarization corpora . Popular abstractive summarization dataset like CNN/Daily Mail ( Hermann et al. , 2015 ) is on news documents . AMI meeting corpus ( McCowan et al. , 2005 ) is the common benchmark , but it only has extractive summary . In this work , we introduce a dataset for abstractive dialog summarization based on MultiWOZ ( Budzianowski et al. , 2018 ) . Seq2Seq models such as Pointer-Generator ( See et al. , 2017 ) have achieved high-quality summaries of news document . However , directly applying a news summarizer to dialog results in two drawbacks : informative entities such as place name are difficult to capture precisely and contents in different domains are summarized unequally . To address these problems , we propose Scaffold Pointer Network ( SPNet ) . SPNet incorporates three types of semantic scaffolds in dialog : speaker role , semantic slot , and dialog domain . Firstly , SPNet adapts separate encoder to attentional Seq2Seq framework , producing distinct semantic representations for different speaker roles . Then , our method inputs delexicalized utterances for producing delexicalized summary , and fills in slot values to generate complete summary . Finally , we incorporate dialog domain scaffold by jointly optimizing dialog domain classification task along with the summarization task . We evaluate SPNet with both automatic and human evaluation metrics on MultiWOZ . SPNet outperforms Pointer-Generator ( See et al. , 2017 ) and Transformer ( Vaswani et al. , 2017 ) on all the metrics . 2 RELATED WORK . Rush et al . ( 2015 ) first applied modern neural models to abstractive summarization . Their approach is based on Seq2Seq framework ( Sutskever et al. , 2014 ) and attention mechanism ( Bahdanau et al. , 2015 ) , achieving state-of-the-art results on Gigaword and DUC-2004 dataset . Gu et al . ( 2016 ) proposed copy mechanism in summarization , demonstrating its effectiveness by combining the advantages of extractive and abstractive approach . See et al . ( 2017 ) applied pointing ( Vinyals et al. , 2015 ) as copy mechanism and use coverage mechanism ( Tu et al. , 2016 ) to discourage repetition . Most recently , reinforcement learning ( RL ) has been employed in abstractive summarization . RL-based approaches directly optimize the objectives of summarization ( Ranzato et al. , 2016 ; Celikyilmaz et al. , 2018 ) . However , deep reinforcement learning approaches are difficult to train and more prone to exposure bias ( Bahdanau et al. , 2017 ) . Recently , pre-training methods are popular in NLP applications . BERT ( Devlin et al. , 2018 ) and GPT ( Radford et al. , 2018 ) have achieved state-of-the-art performance in many tasks , including summarization . For instance , Zhang et al . ( 2019 ) proposed a method to pre-train hierarchical document encoder for extractive summarization . Hoang et al . ( 2019 ) proposed two strategies to incorporate a pre-trained model ( GPT ) to perform the abstractive summarizer and achieved a better performance . However , there has not been much research on adapting pre-trained models to dialog summarization . Dialog summarization , specifically meeting summarization , has been studied extensively . Previous work generally focused on statistical machine learning methods in extractive dialog summarization : Galley ( 2006 ) used skip-chain conditional random fields ( CRFs ) ( Lafferty et al. , 2001 ) as a ranking method in extractive meeting summarization . Wang & Cardie ( 2013 ) compared support vector machines ( SVMs ) ( Cortes & Vapnik , 1995 ) with LDA-based topic models ( Blei et al. , 2003 ) for producing decision summaries . However , abstractive dialog summarization was less explored due to the lack of a suitable benchmark . Recent work ( Wang & Cardie , 2016 ; Goo & Chen , 2018 ; Pan et al. , 2018 ) created abstractive dialog summary benchmarks with existing dialog corpus . Goo & Chen ( 2018 ) annotated topic descriptions in AMI meeting corpus as the summary . However , topics they defined are coarse , such as “ industrial designer presentation ” . They also proposed a model with a sentence-gated mechanism incorporating dialog acts to perform abstractive summarization . Moreover , Li et al . ( 2019 ) first built a model to summarize audio-visual meeting data with an abstractive method . However , previous work has not investigated the utilization of semantic patterns in dialog , so we explore it in-depth in our work . 3 PROPOSED METHOD . As discussed above , state-of-the-art document summarizers are not applicable in conversation settings . We propose Scaffold Pointer Network ( SPNet ) based on Pointer-Generator ( See et al. , 2017 ) . SPNet incorporates three types of semantic scaffolds to improve abstractive dialog summarization : speaker role , semantic slot and dialog domain . 3.1 BACKGROUND . We first introduce Pointer-Generator ( See et al. , 2017 ) . It is a hybrid model of the typical Seq2Seq attention model ( Nallapati et al. , 2016 ) and pointer network ( Vinyals et al. , 2015 ) . Seq2Seq framework encodes source sequence and generates the target sequence with the decoder . The input sequence is fed into the encoder token by token , producing the encoder hidden states hi in each encoding step . The decoder receives word embedding of the previous word and generates a distribution to decide the target element in this step , retaining decoder hidden states st . In Pointer-Generator , attention distribution at is computed as in Bahdanau et al . ( 2015 ) : eti = v T tanh ( Whhi +Wsst + battn ) at = softmax ( et ) ( 1 ) where Wh , Ws , v and battn are all learnable parameters . With the attention distribution at , context vector h∗t is computed as the weighted sum of encoder ’ s hidden states . Context vector is regarded as the attentional information in the source text : h∗t = ∑ i atihi ( 2 ) Pointer-Generator differs from typical Seq2Seq attention model in the generation process . The pointing mechanism combines copying words directly from the source text with generating words from a fixed vocabulary . Generation probability pgen is calculated as “ a soft switch ” to choose from copy and generation : pgen = σ ( wTh∗h ∗ t + w T s st + w T x xt + bptr ) ( 3 ) where xt is the decoder input , wh∗ , ws , wx and bptr are all learnable parameters . σ is sigmoid function , so the generation probability pgen has a range of [ 0 , 1 ] . The ability to select from copy and generation corresponds to a dynamic vocabulary . Pointer network forms an extended vocabulary for the copied tokens , including all the out-of-vocabulary ( OOV ) words appeared in the source text . The final probability distribution P ( w ) on extended vocabulary is computed as follows : Pvocab = softmax ( V ′ ( V [ st , h ∗ t ] + b ) + b ′ ) P ( w ) = pgenPvocab ( w ) + ( 1− pgen ) ∑ i : wi=w ati ( 4 ) where Pvocab is the distribution on the original vocabulary , V ′ , V , b and b′ are learnable parameters used to calculate such distribution . 3.2 SCAFFOLD POINTER NETWORK ( SPNET ) . Our Scaffold Pointer Network ( depicted in Figure 1 ) is based on Pointer-Generator ( See et al. , 2017 ) . The contribution of SPNet is three-fold : separate encoding for different roles , incorporating semantic slot scaffold and dialog domain scaffold . 3.2.1 SPEAKER ROLE SCAFFOLD . Our encoder-decoder framework employs separate encoding for different speakers in the dialog . User utterances xusrt and system utterances x sys t are fed into a user encoder and a system encoder separately to obtain encoder hidden states husri and h sys i . The attention distributions and context vectors are calculated as described in section 3.1 . In order to merge these two encoders in our framework , the decoder ’ s hidden state s0 is initialized as : s0 = concat ( h usr T , h sys T ) ( 5 ) The pointing mechanism in our model follows the Equation 3 , and we obtain the context vector h∗t : h∗t = concat ( ∑ i ausrti h usr i , ∑ i asysti h sys i ) ( 6 ) 3.2.2 SEMANTIC SLOT SCAFFOLD . We integrate semantic slot scaffold by performing delexicalization on original dialogs . Delexicalization is a common pre-processing step in dialog modeling . Specifically , delexicalization replaces the slot values with its semantic slot name ( e.g . replace 18:00 with [ time ] ) . It is easier for the language modeling to process delexicalized texts , as they have a reduced vocabulary size . But these generated sentences lack the semantic information due to the delexicalization . Some previous dialog system research ignored this issue ( Wen et al. , 2015 ) or completed single delexicalized utterance ( Sharma et al. , 2017 ) as generated response . We propose to perform delexicalization in dialog summary , since delexicalized utterances can simplify dialog modeling . We fill the generated templates with slots with the copy and pointing mechanism . We first train the model with the delexicalized utterance . Attention distribution at over the source tokens instructs the decoder to fill up the slots with lexicalized values : value ( wslot ) = max value ( wi ) : slot ( wi ) =wslot ati ( 7 ) Note thatwslot specifies the tokens that represents the slot name ( e.g . [ hotel place ] , [ time ] ) . Decoder directly copies lexicalized value value ( wi ) conditioned on attention distribution ati . If w is not a slot token , then the probability P ( w ) is calculated as Equation 4 . 3.2.3 DIALOG DOMAIN SCAFFOLD . We integrate dialog domain scaffold through a multi-task framework . Dialog domain indicates different conversation task content , for example , booking hotel , restaurant and taxi in MultiWOZ dataset . Generally , the content in different domains varies so multi-domain task summarization is more difficult than single-domain . We include domain classification as the auxiliary task to incorporate the prior that different domains have different content . Feedback from the domain classification task provides domain specific information for the encoder to learn better representations . For domain classification , we feed the concatenated encoder hidden state through a binary classifier with two linear layers , producing domain probability d. The ith element di in d represents the probability of the ith domain : d = σ ( U ′ ( ReLU ( U [ husrT , h sys T ] + bd ) ) + b ′ d ) ( 8 ) where U , U ′ , bd and b′d are all trainable parameters in the classifier . We denote the loss function of summarization as loss1 and domain classification as loss2 . Assume target word at timestep t is w∗t , loss1 is the arithmetic mean of the negative log likelihood of w∗t over the generated sequence : loss1 = 1 T T∑ t=0 − logP ( w∗t ) ( 9 ) The domain classification task is a multi-label binary classification problem . We use binary cross entropy loss between the ith domain label d̂i and predict probability di for this task : loss2 = 1 |D| |D|∑ i=1 d̂i log di + ( 1− d̂i ) log ( 1− di ) ( 10 ) where |D| is the number of domains . Finally , we reweight the classification loss with hyperparameter λ and the objective function is : loss = loss1 + λloss2 ( 11 )

Summary: The authors propose a new abstractive dialog summarization dataset and task based on the MultiWOZ dataset. Unlike previous work which targets very short descriptions of dialog transcripts (e.g. 'industrial designer presentation'), this paper looks to generate long descriptions of the entire dialog using the prompts in the MultiWOZ task. The authors also extend the pointer generator network of See et al.  (2018) to use speaker, semantic slot and domain information.  They show that this new model (SPNet) outperforms the baseline on existing automatic metrics, on a new metric tuned to measure recall on slots (dubbed CIC), and a thorough human evaluation.