Abstractive Dialog Summarization with Semantic Scaffolds

1 INTRODUCTION . Summarization aims to condense a piece of text to a shorter version , retaining the critical information . On dialogs , summarization has various promising applications in the real world . For instance , the automatic doctor-patient interaction summary can save doctors ’ massive amount of time used for filling medical records . There is also a general demand for summarizing meetings in order to track project progress in the industry . Generally , multi-party conversations with interactive communication are more difficult to summarize than single-speaker documents . Hence , dialog summarization will be a potential field in summarization track . There are two types of summarization : extractive and abstractive . Extractive summarization selects sentences or phrases directly from the source text and merges them to a summary , while abstractive summarization attempts to generate novel expressions to condense information . Previous dialog summarization research mostly study extractive summarization ( Murray et al. , 2005 ; Maskey & Hirschberg , 2005 ) . Extractive methods merge selected important utterances from a dialog to form summary . Because dialogs are highly dependant on their histories , it is difficult to produce coherent discourses with a set of non-consecutive conversation turns . Therefore , extractive summarization is not the best approach to summarize dialogs . However , most modern abstractive methods focus on single-speaker documents rather than dialogs due to the lack of dialog summarization corpora . Popular abstractive summarization dataset like CNN/Daily Mail ( Hermann et al. , 2015 ) is on news documents . AMI meeting corpus ( McCowan et al. , 2005 ) is the common benchmark , but it only has extractive summary . In this work , we introduce a dataset for abstractive dialog summarization based on MultiWOZ ( Budzianowski et al. , 2018 ) . Seq2Seq models such as Pointer-Generator ( See et al. , 2017 ) have achieved high-quality summaries of news document . However , directly applying a news summarizer to dialog results in two drawbacks : informative entities such as place name are difficult to capture precisely and contents in different domains are summarized unequally . To address these problems , we propose Scaffold Pointer Network ( SPNet ) . SPNet incorporates three types of semantic scaffolds in dialog : speaker role , semantic slot , and dialog domain . Firstly , SPNet adapts separate encoder to attentional Seq2Seq framework , producing distinct semantic representations for different speaker roles . Then , our method inputs delexicalized utterances for producing delexicalized summary , and fills in slot values to generate complete summary . Finally , we incorporate dialog domain scaffold by jointly optimizing dialog domain classification task along with the summarization task . We evaluate SPNet with both automatic and human evaluation metrics on MultiWOZ . SPNet outperforms Pointer-Generator ( See et al. , 2017 ) and Transformer ( Vaswani et al. , 2017 ) on all the metrics . 2 RELATED WORK . Rush et al . ( 2015 ) first applied modern neural models to abstractive summarization . Their approach is based on Seq2Seq framework ( Sutskever et al. , 2014 ) and attention mechanism ( Bahdanau et al. , 2015 ) , achieving state-of-the-art results on Gigaword and DUC-2004 dataset . Gu et al . ( 2016 ) proposed copy mechanism in summarization , demonstrating its effectiveness by combining the advantages of extractive and abstractive approach . See et al . ( 2017 ) applied pointing ( Vinyals et al. , 2015 ) as copy mechanism and use coverage mechanism ( Tu et al. , 2016 ) to discourage repetition . Most recently , reinforcement learning ( RL ) has been employed in abstractive summarization . RL-based approaches directly optimize the objectives of summarization ( Ranzato et al. , 2016 ; Celikyilmaz et al. , 2018 ) . However , deep reinforcement learning approaches are difficult to train and more prone to exposure bias ( Bahdanau et al. , 2017 ) . Recently , pre-training methods are popular in NLP applications . BERT ( Devlin et al. , 2018 ) and GPT ( Radford et al. , 2018 ) have achieved state-of-the-art performance in many tasks , including summarization . For instance , Zhang et al . ( 2019 ) proposed a method to pre-train hierarchical document encoder for extractive summarization . Hoang et al . ( 2019 ) proposed two strategies to incorporate a pre-trained model ( GPT ) to perform the abstractive summarizer and achieved a better performance . However , there has not been much research on adapting pre-trained models to dialog summarization . Dialog summarization , specifically meeting summarization , has been studied extensively . Previous work generally focused on statistical machine learning methods in extractive dialog summarization : Galley ( 2006 ) used skip-chain conditional random fields ( CRFs ) ( Lafferty et al. , 2001 ) as a ranking method in extractive meeting summarization . Wang & Cardie ( 2013 ) compared support vector machines ( SVMs ) ( Cortes & Vapnik , 1995 ) with LDA-based topic models ( Blei et al. , 2003 ) for producing decision summaries . However , abstractive dialog summarization was less explored due to the lack of a suitable benchmark . Recent work ( Wang & Cardie , 2016 ; Goo & Chen , 2018 ; Pan et al. , 2018 ) created abstractive dialog summary benchmarks with existing dialog corpus . Goo & Chen ( 2018 ) annotated topic descriptions in AMI meeting corpus as the summary . However , topics they defined are coarse , such as “ industrial designer presentation ” . They also proposed a model with a sentence-gated mechanism incorporating dialog acts to perform abstractive summarization . Moreover , Li et al . ( 2019 ) first built a model to summarize audio-visual meeting data with an abstractive method . However , previous work has not investigated the utilization of semantic patterns in dialog , so we explore it in-depth in our work . 3 PROPOSED METHOD . As discussed above , state-of-the-art document summarizers are not applicable in conversation settings . We propose Scaffold Pointer Network ( SPNet ) based on Pointer-Generator ( See et al. , 2017 ) . SPNet incorporates three types of semantic scaffolds to improve abstractive dialog summarization : speaker role , semantic slot and dialog domain . 3.1 BACKGROUND . We first introduce Pointer-Generator ( See et al. , 2017 ) . It is a hybrid model of the typical Seq2Seq attention model ( Nallapati et al. , 2016 ) and pointer network ( Vinyals et al. , 2015 ) . Seq2Seq framework encodes source sequence and generates the target sequence with the decoder . The input sequence is fed into the encoder token by token , producing the encoder hidden states hi in each encoding step . The decoder receives word embedding of the previous word and generates a distribution to decide the target element in this step , retaining decoder hidden states st . In Pointer-Generator , attention distribution at is computed as in Bahdanau et al . ( 2015 ) : eti = v T tanh ( Whhi +Wsst + battn ) at = softmax ( et ) ( 1 ) where Wh , Ws , v and battn are all learnable parameters . With the attention distribution at , context vector h∗t is computed as the weighted sum of encoder ’ s hidden states . Context vector is regarded as the attentional information in the source text : h∗t = ∑ i atihi ( 2 ) Pointer-Generator differs from typical Seq2Seq attention model in the generation process . The pointing mechanism combines copying words directly from the source text with generating words from a fixed vocabulary . Generation probability pgen is calculated as “ a soft switch ” to choose from copy and generation : pgen = σ ( wTh∗h ∗ t + w T s st + w T x xt + bptr ) ( 3 ) where xt is the decoder input , wh∗ , ws , wx and bptr are all learnable parameters . σ is sigmoid function , so the generation probability pgen has a range of [ 0 , 1 ] . The ability to select from copy and generation corresponds to a dynamic vocabulary . Pointer network forms an extended vocabulary for the copied tokens , including all the out-of-vocabulary ( OOV ) words appeared in the source text . The final probability distribution P ( w ) on extended vocabulary is computed as follows : Pvocab = softmax ( V ′ ( V [ st , h ∗ t ] + b ) + b ′ ) P ( w ) = pgenPvocab ( w ) + ( 1− pgen ) ∑ i : wi=w ati ( 4 ) where Pvocab is the distribution on the original vocabulary , V ′ , V , b and b′ are learnable parameters used to calculate such distribution . 3.2 SCAFFOLD POINTER NETWORK ( SPNET ) . Our Scaffold Pointer Network ( depicted in Figure 1 ) is based on Pointer-Generator ( See et al. , 2017 ) . The contribution of SPNet is three-fold : separate encoding for different roles , incorporating semantic slot scaffold and dialog domain scaffold . 3.2.1 SPEAKER ROLE SCAFFOLD . Our encoder-decoder framework employs separate encoding for different speakers in the dialog . User utterances xusrt and system utterances x sys t are fed into a user encoder and a system encoder separately to obtain encoder hidden states husri and h sys i . The attention distributions and context vectors are calculated as described in section 3.1 . In order to merge these two encoders in our framework , the decoder ’ s hidden state s0 is initialized as : s0 = concat ( h usr T , h sys T ) ( 5 ) The pointing mechanism in our model follows the Equation 3 , and we obtain the context vector h∗t : h∗t = concat ( ∑ i ausrti h usr i , ∑ i asysti h sys i ) ( 6 ) 3.2.2 SEMANTIC SLOT SCAFFOLD . We integrate semantic slot scaffold by performing delexicalization on original dialogs . Delexicalization is a common pre-processing step in dialog modeling . Specifically , delexicalization replaces the slot values with its semantic slot name ( e.g . replace 18:00 with [ time ] ) . It is easier for the language modeling to process delexicalized texts , as they have a reduced vocabulary size . But these generated sentences lack the semantic information due to the delexicalization . Some previous dialog system research ignored this issue ( Wen et al. , 2015 ) or completed single delexicalized utterance ( Sharma et al. , 2017 ) as generated response . We propose to perform delexicalization in dialog summary , since delexicalized utterances can simplify dialog modeling . We fill the generated templates with slots with the copy and pointing mechanism . We first train the model with the delexicalized utterance . Attention distribution at over the source tokens instructs the decoder to fill up the slots with lexicalized values : value ( wslot ) = max value ( wi ) : slot ( wi ) =wslot ati ( 7 ) Note thatwslot specifies the tokens that represents the slot name ( e.g . [ hotel place ] , [ time ] ) . Decoder directly copies lexicalized value value ( wi ) conditioned on attention distribution ati . If w is not a slot token , then the probability P ( w ) is calculated as Equation 4 . 3.2.3 DIALOG DOMAIN SCAFFOLD . We integrate dialog domain scaffold through a multi-task framework . Dialog domain indicates different conversation task content , for example , booking hotel , restaurant and taxi in MultiWOZ dataset . Generally , the content in different domains varies so multi-domain task summarization is more difficult than single-domain . We include domain classification as the auxiliary task to incorporate the prior that different domains have different content . Feedback from the domain classification task provides domain specific information for the encoder to learn better representations . For domain classification , we feed the concatenated encoder hidden state through a binary classifier with two linear layers , producing domain probability d. The ith element di in d represents the probability of the ith domain : d = σ ( U ′ ( ReLU ( U [ husrT , h sys T ] + bd ) ) + b ′ d ) ( 8 ) where U , U ′ , bd and b′d are all trainable parameters in the classifier . We denote the loss function of summarization as loss1 and domain classification as loss2 . Assume target word at timestep t is w∗t , loss1 is the arithmetic mean of the negative log likelihood of w∗t over the generated sequence : loss1 = 1 T T∑ t=0 − logP ( w∗t ) ( 9 ) The domain classification task is a multi-label binary classification problem . We use binary cross entropy loss between the ith domain label d̂i and predict probability di for this task : loss2 = 1 |D| |D|∑ i=1 d̂i log di + ( 1− d̂i ) log ( 1− di ) ( 10 ) where |D| is the number of domains . Finally , we reweight the classification loss with hyperparameter λ and the objective function is : loss = loss1 + λloss2 ( 11 )

Summary: Authors proposed an enhanced Pointer-Generator model called SPNet. The key difference between SPNet and PG are the separate handling or using of speaker role, semantic slot and domain labels. Authors also proposed a new metrics called Critical Information Completeness (CIC) to address ROUGE's weakness in assessing if key information is missing in the output.