Improving Abstractive Dialogue Summarization with Conversational Structure and Factual Knowledge

Recently , people have been paying more attention to the abstractive dialogue summarization task . Compared with news text , the information flows of the dialogue exchange between at least two interlocutors , which leads to the necessity of capturing long-distance cross-sentence relations . In addition , the generated summaries commonly suffer from fake facts because the key elements of dialogues often scatter in multiple utterances . However , the existing sequence-to-sequence models are difficult to address these issues . Therefore , it is necessary to explore the implicit conversational structure to ensure the richness and faithfulness of generated contents . In this paper , we present a Knowledge Graph Enhanced Dual-Copy network ( KGEDC ) , a novel framework for abstractive dialogue summarization with conversational structure and factual knowledge . We use a sequence encoder to draw local features and a graph encoder to integrate global features via the sparse relational graph self-attention network , complementing each other . Besides , a dual-copy mechanism is also designed in decoding process to force the generation conditioned on both the source text and extracted factual knowledge . The experimental results show that our method produces significantly higher ROUGE scores than most of the baselines on both SAMSum corpus and Automobile Master corpus . Human judges further evaluate that outputs of our model contain more richer and faithful information . 1 INTRODUCTION . Abstractive summarization aims to understand the semantic information of source texts , and generate flexible and concise expressions as summaries , which is more similar to how humans summarize texts . By employing sequence-to-sequence frameworks , some encouraging results have been made in the abstractive summarization of single-speaker documents like news , scientific publications , etc ( Rush et al. , 2015 ; See et al. , 2017 ; Gehrmann et al. , 2018 ; Sharma et al. , 2019 ) . Recently , with the explosive growth of dialogic texts , abstractive dialogue summarization has begun arousing people ’ s interest . Some previous works have attempted to transfer general neural models , which are designed for abstractive summarization of non-dialogic texts , to deal with abstractive dialogue summarization task ( Goo & Chen , 2018 ; Liu et al. , 2019 ; Gliwa et al. , 2019 ) . Different from news texts , dialogues contain dynamic information exchange flows , which are usually informal , verbose and repetitive , sprinkled with false-starts , backchanneling , reconfirmations , hesitations , and speaker interruptions ( Sacks et al. , 1974 ) . Furthermore , utterances are often turned from different interlocutors , which leads to topic drifts and lower information density . Therefore , previous methods are not suitable to generate summaries for dialogues . We argue that the conversational structure and factual knowledge are important to generate informative and succinct summaries . While the neural methods achieve impressive levels of output fluency , they also struggle to produce a coherent order of facts for longer texts ( Wiseman et al. , 2017 ) , and are often unfaithful to input facts , either omitting , repeating , hallucinating or changing facts . Besides , complex events related to the same element often span across multiple utterances , which makes it challenging for sequence-based models to handle utterance-level long-distance dependencies and capture cross-sentence relations . To mitigate these issues , an intuitive way is to model the relationships between textual units within a conversation discourse using graph structures , which can break the sequential positions of textual units and directly connect the related long-distance contents . In this paper , we present the Knowledge Graph Enhanced Dual-Copy network ( KGEDC ) , a novel network specially designed for abstractive dialogue summarization . A graph encoder is proposed to construct the conversational structure in utterance-level under the assumption that utterances represent nodes and edges are semantic relations between them . Specifically , we devise three types of edge labels : speaker dependency , sequential context dependency , and co-occurring keyword dependency . The edges navigate the model from the core fact to other occurrences of that fact , and explore its interactions with other concepts or facts . The sparse dialogue graph only leverages related utterances and filters out redundant details , retaining the capacity to include concise concepts or events . In order to extract sequential features at token-level , a sequence encoder is also used . These two encoders cooperate to express conversational contents via two different granularities , which can effectively capture long-distance cross-sentence dependencies . Moreover , considering that the fact fabrication is a serious problem , encoding existing factual knowledge into the summarization system should be an ideal solution to avoid fake generation . To achieve this goal , we firstly apply the OpenIE tool ( Angeli et al. , 2015 ) and dependency parser tool ( Manning et al. , 2014 ) to extract the factual knowledge in the form of relational tuples : ( subject , predicate , object ) , which construct a knowledge graph . These tuples describe facts and are regarded as the skeletons of dialogues . Next , we design a dual-copy mechanism to copy contents from tokens of the dialogue text and factual knowledge of the knowledge graph in parallel , which would clearly provide the right guidance for summarization . To verify the effectiveness of KGEDC , we carry out automatic and human evaluations on SAMSum corpus and Automobile Master corpus . The experimental results show that our model yield significantly better ROUGE scores ( Lin & Hovy , 2003 ) than all baselines . Human judges further confirm that KGEDC generates more informative summaries with less unfaithful errors than all models without the knowledge graph . 2 RELATED WORK . Graph-based summarization Graph-based approaches have been widely explored in text summarization . Early traditional works make use of inter-sentence cosine similarity to build the connectivity graph like LexRank ( Erkan & Radev , 2004 ) and TextRank ( Mihalcea & Tarau , 2004 ) . Some works further propose discourse inter-sentential relationships to build the Approximate Discourse Graph ( ADG ) ( Yasunaga et al. , 2017 ) and Rhetorical Structure Theory ( RST ) graph ( Xu et al. , 2019 ) . These methods usually rely on external tools and cause error propagation . To avoid these problems , neural models have been applied to improve summarization techniques . Tan et al . ( 2017 ) proposed a graph-based attention mechanism to discover the salient information of a document . Fernandes et al . ( 2019 ) developed a framework to extend existing sequence encoders with a graph component to reason about long-distance relationships . Zhong et al . ( 2019 ) used a Transformer encoder to create a fully-connected graph that learns relations between pairwise sentences . Nevertheless , the factual knowledge implied in dialogues is largely ignored . Cao et al . ( 2017 ) incorporated the fact descriptions as an additional input source text in the attentional sequence-to-sequence framework . Gunel et al . ( 2019 ) employed an entity-aware transformer structure to boost the factual correctness , where the entities come from the Wikidata knowledge graph . In this work , we design a graph encoder based on conversational structure , which uses the sparse relational graph self-attention network to obtain the global features of dialogues . Abstractive dialogue summarization Due to the lack of publicly available resources , the work for dialogue summarization has been rarely studied and it is still in the exploratory stage at present . Some early works benchmarked the abstractive dialogue summarization task using the AMI meeting corpus , which contains a wide range of annotations , including dialogue acts , topic descriptions , etc ( Carletta et al. , 2005 ; Mehdad et al. , 2014 ; Banerjee et al. , 2015 ) . Goo & Chen ( 2018 ) proposed to use the high-level topic descriptions ( e.g . costing evaluation of project process ) as the gold references and leveraged dialogue act signals in a neural summarization model . They assumed that dialogue acts indicated interactive signals and used these information for a better performance . Customer service interaction is also a common form of dialogue , which contains questions of the user Graph . 3 METHODOLOGY . In this section , we introduce the Knowledge Graph Enhanced Dual-Copy network , as displayed in Figure 1 ( a ) . Our framework consists of four modules including a sequence encoder , a graph encoder , a factual knowledge graph , and a dual-copy decoder . Importantly , we first present two types of graphs : Dialogue Graph which constructs conversational structures , and Factual Knowledge Graph which directly extracts fact descriptions from source dialogues . Then , the dual-copy decoder generates faithful summaries by embedding the semantics of both source utterances and factual knowledge . 3.1 SEQUENCE ENCODER . Considering that the contextual information of dialogues usually flows along the sequence , the sequential aspect of the input text is also rich in meaning . Taking the dialogue D as an example , we feed the tokens of it one-by-one into a single-layer bidirectional LSTM , producing a sequence of encoder hidden states { hS1 , hS2 , ... , hSn } . The BiLSTM at the time step i is defined as follows : hSi = BiLSTM ( wi , h S i−1 ) ( 1 ) where wi is the embedding of the i-th token in dialogue , and hSi is the concatenation of the hidden state of a forward LSTM and a backward LSTM . 3.2 GRAPH ENCODER . Given a constructed dialogue graph Gd = ( Vd , Ed ) , Vd = { vd,1 , ... , vd , m } corresponds to m utterances in the dialogue . Ed is a real-value edge weight matrix and ed , ij 6= 0 ( i ∈ { 1 , ... , m } , j ∈ { 1 , ... , m } ) indicates that there is an edge between the i-th utterance and the j-th utterance . We then use graph neural networks to update the representations of all utterances to capture long-distance cross-sentence dependencies , as shown in Figure 2 ( a ) . 3.2.1 NODE INITIALIZATION . We first use a Convolutional Neural Network ( CNN ) with different filter sizes to obtain the representations xj for the j-th utterance vd , j . We also do pos tagging on dialogues with off-the-shelf tools such as Stanford CoreNLP ( Manning et al. , 2014 ) and select nouns , numerals , adjectives , adverbs , and notional verbs as keywords . Each keyword is transformed into a real-valued vector representation εi by looking up the word embedding matrix , which is initialized by a random process . The attention mechanism is then designed to characterize the strength of contextual correlations between utterances and keywords . The representation of the utterance v̂d , j is updated as a weighted sum of these keyword vector ki ( i ∈ { 1 , ... , K } ) : Aij = f ( εi , xj ) = ε T i xj αij = softmaxi ( Aij ) = exp ( Aij ) ∑ k∈Nj exp ( Akj ) v̂d , j = σ ( K∑ i=1 αijWaεi ) ( 2 ) where K is the number of keywords in each dialogue , Wa is a trainable weight , and αij is the attention coefficient between εi and xj . 3.2.2 EDGE INITIALIZATION . If we hypothesize that each utterance is contextually dependent on all the other utterances in the dialogue , then a fully connected graph would be constructed . However , this leads to a huge amount of computation . Therefore , we adopt a strategy to construct the edges of the graph , which relies on the various semantic relations among utterances . We define three types of edge labels : speaker dependency , sequential context dependency , co-occurring keyword dependency ( A.1 ) . Speaker dependency : The relation depends on where the same speaker appears in the dialogue . In other words , if the utterances belong to the same speaker , we will set an edge between them . Sequential context dependency : The relation describes the sequential utterances that occur within a fixed-size sliding window . In this scenario , each utterance node vd , i has an edge with the immediate p utterances of the past vd , i−1 , vd , i−2 , ... , vd , i−p , and f utterances of the future : vd , i+1 , vd , i+2 , ... , vd , i+f . co-occurring keyword dependency : The relation means that all utterances containing the same keyword are connected . We further change the existing undirected edges into bidirectional edges and add self-loops to enhance the information flow .

Summary: This paper proposes to improve dialogue summarization by encoding the text with a sequential encoder (for token-level contextualization) and a graph encoder (for long-distance and semantic contextualization). A KG is built and considered to be a surrogate for "factual knowledge". A dual-copy mechanism is used while decoding in the hope that direct access to this factual knowledge will enhance the faithfulness of the generated summaries.