Improving Abstractive Dialogue Summarization with Conversational Structure and Factual Knowledge

Recently , people have been paying more attention to the abstractive dialogue summarization task . Compared with news text , the information flows of the dialogue exchange between at least two interlocutors , which leads to the necessity of capturing long-distance cross-sentence relations . In addition , the generated summaries commonly suffer from fake facts because the key elements of dialogues often scatter in multiple utterances . However , the existing sequence-to-sequence models are difficult to address these issues . Therefore , it is necessary to explore the implicit conversational structure to ensure the richness and faithfulness of generated contents . In this paper , we present a Knowledge Graph Enhanced Dual-Copy network ( KGEDC ) , a novel framework for abstractive dialogue summarization with conversational structure and factual knowledge . We use a sequence encoder to draw local features and a graph encoder to integrate global features via the sparse relational graph self-attention network , complementing each other . Besides , a dual-copy mechanism is also designed in decoding process to force the generation conditioned on both the source text and extracted factual knowledge . The experimental results show that our method produces significantly higher ROUGE scores than most of the baselines on both SAMSum corpus and Automobile Master corpus . Human judges further evaluate that outputs of our model contain more richer and faithful information . 1 INTRODUCTION . Abstractive summarization aims to understand the semantic information of source texts , and generate flexible and concise expressions as summaries , which is more similar to how humans summarize texts . By employing sequence-to-sequence frameworks , some encouraging results have been made in the abstractive summarization of single-speaker documents like news , scientific publications , etc ( Rush et al. , 2015 ; See et al. , 2017 ; Gehrmann et al. , 2018 ; Sharma et al. , 2019 ) . Recently , with the explosive growth of dialogic texts , abstractive dialogue summarization has begun arousing people ’ s interest . Some previous works have attempted to transfer general neural models , which are designed for abstractive summarization of non-dialogic texts , to deal with abstractive dialogue summarization task ( Goo & Chen , 2018 ; Liu et al. , 2019 ; Gliwa et al. , 2019 ) . Different from news texts , dialogues contain dynamic information exchange flows , which are usually informal , verbose and repetitive , sprinkled with false-starts , backchanneling , reconfirmations , hesitations , and speaker interruptions ( Sacks et al. , 1974 ) . Furthermore , utterances are often turned from different interlocutors , which leads to topic drifts and lower information density . Therefore , previous methods are not suitable to generate summaries for dialogues . We argue that the conversational structure and factual knowledge are important to generate informative and succinct summaries . While the neural methods achieve impressive levels of output fluency , they also struggle to produce a coherent order of facts for longer texts ( Wiseman et al. , 2017 ) , and are often unfaithful to input facts , either omitting , repeating , hallucinating or changing facts . Besides , complex events related to the same element often span across multiple utterances , which makes it challenging for sequence-based models to handle utterance-level long-distance dependencies and capture cross-sentence relations . To mitigate these issues , an intuitive way is to model the relationships between textual units within a conversation discourse using graph structures , which can break the sequential positions of textual units and directly connect the related long-distance contents . In this paper , we present the Knowledge Graph Enhanced Dual-Copy network ( KGEDC ) , a novel network specially designed for abstractive dialogue summarization . A graph encoder is proposed to construct the conversational structure in utterance-level under the assumption that utterances represent nodes and edges are semantic relations between them . Specifically , we devise three types of edge labels : speaker dependency , sequential context dependency , and co-occurring keyword dependency . The edges navigate the model from the core fact to other occurrences of that fact , and explore its interactions with other concepts or facts . The sparse dialogue graph only leverages related utterances and filters out redundant details , retaining the capacity to include concise concepts or events . In order to extract sequential features at token-level , a sequence encoder is also used . These two encoders cooperate to express conversational contents via two different granularities , which can effectively capture long-distance cross-sentence dependencies . Moreover , considering that the fact fabrication is a serious problem , encoding existing factual knowledge into the summarization system should be an ideal solution to avoid fake generation . To achieve this goal , we firstly apply the OpenIE tool ( Angeli et al. , 2015 ) and dependency parser tool ( Manning et al. , 2014 ) to extract the factual knowledge in the form of relational tuples : ( subject , predicate , object ) , which construct a knowledge graph . These tuples describe facts and are regarded as the skeletons of dialogues . Next , we design a dual-copy mechanism to copy contents from tokens of the dialogue text and factual knowledge of the knowledge graph in parallel , which would clearly provide the right guidance for summarization . To verify the effectiveness of KGEDC , we carry out automatic and human evaluations on SAMSum corpus and Automobile Master corpus . The experimental results show that our model yield significantly better ROUGE scores ( Lin & Hovy , 2003 ) than all baselines . Human judges further confirm that KGEDC generates more informative summaries with less unfaithful errors than all models without the knowledge graph . 2 RELATED WORK . Graph-based summarization Graph-based approaches have been widely explored in text summarization . Early traditional works make use of inter-sentence cosine similarity to build the connectivity graph like LexRank ( Erkan & Radev , 2004 ) and TextRank ( Mihalcea & Tarau , 2004 ) . Some works further propose discourse inter-sentential relationships to build the Approximate Discourse Graph ( ADG ) ( Yasunaga et al. , 2017 ) and Rhetorical Structure Theory ( RST ) graph ( Xu et al. , 2019 ) . These methods usually rely on external tools and cause error propagation . To avoid these problems , neural models have been applied to improve summarization techniques . Tan et al . ( 2017 ) proposed a graph-based attention mechanism to discover the salient information of a document . Fernandes et al . ( 2019 ) developed a framework to extend existing sequence encoders with a graph component to reason about long-distance relationships . Zhong et al . ( 2019 ) used a Transformer encoder to create a fully-connected graph that learns relations between pairwise sentences . Nevertheless , the factual knowledge implied in dialogues is largely ignored . Cao et al . ( 2017 ) incorporated the fact descriptions as an additional input source text in the attentional sequence-to-sequence framework . Gunel et al . ( 2019 ) employed an entity-aware transformer structure to boost the factual correctness , where the entities come from the Wikidata knowledge graph . In this work , we design a graph encoder based on conversational structure , which uses the sparse relational graph self-attention network to obtain the global features of dialogues . Abstractive dialogue summarization Due to the lack of publicly available resources , the work for dialogue summarization has been rarely studied and it is still in the exploratory stage at present . Some early works benchmarked the abstractive dialogue summarization task using the AMI meeting corpus , which contains a wide range of annotations , including dialogue acts , topic descriptions , etc ( Carletta et al. , 2005 ; Mehdad et al. , 2014 ; Banerjee et al. , 2015 ) . Goo & Chen ( 2018 ) proposed to use the high-level topic descriptions ( e.g . costing evaluation of project process ) as the gold references and leveraged dialogue act signals in a neural summarization model . They assumed that dialogue acts indicated interactive signals and used these information for a better performance . Customer service interaction is also a common form of dialogue , which contains questions of the user Graph . 3 METHODOLOGY . In this section , we introduce the Knowledge Graph Enhanced Dual-Copy network , as displayed in Figure 1 ( a ) . Our framework consists of four modules including a sequence encoder , a graph encoder , a factual knowledge graph , and a dual-copy decoder . Importantly , we first present two types of graphs : Dialogue Graph which constructs conversational structures , and Factual Knowledge Graph which directly extracts fact descriptions from source dialogues . Then , the dual-copy decoder generates faithful summaries by embedding the semantics of both source utterances and factual knowledge . 3.1 SEQUENCE ENCODER . Considering that the contextual information of dialogues usually flows along the sequence , the sequential aspect of the input text is also rich in meaning . Taking the dialogue D as an example , we feed the tokens of it one-by-one into a single-layer bidirectional LSTM , producing a sequence of encoder hidden states { hS1 , hS2 , ... , hSn } . The BiLSTM at the time step i is defined as follows : hSi = BiLSTM ( wi , h S i−1 ) ( 1 ) where wi is the embedding of the i-th token in dialogue , and hSi is the concatenation of the hidden state of a forward LSTM and a backward LSTM . 3.2 GRAPH ENCODER . Given a constructed dialogue graph Gd = ( Vd , Ed ) , Vd = { vd,1 , ... , vd , m } corresponds to m utterances in the dialogue . Ed is a real-value edge weight matrix and ed , ij 6= 0 ( i ∈ { 1 , ... , m } , j ∈ { 1 , ... , m } ) indicates that there is an edge between the i-th utterance and the j-th utterance . We then use graph neural networks to update the representations of all utterances to capture long-distance cross-sentence dependencies , as shown in Figure 2 ( a ) . 3.2.1 NODE INITIALIZATION . We first use a Convolutional Neural Network ( CNN ) with different filter sizes to obtain the representations xj for the j-th utterance vd , j . We also do pos tagging on dialogues with off-the-shelf tools such as Stanford CoreNLP ( Manning et al. , 2014 ) and select nouns , numerals , adjectives , adverbs , and notional verbs as keywords . Each keyword is transformed into a real-valued vector representation εi by looking up the word embedding matrix , which is initialized by a random process . The attention mechanism is then designed to characterize the strength of contextual correlations between utterances and keywords . The representation of the utterance v̂d , j is updated as a weighted sum of these keyword vector ki ( i ∈ { 1 , ... , K } ) : Aij = f ( εi , xj ) = ε T i xj αij = softmaxi ( Aij ) = exp ( Aij ) ∑ k∈Nj exp ( Akj ) v̂d , j = σ ( K∑ i=1 αijWaεi ) ( 2 ) where K is the number of keywords in each dialogue , Wa is a trainable weight , and αij is the attention coefficient between εi and xj . 3.2.2 EDGE INITIALIZATION . If we hypothesize that each utterance is contextually dependent on all the other utterances in the dialogue , then a fully connected graph would be constructed . However , this leads to a huge amount of computation . Therefore , we adopt a strategy to construct the edges of the graph , which relies on the various semantic relations among utterances . We define three types of edge labels : speaker dependency , sequential context dependency , co-occurring keyword dependency ( A.1 ) . Speaker dependency : The relation depends on where the same speaker appears in the dialogue . In other words , if the utterances belong to the same speaker , we will set an edge between them . Sequential context dependency : The relation describes the sequential utterances that occur within a fixed-size sliding window . In this scenario , each utterance node vd , i has an edge with the immediate p utterances of the past vd , i−1 , vd , i−2 , ... , vd , i−p , and f utterances of the future : vd , i+1 , vd , i+2 , ... , vd , i+f . co-occurring keyword dependency : The relation means that all utterances containing the same keyword are connected . We further change the existing undirected edges into bidirectional edges and add self-loops to enhance the information flow .

Summary: This paper proposes a knowledge graph enhanced network to improve abstractive dialog summarization with graphs constructed from the dialog structure and factual knowledge. The dialog graph is composed of utterances as nodes and 3 heuristic types of edges (such as utterances of the same speaker, adjacent utterances). The factual graph is constructed via openIE and dependency parsing, which the authors claim are complementary as the triplets (results of openIE) are not always available.