Year: 2020

Abstract:
Recently, people have been paying more attention to the abstractive dialogue summarization task. Compared with news text, the information flows of the dialogue exchange between at least two interlocutors, which leads to the necessity of capturing long-distance cross-sentence relations. In addition, the generated summaries commonly suffer from fake facts because the key elements of dialogues often scatter in multiple utterances. However, the existing sequence-to-sequence models are difficult to address these issues. Therefore, it is necessary to explore the implicit conversational structure to ensure the richness and faithfulness of generated contents. In this paper, we present a Knowledge Graph Enhanced Dual-Copy network (KGEDC), a novel framework for abstractive dialogue summarization with conversational structure and factual knowledge. We use a sequence encoder to draw local features and a graph encoder to integrate global features via the sparse relational graph self-attention network, complementing each other. Besides, a dual-copy mechanism is also designed in decoding process to force the generation conditioned on both the source text and extracted factual knowledge. The experimental results show that our method produces significantly higher ROUGE scores than most of the baselines on both SAMSum corpus and Automobile Master corpus. Human judges further evaluate that outputs of our model contain more richer and faithful information.

text: Recently, people have been paying more attention to the abstractive dialogue summarization task. Compared with news text, the information flows of the dialogue exchange between at least two interlocutors, which leads to the necessity of capturing long-distance cross-sentence relations. In addition, the generated summaries commonly suffer from fake facts because the key elements of dialogues often scatter in multiple utterances. However, the existing sequence-to-sequence models are difficult to address these issues. Therefore, it is necessary to explore the implicit conversational structure to ensure the richness and faithfulness of generated contents. In this paper, we present a Knowledge Graph Enhanced Dual-Copy network (KGEDC), a novel framework for abstractive dialogue summarization with conversational structure and factual knowledge. We use a sequence encoder to draw local features and a graph encoder to integrate global features via the sparse relational graph self-attention network, complementing each other. Besides, a dual-copy mechanism is also designed in decoding process to force the generation conditioned on both the source text and extracted factual knowledge. The experimental results show that our method produces significantly higher ROUGE scores than most of the baselines on both SAMSum corpus and Automobile Master corpus. Human judges further evaluate that outputs of our model contain more richer and faithful information.

heading: 1 INTRODUCTION
text: Abstractive summarization aims to understand the semantic information of source texts, and generate flexible and concise expressions as summaries, which is more similar to how humans summarize texts. By employing sequence-to-sequence frameworks, some encouraging results have been made in the abstractive summarization of single-speaker documents like news, scientific publications, etc (Rush et al., 2015; See et al., 2017; Gehrmann et al., 2018; Sharma et al., 2019). Recently, with the explosive growth of dialogic texts, abstractive dialogue summarization has begun arousing people’s interest. Some previous works have attempted to transfer general neural models, which are designed for abstractive summarization of non-dialogic texts, to deal with abstractive dialogue summarization task (Goo & Chen, 2018; Liu et al., 2019; Gliwa et al., 2019).
Different from news texts, dialogues contain dynamic information exchange flows, which are usually informal, verbose and repetitive, sprinkled with false-starts, backchanneling, reconfirmations, hesitations, and speaker interruptions (Sacks et al., 1974). Furthermore, utterances are often turned from different interlocutors, which leads to topic drifts and lower information density. Therefore, previous methods are not suitable to generate summaries for dialogues. We argue that the conversational structure and factual knowledge are important to generate informative and succinct summaries. While the neural methods achieve impressive levels of output fluency, they also struggle to produce a coherent order of facts for longer texts (Wiseman et al., 2017), and are often unfaithful to input facts, either omitting, repeating, hallucinating or changing facts. Besides, complex events related to the same element often span across multiple utterances, which makes it challenging for sequence-based models to handle utterance-level long-distance dependencies and capture cross-sentence relations.
To mitigate these issues, an intuitive way is to model the relationships between textual units within a conversation discourse using graph structures, which can break the sequential positions of textual units and directly connect the related long-distance contents. In this paper, we present the Knowledge Graph Enhanced Dual-Copy network (KGEDC), a novel network specially designed for abstractive dialogue summarization. A graph encoder is proposed to construct the conversational structure in utterance-level under the assumption that utterances represent nodes and edges are semantic relations between them. Specifically, we devise three types of edge labels: speaker dependency, sequential context dependency, and co-occurring keyword dependency. The edges navigate the model from the core fact to other occurrences of that fact, and explore its interactions with other concepts or facts. The sparse dialogue graph only leverages related utterances and filters out redundant details, retaining the capacity to include concise concepts or events. In order to extract sequential features at token-level, a sequence encoder is also used. These two encoders cooperate to express conversational contents via two different granularities, which can effectively capture long-distance cross-sentence dependencies.
Moreover, considering that the fact fabrication is a serious problem, encoding existing factual knowledge into the summarization system should be an ideal solution to avoid fake generation. To achieve this goal, we firstly apply the OpenIE tool (Angeli et al., 2015) and dependency parser tool (Manning et al., 2014) to extract the factual knowledge in the form of relational tuples: (subject, predicate, object), which construct a knowledge graph. These tuples describe facts and are regarded as the skeletons of dialogues. Next, we design a dual-copy mechanism to copy contents from tokens of the dialogue text and factual knowledge of the knowledge graph in parallel, which would clearly provide the right guidance for summarization.
To verify the effectiveness of KGEDC, we carry out automatic and human evaluations on SAMSum corpus and Automobile Master corpus. The experimental results show that our model yield significantly better ROUGE scores (Lin & Hovy, 2003) than all baselines. Human judges further confirm that KGEDC generates more informative summaries with less unfaithful errors than all models without the knowledge graph.

heading: 2 RELATED WORK
text: Graph-based summarization Graph-based approaches have been widely explored in text summarization. Early traditional works make use of inter-sentence cosine similarity to build the connectivity graph like LexRank (Erkan & Radev, 2004) and TextRank (Mihalcea & Tarau, 2004). Some works further propose discourse inter-sentential relationships to build the Approximate Discourse Graph (ADG) (Yasunaga et al., 2017) and Rhetorical Structure Theory (RST) graph (Xu et al., 2019). These methods usually rely on external tools and cause error propagation. To avoid these problems, neural models have been applied to improve summarization techniques. Tan et al. (2017) proposed a graph-based attention mechanism to discover the salient information of a document. Fernandes et al. (2019) developed a framework to extend existing sequence encoders with a graph component to reason about long-distance relationships. Zhong et al. (2019) used a Transformer encoder to create a fully-connected graph that learns relations between pairwise sentences. Nevertheless, the factual knowledge implied in dialogues is largely ignored. Cao et al. (2017) incorporated the fact descriptions as an additional input source text in the attentional sequence-to-sequence framework. Gunel et al. (2019) employed an entity-aware transformer structure to boost the factual correctness, where the entities come from the Wikidata knowledge graph. In this work, we design a graph encoder based on conversational structure, which uses the sparse relational graph self-attention network to obtain the global features of dialogues.
Abstractive dialogue summarization Due to the lack of publicly available resources, the work for dialogue summarization has been rarely studied and it is still in the exploratory stage at present. Some early works benchmarked the abstractive dialogue summarization task using the AMI meeting corpus, which contains a wide range of annotations, including dialogue acts, topic descriptions, etc (Carletta et al., 2005; Mehdad et al., 2014; Banerjee et al., 2015). Goo & Chen (2018) proposed to use the high-level topic descriptions (e.g. costing evaluation of project process) as the gold references and leveraged dialogue act signals in a neural summarization model. They assumed that dialogue acts indicated interactive signals and used these information for a better performance. Customer service interaction is also a common form of dialogue, which contains questions of the user

heading: Graph

heading: 3 METHODOLOGY
text: In this section, we introduce the Knowledge Graph Enhanced Dual-Copy network, as displayed in Figure 1 (a). Our framework consists of four modules including a sequence encoder, a graph encoder, a factual knowledge graph, and a dual-copy decoder. Importantly, we first present two types of graphs: Dialogue Graph which constructs conversational structures, and Factual Knowledge Graph which directly extracts fact descriptions from source dialogues. Then, the dual-copy decoder generates faithful summaries by embedding the semantics of both source utterances and factual knowledge.

heading: 3.1 SEQUENCE ENCODER
text: Considering that the contextual information of dialogues usually flows along the sequence, the sequential aspect of the input text is also rich in meaning. Taking the dialogue D as an example, we feed the tokens of it one-by-one into a single-layer bidirectional LSTM, producing a sequence of encoder hidden states {hS1 , hS2 , ..., hSn}. The BiLSTM at the time step i is defined as follows:
hSi = BiLSTM(wi, h S i−1) (1)
where wi is the embedding of the i-th token in dialogue, and hSi is the concatenation of the hidden state of a forward LSTM and a backward LSTM.

heading: 3.2 GRAPH ENCODER
text: Given a constructed dialogue graph Gd = (Vd, Ed), Vd = {vd,1, ..., vd,m} corresponds to m utterances in the dialogue. Ed is a real-value edge weight matrix and ed,ij 6= 0 (i ∈ {1, ...,m}, j ∈ {1, ...,m}) indicates that there is an edge between the i-th utterance and the j-th utterance. We then use graph neural networks to update the representations of all utterances to capture long-distance cross-sentence dependencies, as shown in Figure 2 (a).

heading: 3.2.1 NODE INITIALIZATION
text: We first use a Convolutional Neural Network (CNN) with different filter sizes to obtain the representations xj for the j-th utterance vd,j . We also do pos tagging on dialogues with off-the-shelf tools such as Stanford CoreNLP (Manning et al., 2014) and select nouns, numerals, adjectives, adverbs, and notional verbs as keywords. Each keyword is transformed into a real-valued vector representation εi by looking up the word embedding matrix, which is initialized by a random process. The attention mechanism is then designed to characterize the strength of contextual correlations between utterances and keywords. The representation of the utterance v̂d,j is updated as a weighted sum of these keyword vector ki (i ∈ {1, ...,K}):
Aij = f(εi, xj) = ε T i xj αij = softmaxi(Aij) = exp(Aij)∑
k∈Nj exp(Akj)
v̂d,j = σ( K∑ i=1 αijWaεi)
(2)
where K is the number of keywords in each dialogue, Wa is a trainable weight, and αij is the attention coefficient between εi and xj .

heading: 3.2.2 EDGE INITIALIZATION
text: If we hypothesize that each utterance is contextually dependent on all the other utterances in the dialogue, then a fully connected graph would be constructed. However, this leads to a huge amount of computation. Therefore, we adopt a strategy to construct the edges of the graph, which relies on the various semantic relations among utterances. We define three types of edge labels: speaker dependency, sequential context dependency, co-occurring keyword dependency (A.1).
Speaker dependency: The relation depends on where the same speaker appears in the dialogue. In other words, if the utterances belong to the same speaker, we will set an edge between them.
Sequential context dependency: The relation describes the sequential utterances that occur within a fixed-size sliding window. In this scenario, each utterance node vd,i has an edge with the immediate p utterances of the past vd,i−1, vd,i−2, ..., vd,i−p, and f utterances of the future: vd,i+1, vd,i+2, ..., vd,i+f .
co-occurring keyword dependency: The relation means that all utterances containing the same keyword are connected.
We further change the existing undirected edges into bidirectional edges and add self-loops to enhance the information flow.

heading: 3.2.3 ITERATIVE REFINEMENT
text: To modularize architecture design, multiple blocks, which all consist of the sparse relational graph self-attention layer and the linear combination layer, are stacked to capture long-distance dependencies missed by sequence encoder for better summarization.
Sparse Relational Graph Self-Attention Layer Different parts of the dialogue history have distinct levels of importance that may influence the summary generation process. We use a sparse attention mechanism to focus more on the salient utterances. The full self-attention operation in Transformer (Vaswani et al., 2017) captures the interactions between two arbitrary positions of a single sequence. However, our sparse self-attention operation only calculates the similarities between two connected nodes in the graph and masks the irrelevant edges, which reduces the computation amount to a certain extent. In a single sparse graph attention layer, a node in the graph attends over the local information from 1-hop neighbors, which assigns different attention weights to different neighboring edges. These weights can be learned by the model in an end-to-end fashion. Although the sparse graph attention mechanism aggregates the representations of neighborhood nodes along the dependency paths, this process fails to take semantic relations into consideration, which may lose some important dependency information. Intuitively, neighborhood nodes with different dependency relations should have different influences. We extend the sparse graph attention mechanism with the additional edge labels, which can learn better inter-sentence relations. Specifically, we first map the edge labels into vector representations R, and then this layer is designed as follows:
headli = Attention(QW Q,l i ,KW K,l i , V W V,l i , RW R,l i ) Attention(Q,K, V,R) = softmax( Q×K√
d +R)(V +R)
gl = [headl1; ...;head l H ]W o,l
(3)
where W o, WQi , W V i , W K i , and W R i are weight matrices, H is the head number, and d is the dimension of utterance node features v̂d ∈ Rm×d. In the first block, Q, K, and V are v̂d. For the following blocks l, they are the linear combination layer output vector zl−1 ∈ Rm×d of block l− 1. Linear Combination Layer This layer contains two linear combinations with aReLU activation in between just as Transformer (Vaswani et al., 2017). Formally, the output of the linear transformation layer is defined as:
zl = ReLU(glwl1 + b l 1)w l 2 + b l 2 (4)
where w1, and w2 are weight matrices. b1, and b2 are bias vectors.
After M identical blocks, we take mean pooling over all nodes and obtain the final representation of the dialogue graph, hG = 1m ∑m i=1 z M i . We then concatenate the representation of sequence encoder hSn and graph encoder h G to get the initial state of the decoder s0 = [hSn ;h G].

heading: 3.3 FACTUAL KNOWLEDGE GRAPH CONSTRUCTION
text: To construct a factual knowledge graph from an input dialogue, we leverage the Open Information Extraction (OpenIE) tools (Angeli et al., 2015) to obtain relation triples. It is worth noting that although OpenIE is able to give a complete description of the entity relations, the relation triples are not always extractable. Hence, we further adopt a dependency parser and dig out some binary tuples from the parse tree of the sentence to supplement the fact descriptions (A.2). Given a factual knowledge graph Gf = (Vf , Ef ), nodes Vf = {vf,1, ..., vf,m} represent subjects and objects of the triples, and directed edgesEf represent links which pointed from subjects to objects, with predicates as edge labels. We then obtain mentions via the coreference resolution tool and collapse coreferential mentions of the same entity into one node. The node representations v̂f,i are updated by using sparse relational graph self-attention network (Sec 3.2.3). Figure 2 (b) presents an example of a constructed factual knowledge graph from a dialogue.

heading: 3.4 DUAL-COPY DECODER
text: Our decoder is a hybrid between a single-layer unidirectional LSTM and a pointer network. To obtain high faithfulness summaries, we also devise a dual copy mechanism to focus on both the input tokens and the factual knowledge. At each decoding step t, we compute a sequence context vector cst with the attention mechanism:
cst = ∑ i asi,th s i
asi,t = softmax(us T tanh(W shh s i +W s s st + b s attn))
(5)
where us, W sh , W s s , and b s attn are learnable parameters. The context vector of the factual knowledge graph cgt can be computed similarly. We fuse c s t and c g t to build the overall context vector ct. We explore two alternative fusion approaches. The first one is called KGEDCc, as shown in Figure 1 (b), which simply concatenates two context vectors:
ct = [c s t ; c g t ] (6)
The other approach is denoted as KGEDCg, as shown in Figure 1 (c), where we also use MLP to build a gate network and combine context vectors with the weighted sum:
gt =MLP (c s t , c g t ) ct = gt cst + (1− gt) c g t
(7)
where means the element-wise dot, and MLP stands for a multi-layer perceptron. Finally, the next token is generated based on the context vector ct, the decoder state st, and the previous word yt−1.
Pvocab = softmax(U ′(U [st, ct] + b) + b ′)
Pgen = σ(w T s st + w T c ct + w T y yt−1 + bgen) P (yt) = pgenPvocab(yt) + (1− pgen)( ∑
i:yi=yt
asi,t + ∑
i:yi=yt
agi,t) (8)
where U , U ′, b, b′, wTs , w T c , w T y , and bgen are learnable parameters. σ is the sigmoid function.

heading: 4 DATASET AND EXPERIMENTAL SETUP

heading: 4.1 DATASET
text: We perform our experiments on the SAMSum corpus and the Automobile Master corpus, which are both new corpora for dialogue summarization. Table 2 shows some statistics on two datasets. The SAMSum corpus is an English dataset about natural conversations in various scenes of the real-life
(Gliwa et al., 2019). The standard dataset is split into 14732, 818, and 819 examples for training, development, and test. The Automobile Master corpus is from the customer service question and answer scenarios1. We use a portion of the corpus that consists of high-quality text data, excluding picture and speech data. It is split into 183460, 1000, and 1000 for training, development, and test. More statistics are in the A.3.

heading: 4.2 TRAINING DETAILS
text: We filter stop words and punctuations from the training set to generate a limited vocabulary size of 40k. The dialogues and summaries are truncated to 500, and 50 tokens, and we limit the length of each utterance to 20 tokens. The word embeddings and edge label embeddings are set to 128 and 32, which are both initialized randomly. The dimension for hidden layer units are 256 and 128 for the sequence encoder and the graph encoder, respectively. For factual knowledge graph, the hidden size is set to 128. For the CNN, each feature map is 128 with filter sizes of 3, 4, and 5. The size of sliding window for sequential context dependency in the dialogue graph is set to 1. We use a block number of 2, and the head number of 4 for sparse relational graph self-attention network. At test time, the minimum length of generated summaries is set to 15, and the beam size is 5. For all the models, we train for 30000 iterations using Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.001 and the batch size of 8. For each of our models, we run five times and report the average R-1, R-2, and R-L scores along with the standard deviation. More details can be seen in A.4.

heading: 4.3 BASELINE METHODS
text: For both datasets, we compare our proposed method with the following abstractive models: (1) Longest-3; (2) Seq2Seq+Attention (Rush et al., 2015); (3) Transformer (Vaswani et al., 2017); (4) LightConv (Wu et al., 2019); (5) DynamicConv (Wu et al., 2019); (6) Pointer Generator (See et al., 2017); (7) Fast Abs RL (Chen & Bansal, 2018); (8) Fast Abs RL Enhanced (Chen & Bansal, 2018). More details about baselines are shown in A.5.

heading: 5 RESULTS AND DISCUSSIONS

heading: 5.1 MAIN RESULTS
text: Results on SAMSum Corpus The results of the baselines and our model on SAMSum corpus are shown in Table 1. We evaluate our models with the standard ROUGE metric, reporting the F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L. Experiments show that KGEDCg significantly outperforms KGEDCc, and the gate values apparently reflect the relative reliability of dialogues and fact descriptions. By observation, the inclusion of a Separator2 is advantageous for most models, because it improves the discourse structure. Compared to the best performing model Fast Abs RL Enhanced, the KGEDCg model obtains 1.92, 1.60, and 1.79 points higher than it for R-1, R-2, and R-L. The sparse relational graph self-attention operation of graph encoder in our model and the extractive method of Fast Abs RL Enhanced model play a similar role in filtering important contents in dialogues. However, our model does not need to use reinforcement learning strategies, which greatly simplifies the training process. Our model also surpasses the pointer generator model without the graph encoder and factual knowledge graph by 2.99, 4.38, and 4.39 points. This demonstrates the benefit of using implicit structures and knowledge to enhance the faithfulness of summaries.
Results on Automobile Master Corpus We observe similar trends on Automobile Master corpus as shown in Table 1. Combined with graph encoder and dual-copy decoder, KGEDCg achieves Rouge-1, Rouge-2, and Rouge-L of 43.35, 18.23, and 38.98, which outperforms the baselines and our KGEDCc by different margins. Noticeably, unlike SAMSum corpus, The Fast Abs RL Enhanced model has no obvious advantage over other sequence models. This is because the average number of utterances in this dataset is larger and the information is more scattered. Due to the limited computational resource, we do not apply a pre-trained encoder (i.e. BERT) to our model, which
1This dataset is released by the AI industry application competition of Baidu. 2Separator is a special token added artificially, e.g. <EOU> for models using word embeddings, | for
models using subword embeddings. The use of it is proposed by Gliwa et al. (2019).
Table 3: Human Evaluation for the relevance and readability on the test set of SAMSum corpus and Automobile Master corpus.
Dataset Model Relevance Readability
SAMSum PGS 2.36 4.25 FARE 2.67 4.73 KGEDCg 2.97 4.91
Automobile Master PGS 2.41 4.18 FARE 2.59 4.35 KGEDCg 2.92 4.66
Table 5: Ablation results w.r.t the speaker, sequential context, and co-occurring keyword dependency on SAMSum dataset.
SD SCD CKD R-1(σ) R-2(σ) R-L(σ) ! ! ! 43.87(0.4) 19.66(0.5) 41.02(0.7) ! % ! 43.05(0.5) 18.93(0.5) 40.33(0.6) ! ! % 42.38(0.7) 18.16(0.5) 39.04(0.4) % ! ! 41.43(0.6) 17.54(0.3) 38.26(0.5) ! % % 41.05(0.6) 16.78(0.4) 37.23(0.4) % % ! 40.66(0.5) 16.13(0.7) 36.85(0.7) % ! % 40.28(0.8) 15.71(0.9) 36.49(0.6) % % % 39.62(0.6) 15.05(0.6) 35.87(0.5)
we will regard as our future work. For the sake of fairness, we only compare with models without BERT.
Human Evaluation We further conduct a manual evaluation to analyze the relevance and readability of generated summaries. 50 samples are randomly selected from the test set of SAMSum corpus and Automobile Master corpus respectively, and five native or fluent speakers of English are hired as human annotators to rate summaries generated by our KGEDCg, along with outputs by Pointer Generator+Separator (PGS) and Fast Abs RL Enhanced (FARE). After reading the dialogues, each annotator scores each perspective from 1 (worst) to 5 (best). Relevance is a measure of how much faithful information the summary contains, and readability is a measure of how fluent and grammatical the summary is. For relevance, we consider two types of unfaithful errors: (1) deletion or substitution error-mistakenly deleting or substituting subjects, objects, or clauses, and (2) hallucination error-creating content not present in the input. We ask the annotators to label each type as 1 for the existence of errors and 0 otherwise, and give the scores of relevance according to the percentage of errors. From Table 3, we can see that the KGEDCg obtains better scores, compared to the baselines without the dialogue graph and factual knowledge graph. This indicates the effectiveness of leveraging conversational structures and knowledge graph representation. More details are in A.6.

heading: 5.2 ABLATION STUDY
text: We examine the contributions of three main modules, namely, sequence encoder, graph encoder, and factual knowledge graph, using the best-performing KGEDCg model on SAMSum corpus. The average results of five experiments with standard deviation (σ) are shown in Table 4. Firstly, we study the effects of two encoders while keeping the FKG module. We only remove one encoder at a time and find that removing either of them leads performance to drop greatly. Especially, the sequence encoder is slightly more important in overall performance, which suggests that the token-level information is indispensable for generating summaries. Removing both of them results in a very poor R-1, R-2, and R-L scores of 19.04%, 8.57%, and 17.82%, which is because the conversational context is not well represented. Next, after getting rid of the FKG module, all models could not keep as competitive as models with the FKG, which suggests the importance of factual knowledge. The framework of the model without GE and FKG resembles the Pointer Generator and their performances are also similar.
Further, we conduct ablation analysis on the edge labels of the dialogue graph, as shown in Table 5. Removal of SCD edges does not significantly affect the performance because the sequence encoder
can replace their impact to some extent. Removing either of SD edges and CKD edges will result in poor performance. Besides, when two of the three edge types are arbitrarily removed, the performance will be greatly reduced, which demonstrates that capturing the long-distance cross sentence dependencies is important for making information flows in dialogues clearer. As we can also see, removing three types of edges completely is equivalent to deleting the graph encoder, which can not aggregate utterance-level features.

heading: 5.3 GATE FUSION ANALYSIS
text: As shown in Table 1, KGEDCg achieves higher ROUGE scores than KGEDCc. Now, we investigate what the gate network (Eq.7) actually learns and what ratio the cs and cf are combined. Figure 9 in A.7 shows the changes of the gate values for the development set during training. For SAMSum corpus, at the beginning, the average gate value exceeds 0.5, which suggests the generated content is biased to choose the source dialogue. As the training goes on, our model gradually realizes that the fact descriptions are more reliable, which leads to a consecutive drop of the gate value. Finally, the average gate value gt is gradually stabilized to 0.422. Notably, the ratio of gate values is (1 − 0.422)/0.422 ≈ 1.37 and it is extremely close to the ratio of copying proportions (0.44/0.32 ≈ 1.38) shown in Table 2, which seems that our model predicts the copy proportion relative accurately and normalizes it as the gate value.

heading: 5.4 CASE STUDY
text: Table 8 in A.8 shows an example of dialogue summaries generated by different models. The summary generated by the PGS repeats the same name “Lilly” and only focuses on some pieces of information in the dialogue. FARE model uses both of the extractive and abstractive methods, which makes the generated summary contain both interlocutors’ names: Lilly and Gabriel, and obtains some valid keywords, e.g. pasta with salmon and basil. However, FARE combines the subject and object from different parts of a complex sentence, and usually makes mistakes in deciding who performs the action (the subject) and who receives the action (the object). By using various semantic relations, our model constructs a dialogue graph to capture long-distance cross-sentence dependencies and combines the elements of events correctly to some extent. Factual knowledge is also enhanced via the dual-copy mechanism, which makes generated summaries identify more key information. However, there are some deficiencies in logical reasoning for our models. For example, the future tense is not inferred and the logical relationship between “order food” and “for Lilly” is not recognized.

heading: 5.5 NON-DIALOGIC SETTING
text: To verify the generalization ability of our method, we run our models on a non-dialogic dataset XSum with more abstractive summaries. In this case, we remove the edges of speaker dependency. As shown in Table 6, the KGEDCg gets the best scores, which further demonstrates that our method can capture long-distance cross-sentence dependencies and generate more informative and faithful summaries.

heading: 6 CONCLUSION
text: We propose a Knowledge Graph Enhanced Dual-Copy network for abstractive dialogue summarization. Our model explores the conversational structure via various semantic relations to construct a dialogue graph, which can capture long-distance cross-sentence dependencies. In tandem with copying from the source dialogue, our dual-copy mechanism utilizes factual knowledge graph to improve generated summaries. The human evaluation further confirms that the KGEDC produces more informative summaries and significantly reduces unfaithful errors.

heading: A APPENDIX

heading: A.1 EDGE INITIALIZATION FOR DIALOGUE GRAPH
text: We illustrate the edge initialization process for the dialogue graph and the dialogue content of which is shown in Figure 4. For speaker dependency, Neville, Don, and Wyatt corresponding to (1, 3), (2, 5), and (4, 6), respectively. For sequential context dependency, the size of sliding window is set to 1. For co-occurring keyword dependency, the selected keywords are Neville, remember, date, wedding, Don, and Wyatt, which construct this type edges between (1, 3), (1, 6), (2, 5), (4, 6).

heading: A.2 FACT TUPLE EXTRACTION
text: Although OpenIE is able to give a complete description of parts of a fact, the relation triples are not always extractable. In fact, about 18% and 21% of the OpenIE outputs are empty on the SAMSum corpus and Automobile Master corpus, respectively. These empty instances are likely to damage the performance of our model. As observed, each utterance almost contains some binary tuples, while the complete fact tuples are not always available. Therefore, we adopt a strategy to solve this issue. Firstly, we give the part of speech of each token in the utterance via Stanford CoreNLP (Manning et al., 2014). The noun, numeral, adjective, and notional verb, which can express practical significance, are selected as keywords. Then we leverage the dependency parser to dig out the appropriate tuples to supplement the fact descriptions. The dependency parser converts an utterance into the labeled tuples and only the tuples related to keywords are chosen. Finally, we merge the tuples containing the same words, collapse coreferential mentions of the same entity into one word, and order words based on the original sentence to form the fact descriptions. As shown in Figure 4, we give a demonstration of fact tuple extraction in detail. The outputs of OpenIE are empty for some utterances in the dialogue. For 1-th utterance, we filter out the verb-related and noun-related tuples: (anyone, remember), (remember, wedding), and (wedding, date) to form a fact description: anyone remember the wedding date. For 3-th utterance, we filter out the noun-related ,adjectiverelated, and verb-related tuples: (Tina, mad), (mad, something), (wedding, anniversary), and (have, check) to form a fact description: Tina is mad wedding anniversary. For 6-th utterance, we filter out the noun-related, numeral-related and verb-related tuples: (September, 17), (hope, remember), and (remember, date) to form a fact description: I hope you to remember the date September 17. All these tuples are merged to form facts of the dialogue.

heading: A.3 STATISTICS OF SAMSUM CORPUS AND AUTOMOBILE MASTER CORPUS
text: As shown in Figure 5-8, we analyze the length distributions of the dialogue and summary for SAMSum corpus and Automobile Master corpus, respectively.
11408
3876
833 189 47 16
0
2000
4000
6000
8000
10000
12000
(0, 100] (100, 200] (200, 300] (300, 400] (400, 500] (500, 1000]
F re
qu en
cy
Length of dialogue
Figure 5: The length distribution of the dialogue for SAMSum corpus.
3187
6458
3873
1759
812 280
0
1000
2000
3000
4000
5000
6000
7000
(0, 10] (10, 20] (20, 30] (30, 40] (40, 50] (50, 100]
F re
qu en
cy
Length of summary
Figure 6: The length distribution of the summary for SAMSum corpus.
68006
56759
28076
13855
7189 11575
0
10000
20000
30000
40000
50000
60000
70000
(0, 100] (100, 200] (200, 300] (300, 400] (400, 500] (500, 1500]
F re
qu en
cy
Length of dialogue
Figure 7: The length distribution of the dialogue for Automobile Master corpus.
37198
61800
42508
21923
10804 11227
0
10000
20000
30000
40000
50000
60000
70000
(0, 10] (10, 20] (20, 30] (30, 40] (40, 50] (50, 150]
F re
qu en
cy
Length of summary
Figure 8: The length distribution of the summary for Automobile Master corpus.

heading: A.4 HYPER-PARAMETER SETTINGS
text: We tune the hyper-parameters according to results on the development set. We choose the number of blocks nb from {1, 2, 3}, and the number of self-attention heads nh from {2, 4, 6}. The learning rate λ is searched within the range of {0.0005, 0.0007, 0.001, 0.005}, and the batch size nbatch is in the range of {8, 16, 32}. The settings of hyper-parameters are shown in Table 7.

heading: A.5 BASELINE DESCRIPTION
text: In this subsection, we describe baselines in detail.
Longest-3: This model is commonly used in the news summarization task, which treats 3 longest utterances in order of length as a summary.
Seq2Seq+Attention: This model is proposed by Rush et al. (2015), which uses an attention-based encoder that learns a latent soft alignment over the input text to help inform the summary.
Transformer: This model is proposed by Vaswani et al. (2017), which relies entirely on an attention mechanism to draw global dependencies between the input and output.
LightConv: This model is proposed by Wu et al. (2019), which has a very small parameter footprint and the kernel does not change over time-steps.
DynamicConv: This model is also proposed by Wu et al. (2019), which predicts a different convolution kernel at every time-step and the dynamic weights are a function of the current time-step only rather than the entire context.
Pointer Generator: Fast Abs RL: This model is proposed by Chen & Bansal (2018), which constructs a hybrid extractive-abstractive architecture, with the policy-based reinforcement learning to bridge together the two networks.
Fast Abs RL Enhanced: This model is a variant of Fast Abs RL, which adds the names of all other interlocutors at the end of utterances.

heading: A.6 DETAILS OF HUMAN ANNOTATIONS

heading: A.6.1 READABILITY ANNOTATION GUIDELINES
text: For readability, we make the annotators focus on how fluent and grammatical the summary is and we provide them the following guidelines:
1. First, the annotators judge whether the given sentence is complete or not. If the sentence is incomplete, the annotators rate the score as 1 for it.
2. The annotators can understand the meaning of a complete sentence through their analysis, but there are many grammatical problems in the sentence. The annotators rate the score as 2 or 3 for this sentence.
3. The annotator can easily understand the meaning of the sentence, and there are only minor grammatical problems in it. The annotators rate the score as 4 or 5 for this sentence.

heading: A.6.2 RELEVANCE ANNOTATION GUIDELINES
text: For relevance, we make the annotators focus on how much faithful information the summary contains and we consider two types of unfaithful errors: (a) deletion or substitution error-mistakenly deleting or substituting subjects, objects, or clauses, and (b) hallucination error-creating content not present in the input. The guidelines are as follows:
1. Firstly, we ask the annotators to avoid using general knowledge and check whether the given sentence is consistent with the source texts.
2. If the given sentence is faithful to the source texts, the annotators rate the score as 4 or 5.
3. If the given sentence is not faithful to the source texts, the annotators will label each type of unfaithful errors as 1 for the existence of errors and 0 otherwise and give the scores (1, 2, or 3) of relevance according to the percentage of errors.

heading: A.7 GATE ANALYSIS

heading: A.8 CASE STUDY

Review summaries:
This paper proposes a knowledge graph enhanced network to improve abstractive dialog summarization with graphs constructed from the dialog structure and factual knowledge. The dialog graph is composed of utterances as nodes and 3 heuristic types of edges (such as utterances of the same speaker, adjacent utterances). The factual graph is constructed via openIE and dependency parsing, which the authors claim are complementary as the triplets (results of openIE) are not always available. 

This paper proposes to improve dialogue summarization by encoding the text with a sequential encoder (for token-level contextualization) and a graph encoder (for long-distance and semantic contextualization). A KG is built and considered to be a surrogate for "factual knowledge". A dual-copy mechanism is used while decoding in the hope that direct access to this factual knowledge will enhance the faithfulness of the generated summaries.