Improving Abstractive Dialogue Summarization with Conversational Structure and Factual Knowledge
Authors: Lulu Zhao
Zeyuan Yang
Weiran Xu
Sheng Gao
Jun Guo
Venue:  (2021)
Keywords: abstractive dialogue summarization
long-distance cross-sentence relation
conversational structure
factual knowledge
sparse relational graph self-attention network
dual-copy mechanism

Abstract: Recently, people have been paying more attention to the abstractive dialogue summarization task. Compared with news text, the information flows of the dialogue exchange between at least two interlocutors, which leads to the necessity of capturing long-distance cross-sentence relations. In addition, the generated summaries commonly suffer from fake facts because the key elements of dialogues often scatter in multiple utterances. However, the existing sequence-to-sequence models are difficult to address these issues. Therefore, it is necessary for researchers to explore the implicit conversational structure to ensure the richness and faithfulness of generated contents. In this paper, we present a Knowledge Graph Enhanced Dual-Copy network (KGEDC), a novel framework for abstractive dialogue summarization with conversational structure and factual knowledge. We use a sequence encoder to draw local features and a graph encoder to integrate global features via the sparse relational graph self-attention network, complementing each other. Besides, a dual-copy mechanism is also designed in decoding process to force the generation conditioned on both the source text and extracted factual knowledge. The experimental results show that our method produces significantly higher ROUGE scores than most of the baselines on both SAMSum corpus and Automobile Master corpus. Human judges further evaluate that outputs of our model contain more richer and faithful information.

Under review as a conference paper at ICLR 2021
IMPROVING ABSTRACTIVE DIALOGUE SUMMARIZATION WITH CONVERSATIONAL STRUCTURE AND FACTUAL KNOWLEDGE
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, people have been paying more attention to the abstractive dialogue sum- marization task. Compared with news text, the information flows of the dialogue exchange between at least two interlocutors, which leads to the necessity of captur- ing long-distance cross-sentence relations. In addition, the generated summaries commonly suffer from fake facts because the key elements of dialogues often scat- ter in multiple utterances. However, the existing eqquence-to-sequene models are difficult to address these issues. Therefore, it is necessary explore the implicit conversational structure to ensure the richness and faithfulness generated con- tents. In this paper, present Knowledge Graph Enhanced Dual-Copy work (KGEDC), a novel framework for abstractive dialogue summarization with conversational structure and factual knowledge. We use sequence encoder draw local features and graph encoder to integrate global features via the sparse relational graph self-attention network, complementing each other. Besides, a dual-copy mechanism is also designed in decoding process to force the generation conditioned on both the source text and extracted factual knowledge. The experi- mental results show that our method produces significantly higher ROUGE scores than most of the baselines on both SAMSum corpus and Automobile Master cor- pus. Human judges further evaluate that outputs of our model contain more richer and faithful information.
INTRODUCTION
Abstractive summarization aims to understand the semantic information of source texts, and gener- ate flexible and concise expressions as summaries, which is more similar to how humans summarize texts. By employing sequence-to-sequenee frameworks, some encouraging results have been made in the abstractive summarization of single-speaker documents like news, scientific publications, etc (Rush et al. 2015 See et al. 2017] Gehrmann et al.) 2018; Sharma et al.) 2019). Recently, with the explosive growth of dialogic texts, abstractive dialogue summarization has begun arousing people's interest. Some previous works have attempted to transfer general neural models, which are designed for abstractive summarization of non-dialogic texts, deal with abstractive dialogue summarization task (Goo & Chen 2018 Liu et al. 2019 Gliwa et al. 2019).
texts, dialogues contain dynamic information exchange flows, which are usu- ally informal, verbose and repetitive, sprinkled with false-starts, backchanneling, reconfirmations, hesitations, and speaker interruptions (Sacks et al., 1974). Furthermore, utterances are often turned from different interlocutors, which leads to topic drifts and lower information density. Therefore, previous methods are not suitable generate summaries for dialogues. We argue that the conver- sational structure and factual knowledge are important to generate informative and succinct sum- maries. While the neural methods achieve impressive levels of output fluency, they also struggle to produce a coherent order of facts for longer texts (Wiseman et al. 2017), and are often unfaith- ful to input facts, either omitting, repeating, hallucinating or changing facts. Besides, complex events related to the same element often span across multiple utterances, which makes it challeng- ing for sequence-bat based models to handle utterance-lewe long- ding-distnnce dependencies and capture cens-sentenne relations.
Different from

---

Under review as a conference paper at ICLR 2021
To mitigate these issues, intuitive way is to model the relationships between textual units within a conversation discourse using graph structures, which can break the sequential positions of tex- tual units and directly connect the related long-distance contents. In this paper, we present the Knowledge Graph Enhanced Dual-Copy network (KGEDC), a novel network specially designed for abstractive dialogue summarization. A graph encoder is proposed tional structure in utterance-level under the assumption that utterances represent nodes and edges are semantic relations between them. Specifically, we devise three types of edge labels: speaker dependency, sequential context dependency, and co-occurring keyword dependency. The edges nav- igate the model from the core fact to other occurrences of that fact, and explore its interactions with other concepts or facts. The sparse dialogue graph only leverages related utterances and filters out redundant details, retaining the capacity to include concise concepts or events. In order to extract sequential features at token-level, sequence encoder is also used. These two encoders cooper- ate to express conversational contents via two different granularitics, which can effectively capture long-distance cross-sentence dependencies.
construct the conversa-
Moreover, considering that the fact fabrication is serious problem, encoding existing factual knowl- edge into the summarization system should be an ideal solution to avoid fake generation. To achieve this goal, we firstly apply the OpenlE tool (Angeli et al. 2013) and dependency parser tool (Man- ning al., 2014) to extract the factual knowledge in the form of relational tuples (subject, predi- cate, object), which construct a knowledge graph. These tuples describe facts and are regarded as the skeletons of dialogues. Next, we design dual-copy mechanism to copy contents from tokens of the dialogue text and factual knowledge of the knowledge graph in parallel, which would clearly provide the right guidance for summarization.
To verify the effectiveness of KGEDC, we carry out automatic and human evaluations on SAMSum corpus and Automobile Master corpus. The experimental results show that our model yield signifi- cantly better ROUGE scores (Lin & Hovy (003) than all baselines. Human judges further confirm that KGEDC generates more informative summaries with less unfaithful errors than all models with- out the knowledge graph.
2 RELATED WORK
Graph-based summarization Graph-based approaches have been widely explored in text sum- marization. Early traditional works make use of inter-sentence cosine similarity to build the connec- tivity graph like LexRank (Erkan & Radev (2004) and TextRank (Mihalcea & Tarau 2004). Some works further propose discourse inter-sentential relationships to build the Approximate Discourse Graph (ADG) (Yasunaga et al. (017) and Rhetorical Structure Theory (RST) graph (Xu et al., 2019). These methods usually rely on external tools and cause error propagation. To avoid these problems, neural models have been applied to improve summarization techniques. TTan et al. (2017) proposed a graph-based attention mechanism to discover the salient information of a document. Fernandes) et al. 2019) developed a framework to extend existing sequence encoders with graph component to reason about
relationships. Zhong et al.) (2019) used a Transformer encoder to create fully-connected graph that learns relations between pairwise sentences. Nevertheless, the factual knowledge implied in dialogues is largely ignored. [Coo et al. (2017) incorporated the fact descrip- framework. [uunel
tions as an additional input source text in the attentional sequence-to-sequene et al. (2019) employed an entity-aware transformer structure to boost the factual correctness, where the entities come from the Wikidata knowledge graph. In this work, we design a graph encoder based on conversational structure, which uses the sparse relational graph self-attention network to obtain the global features of dialogues.
Abstractive dialogue summarization Due to the lack of publicly available resources, the work for dialogue summarization has been rarely studied and it is still in the exploratory stage at present. Some early works benchmarked the abstractive dialogue summarization task using the AMI meeting corpus, which contains a wide range of annotations, including dialogue acts, topic descriptions, etc (Carletta et al. 2005 Mehdad et al. 2014 Banerjee et al. 2015). Goo & Chen (2018) proposed to use the high-level topic descriptions (e.g. costing evaluation of project process) as the gold ref- erences and leveraged dialogue act signals in neural summarization model. They assumed that dialogue acts indicated interactive signals and used these information for a better performance, Cus- tomer service interaction is also common form of dialogue, which contains questions of the user
2

---

Under review as a conference paper at ICLR 2021
Pon
Find Disrltaioo
Ruub
Facuad Kaunhekkp
Graph
Atlention
Attention
Segaaeea
Einishe
9+
0
M.P
Guugh
95
000
Daal-crpyy
Figure I: (a) A general architecture of Knowledge Graph Enhanced Dual-copy network for ab- stractive dialogue summarization. (b) The first approach for fusion, which simply concatenates two context vectors. (c) The second approach for fusion, which uses MLP to build a gate network and combines context vectors with the weighted sum.
and solutions of the agent. Liu et al. (219)) collected a dialogue-summan summary dataset from the logs in the DiDi customer service center. They proposed a novel Leader-Writer network, which relies on auxiliary key point sequences to ensure the logic and integrity of dialogue summaries and designs hierarchical decoder. The rules labeling the key point sequences are given by domain experts, which needs to consume lot of human effort. For Argumentative Dialogue Summary Corpus, Ganesh Dingliwal (2019) used the sequence tagging of utterances for identifying the discourse relations of the dialogue and fed these relations into attention-based pointer network. From con- sultations between nurses and patients, Liu et al. (2019) arranged a pilot dataset. They presented an architecture that integrates the topic-level attention mechanism in the pointer-generator network, utilizing the hierarchical structure of dialogues. Because the above datasets all have a low num- ber of instances and the quality of them is also low, Gliwa al. intoouiddeddeued middle-scale abstractive dialogue summarization dataset (namely SAMSum) and evaluated a number of generic abstractive summarization methods. Although some progress has been made abstractive dialogue summarization task, previous methods do not develop specially designed solutions for dialogues and are all dependent on seneene-to-sequenee models, which can not handle the utterance-leva long- distance dependency, thus causing inaccuracies for rephrased summaries. We present a dual-copy mechanism to directly extract facts from the knowledge graph, which enhances the credibility the summarization generation.
3 METHODOLOGY
In this section, we introduce the Knowledge Graph Enhanced Dual-Copy network, as displayed in Figure (a). Our framework consists of four modules including sequence encoder, graph encoder, a factual knowledge graph, and a dual-copy decoder. Importantly, we first present two types of graphs: Dialogue Graph which constructs conversational structures, and Factual Knowledge Graph which directly extracts fact descriptions from source dialogues. Then, the dual-copy decoder generates faithful summaries by embedding the semantics of both source utterances and factual knowledge.
3.1 SEQUENCE ENCODER
Considering that the contextual information of dialogues usually flows along the sequence, the se- quential aspect of the input text is also rich in meaning. Taking the dialogue D as an example, we feed the tokens of it one-by-one into single-layer bidirectional LSTM, producing a sequence of encoder hidden states { his hS). The BiLSTM the time step is defined as follows:
h BiLSTM (w,, k(_1)
where w; is the embedding of the i-th token in dialogue, and he is the concatenation of the hidden state of forward LSTM and backward L.T..

---

Under review as a conference paper at ICLR 2021
Comtracted Dialogue Graph
Aatt
Agues
Condracted Factual Krzwildge
Lioer Combitation Laper
Mall
Kazimierz
Sell Attrtio Lager
Restauratt,
Saturday
6 pm
CAT
Diblogue:
CNN
Seguartbil extnst
Speaker depeadency
depeedlaryy
Mort: Do you west to for dist? Malt: This io the perfect thee go loone other pllk you
the Grampan kankmierz Mult: Saturalay
vay reeturraatt
C 5
Coocturring lopword
Samumas::
Mart instirs Agnes for a date get izzw cazt ether better: They'l go to the Ciarrian kannnii Kunitiier Saturday 616 pau and ba'll jack las way
(x)
[hi
Figure 2: (a) The detailed construction process of graph encoder (self-loops are omitted for sim- plification in the constructed dialogue graph). (b) An example of a constructed factual knowledge graph from a dialogue.
3.2 GRAPH ENCODER
Given a constructed dialogue graph Ga = (Va. Ea). Va = {vd.1.... vd corresponds to m utter- ances in the dialogue. Fa is a real-value edge weight matrix and Cd,ij * (i (1....,m) {1, m)) indicates that there is an edge between the i-th utterance and the j-th utterance. We then use graph neural networks to update the representations of all utterances to capture long-distance cross-sentence dependencies, shown in Figure 2 (a).
3.2.1 NODE INITIALIZATION
We first use a Convolutional Neural Network (CNN) with different filter sizes to obtain the represen- tations I, for the j-th utterance Vd.j. We also do pos tagging on dialogues with off-the-shelf tools such as Stanford CoreNLP (Manning et al. 2014) and select nouns, numerals, adjectives, adverbs, and notional verbs as keywords. Each keyword transformed into real-valued vector represen- tation & by looking up the word embedding matrix, which is initialized by random process. The attention mechanism is then designed to characterize the strength of contextual correlations between utterances and keywords. The representation of the utterance ěd is updated as weighted sum of these keyword vector k; (i E (.....,,))
A. f(e,, = z.j
cop(AAg)
0g softmar, (Ai)) EKEN, erp(Asj)
(2)
K
0j.j a
Oiy W.e;)
where K is the number of keywords in each dialogue, W, trainable weight, and ai is the attention coefficient between E; and Trj-
3.2.2 EDGE INITIALIZATION
If we hypothesize that each utterance is contextually dependent on all the other utterances in the dialogue, then fully connected graph would be constructed. However, this leads to a huge amount of computation. Therefore, adopt strategy to construct the edges of the graph, which relies on the various semantic relations among utterances. We define three types of edge labels: speaker dependency, sequential context dependency, co-occurring keyword dependency (A.1).
Speaker dependency: The relation depends on where the same speaker appears in the dialogue. In other words, if the utterances belong to the same speaker, will set edge between them.
Sequential context dependency: The relation describes the sequential utterances that occur within fixed-size sliding window. In this scenario, each utterance node Va,i has edge with the immediate utterances of the past 0d Ud,i-2, Ud,i-p- and f utterances of the future:
U4i++. 1. 0d,i42, ..., U-
co-occurring keyword dependency: The relation means that all utterances containing the same keyword are connected.
4

---

Under review as a conference paper at ICLR 2021
We further change the existing undirected edges into bidirectional edges and add self-loops to hance the information flow.
3.2.3 ITERATIVE REFINEMENT
To modularize architecture design, multiple blocks, which all consist of the sparse relational graph self-attention layer and the linear combination layer, are stacked to capture long-distance dependen- cies missed by sequence encoder for better summarization.
Sparse Relational Graph Self-Attention Layer Different parts of the dialogue history have dis- tinct levels of importance that may influence the summary generation process. We use sparse attention mechanism to focus more on the salient utterances. The full self-attention operation in Transformer (Vaswani et al., 2017) captures the interactions between two arbitrary positions of sin- gle sequence. However, our sparse self- attention operation only calculates the similarities between two connected nodes in the graph and masks the irrelevant edges, which reduces the computation amount to a certain extent. In a single sparse graph attention layer, a node in the graph attends over local information from --hop neighbors, which assigns different attention weights to different neighboring edges. These weights can be learned by the model in end-to-end fashion. Although the sparse graph attention mechanism aggregates the representations of neighborhood nodes along the dependency paths, this process fails to take semantic relations into consideration, which may lose some important dependency information. Intuitively, neighborhood nodes with different depen- dency relations should have different influences. We extend the sparse graph attention mechanism with the additional edge labels, which can learn better inter-sentence relations. Specifically, we first map the edge labels into vector representations R, and then this layer is designed as follows:
head; Attention(QW .WW.. K. .WW. RW...)
Attention (2, softmar Q x K
+ RV + +R)
(3)
Vd
g' =
Woo
where W", W.O. W.. W.K, and W are weight matrices, H is the head number, and d is the dimension of utterance node features 0, E m@ In the first block. K, and V are 0g. For the following blocks 4, they are the linear combination layer output vector -!-1 E 1md of block 1.
Linear Combination Layer in between just as Transformer tion layer defined as:
This layer contains linear combinations with a RU activation Vaswani et al. 2017). Formally, the output of the linear transforma-
( w/ + b1)mds + b9 where wi, and we weight matrices. b, and b2 are bias vectors.
After M identical blocks, we take mean pooling over all nodes and obtain the final representation of the dialogue graph, G m E z.M. We then concatenate the representation of sequence
encoder h and graph encoder S get the initial state of the decoder su [h,;
3.3 FACTUAL KNOWLEDGE GRAPH CONSTRUCTION
To construct a factual knowledge graph from an input dialogue, we leverage the Open Information Extraction (OpenIE) tools (Angeli et al. 2015) obtain relation triples. It is worth noting that although OpenIE is able to give complete description of the entity relations, the relation triples are not always extractable. Hence, we further adopt a dependency parser and dig out some binary tuples from the parse tree of the sentence to supplement the fact descriptions (A.2). Given a factual knowledge graph G2 = (Vy, Ey), nodes V = (U .... Uf. represent subjects and objects of the triples, and directed edges E, represent links which pointed from subjects to objects, with predicates as edge labels. We then obtain mentions via the coreference resolution tool and collapse coreferential mentions of the same entity into node. The node representations é a updated by using sparse relational graph self-attent attention network (Sec 3.2.3). Figure (b) presents an example a constructed factual knowledge graph from dialogue.
5

---

Under review as a conference paper at ICLR 2021
Table I: Results in terms of R-1, R-2, and R-L on the SAMSum corpus and Automobile
Master corpus test set. For our models, we report average scores with the standard deviation.
SAMSum
Automobile Master
Model
R-
R-2
R-L
R-1
R-2
R-L
Longest-3
32.46
10.27
29.92
30.72
9.07
28. 14
Seq2Seq
21.51
10.83
20.38
25.84
13.82
25.46
Seq2Seq+Attention
29.35
15.90
28.16
30.18
16.52
29.37
Transformer
36.62
11.18
33.06
36.21
11.13
34.08
Transformer++epprator
37.27
10.76
32.73
37.43
11.87
34.97
LightConv
33.19
11.14
30.34
34.68
12.41
31.62
DynamicConv
33.79
11.19
30.41
34.72
12.45
31.86
DynamicConv+-Sepe SSpaarato
33.69
10.88
30.93
34.41
12.38
31.22
Pointer Generator
38.55
14.14
34.85
39.17
15.39
34.76
Pointer Generator+Separator
40.88
15.28
36.63
39.23
15.42
34.53
Fast Abs RL
40.96
17.18
39.05
39.82
15.86
36.03
Fast Abs RL Enhanced
41.95
18.06
39.23
40.13
16.17
36.42
KGEDC.
43.5110..6
19.3410.33
40.57.10.5
42.85.11.2
18.11-0.9
38.02.11.0
KGEDC₂
43.87.10.4
19.66.1..5
41.02.0..7
43.35
38.9811. 18.23
3.4
DUAL-COPY DECODER
Our decoder is hybrid between single-layer unidirectional LSTM and pointer network. To obtain high faithfulness summaries, we also devise dual copy mechanism to focus on both the input tokens and the factual knowledge. At each decoding step t, we compute sequence context vector with the attention mechanism:
Σ oh
(5)
of = so smmoruu,,
+ W. + bate))
where ug. W W and betn are learnable parameters. The context vector of the factual knowledge graph of can be computed similarly. We fuse of and of to build the overall context vector q. We explore two alternative fusion approaches. The first one called KGEDCe, as shown in Figure (b), which simply concatenates context vectors:
C = [c:
(6)
The other approach is denoted as KGEDC,, as shown in Figure (c), where we also use MLP to build gate network and combine context vectors with the weighted sum:
9t MLP(G,
Gt 9i ② ef 9))
where @ means the element-wiss dot, and MLP stands for multi-layer perceptron.
Finally, the next token generated based on the context vector 6t, decoder state 8t, and previous word 9/-1-
P Procab softmax(UU" (U' ce] + b) b')
P Pgen ow T T wiy 1 1 + bgen)
(8)
P(v)) Pgen P Poocab (ve) + (1 Pgen ) Bsgy
Σ
where U, U², b, U, wT, w., wT. and byen are learnable parameters, or is the sigmoid function.
DATASET AND EXPERIMENTAL SETUP
4.1 DATASET
We perform experiments the SAMSum corpus and the Automobile Master corpus, which both new corpora for dialogue summarization. Table shows some statistics on two datasets. The SAMSum corpus is an English dataset about natural conversations in various scenes of the real-life

---

Under review as a conference paper at ICLR 2021
Gliwa al. 2019). The standard dataset is split into 14732, 818, and 819 examples for training, development, and test. The Automobile Master corpus is from the customer service question and answer scenarios We use portion of the corpus that consists of high-quality text data, excluding picture and speech data. It is split into 183460, 1000, and 1000 for training, development, and test. More statistics are in the A.3.
4.2 TRAINING DETAILS
We filter stop words and punctuations from the training set to generate limited vocabulary size of 40k. The dialogues and summaries are truncated to 500, and 50 tokens, and we limit the length of each utterance to 20 tokens. The word embeddings and edge label embeddings are set to 128 and 32, which are both initialized randomly. The dimension for hidden layer units are 256 and 128 for the sequence encoder and the graph encoder, respectively. For factual knowledge graph, the hidden size is set to 128. For the CNN, each feature map is 128 with filter sizes of 3, 4, and 5. The size of sliding window for sequential context dependency in the dialogue graph set to We use block number of 2, and the head number of for sparse relational graph self-attention network. At test time, the minimum length of generated summaries set to 15, and the beam size is 5. For all the models, we train for 30000 iterations using Adam optimizer (Kingma & Bal (014) with an initial learning rate of 0.001 and the batch size of 8. For each of our models, we run five times and report the average R-1, R-2, and R-L scores along with the standard deviation. More details can be seen in A.4.
4.3 BASELINE METHODS
For both datasets, we compare our proposed method with the following abstractive models: (1)
Longest-3; (2) SequSeg+AAtention (Rush et al. 2015); (3) Transformer (Vaswani et al. 2017); ; (4)
LightConv (Wu al. 2019; (5) DynamicConv (Wu al. 2019); (6) Pointer Generator (See al.
2017]; (7) Fast Abs RL (Chen & Bansal 2018); (8) Fast Abs RL Enhanced (Chen & Bansa 2018
More details about baselines are shown in A.5.
RESULTS AND DISCUSSIONS
5.1 MAIN RESULTS
Results on SAMSum Corpus The results of the baselines and our model on SAMSum corpus are shown in Table 1. We evaluate our models with the standard ROUGE metric, reporting the Fl scores for ROUGE-I, ROUGE-2, and ROUGE-L. Experiments show that KGEDCg significantly outperforms KGEDC, and the gate values apparently reflect the relative reliability of dialogues and fact descriptions. By observation, the inclusion of a Separato is advantageous for most models, because it improves the discourse structure. Compared to the best performing model Fast Abs RL Enhanced, the KGEDC₂ model obtains 1.92, 1.60, and 1.79 points higher than it for R-1, R-2, and R-L. The sparse relational graph self-attention operation of graph encoder in our model and the extractive method of Fast Abs RL Enhanced model play similar role filtering important contents in dialogues. However, our model does not need to use reinforcement learning strategies, which greatly simplifies the training process. Our model also surpasses the pointer generator model without the graph encoder and factual knowledge graph by 2.99, 4.38, and 4.39 points. This demonstrates the benefit of using implicit structures and knowledge to enhance the faithfulness of summarics.
Results on Automobile Master Corpus We observe similar trends Automobile Master cor- pus as shown Table 1. Combined with graph encoder and dual-copy decoder, KGEDCg achieves Rouge- 1, Rouge-2, and Rouge-L of 43.35, 18.23, and 38.98, which outperforms the baselines and our KGEDC, by different margins. Noticeably, unlike SAMSum corpus, The Fast Abs RL En- hanced model has obvious advantage over other sequence models. This is because the average number of utterances in this dataset is larger and the information is more scattered. Due to the lim- ited computational resource, we do not apply pre-trained encoder (i.e. BERT) to our model, which
This dataset is released by the Al industry application competition Baidu.
Separator is special token added artificially, e.g. <BOU> for models using word embeddings, for models using subword emheddings. The use of is proposed by Gliwa al. (201)).
7

---

Under review as a conference paper at ICLR 2021
Table 2:
Data statisties. A, and A.S are the average lengths of dialogues and summaries. A.T, Table 3: Human Evaluation for the relevance and
A.P, A.K, and A_F are the average numbers of readability on the test set of SAMSum corpus and turns, speakers, keywords, and facts. C.T and C_F Automobile Master corpus.
mean the proportions of tokens and facts can be
Dataset
Model
Relevance
Readability
found in the summary.
PGS
2.36
4.25
SAMSum
FARE
2.67
4.73
SAMSum
Automobile Master
KGEDC
2.97
4.91
A.
119
A.S
23
A.D
182
A_S
23
PGS
2.41
4.18
AT
P
A
11
P
P
Automobile
FARE
2.59
4.35
A.K
A_F
29
A.K
41
A_F
37
Master
KGEDC
2.92
4.66
CT
32%
C
44%
CT
18%
C
27%
Table 4: Ablation results w.r.t the sequence en- coder, graph encoder, and factual knowledge graph on SAMSum dataset.
Table 5: Ablation results w.r.t the speaker, se-
quential context, and co-oceurring keyword de- pendency on SAMSum dataset.
SE
GE
FKG
R-1(o) R-2(a) R-L(a)
SD
SCD CKD
R-T(Ø) R-2(a) R-I((r)
43.87(0.4) 19.66(0.5) (1.6600)))41.222 41.02(0.7)
43.87(0.4) 19. (6(0.5) 41.00..7
X
39.62(0.6) 15.05(0.6) 35.8 (0.5)
X
43.05(0.5) 18.93(0.5) (0.63
V
36.15(0.5) 12.38(0.8) 33.79(0.6)
V
/
X
42.38(0.7) 18..)) 39. 39.04(0.44
X
X
V
19.04(1.0) (004(1.)).570009) 8.57(0.9) 17.82(1.2)
X
/
/
41.43(0.6) 17.54(0.3) 38.226005
V
X
42.71(0.5) 19.15().4) 40.32(0..)
X
X
41.05(0.6) (7.23(04)
X
X
38.93(0.7) 14.16(0.6) 38.8)(
X
X
V
40.66(0.5) 16. 13..7( 36.85(0.7)
XXX
V
35.65(0.8) 11 (1.49)0.7) 32.57(0.44
X
V
X
40.28(0.8) 15.71 36. (6.49(
X
X
X
17.52(1.2) (12((12)7.90)) 7.90(1.5) 16.26(1.0)
X
X
X
39.62(0.6) 15.05(0.6) 30.5)
will regard our future work. For the sake of fairness, we only compare with models without
BERT.
Human Evaluation We further conduct manual evaluation to analyze the relevance and read- ability of generated summaries. 50 samples are randomly selected from the test set of SAMSum corpus and Automobile Master corpus respectively, and five native fluent speakers of English are hired as human annotators to rate summaries generated by our KGEDC g. along with outputs by Pointer Sepprrator+Separaoor (PGS) and Fast Abs RL Enhanced (FARE). After reading the dia- logues, each annotator scores each perspective from 1 (worst) to (best). Relevance is a measure of how much faithful information the summary contains, and readability measure of how fluent and grammatical the summary is. For relevance, we consider two types of unfaithful errors: (1) deletion or substitution mrrr-miitakenly deleting or substituting subjects, objects, or clauses, and (2) hallucination error- creating content not present in the input. We ask the annotators to label each type as for the existence of errors and otherwise, and give the scores of relevance according the percentage of errors. From Table 3, we can see that the KGEDC, obtains better scores, compared to the baselines without the dialogue graph and factual knowledge graph. This indicates the effec- tiveness of leveraging conversational structures and knowledge graph representation. More details are A.6.
5.2 ABLATION STUDY
We examine the contributions of three main modules, namely, sequence encoder, graph encoder, and factual knowledge graph, using the best-performing KGEDC, model on SAMSum corpus. The average results of five experiments with standard deviation (a) are shown in Table Firstly, we study the effects of two encoders while keeping the FKG module. We only remove one encoder a time and find that removing either of them leads performance to drop greatly. Especially, the sequence encoder is slightly more important in overall performance, which suggests that the level-level information is indispensable for generating summaries. Removing both them results in very poor R-1, R-2, and R-L scores of 19.04%, 8.57%, and 17.82%, which is because the conversational context is not well represented. Next, after getting rid of the FKG module, all models could not keep as competitive as models with the FKG, which suggests the importance of factual knowledge. The framework of the model without GE and FKG resembles the Pointer Generator and their performances also similar.
Further, we conduct ablation analysis on the edge labels of the dialogue graph, as shown in Table 5. Removal of SCD edges does not significantly affect the performance because the sequence encoder
8

---

Under review as a conference paper at ICLR 2021
Table 6: Results in terms of Rouge-1, Rouge-2, and Rouge-L on the XSum corpus test set.
Model
R-1
R-2
R-L
PGS
30.27
9.85
23.63
FARE
32.03
11,64
26.11
KGEDC.
34.71
14,38
29,56
KGEDC,
35.16
14.90
30.02
can replace their impact to some extent. Removing either of SD edges and CKD edges will result in poor performance. Besides, when two of the three edge types arbitrarily removed, the perfor- mance will be greatly reduced, which demonstrates that capturing the long-distance cross sentence dependencies is important for making information flows in dialogues clearer. As we can also see, removing three types edges completely is equivalent to deleting the graph encoder, which can not aggregate utterance-level features.
5.3 GATE FUSION ANALYSIS
As shown in Table 1, KGEDC, achieves higher ROUGE scores than KGEDC Now, we in- vestigate what the gate network (Eq.7) actually leams and what ratio the e and ed are combined. Figure in A.7 shows the changes of the gate values for the development set during training. For SAMSum corpus, at the beginning, the average gate value exceeds 0.5, which suggests the gener- ated content is biased choose the source dialogue. As the training goes on, our model gradually realizes that the fact descriptions are more reliable, which leads to consecutive drop of the gate value. Finally, the average gate value Ir gradually stabilized to 0.422. Notably, the ratio of gate values is (1 (0.44/0.35 /0.32 1 1.38) shown in Table which seems that our model predicts the copy proportion relative accurately and normalizes it as the gate value.
(412))00.22 R 1.37 and it is extremely close to the ratio of copying proportions
5.4
CASE STUDY
Table 8 in A.8 shows an example of dialogue summaries generated by different models. The sum- mary generated by the PGS repeats the same name "Lilly" and only focuses some pieces of information in the dialogue. FARE model uses both of the extractive and abstractive methods, which makes the generated summary contain both interlocutors' names: Lilly and Gabriel, and obtains some valid keywords, c.g. pasta with salmon and basil. However, FARE combines the sub- ject and object from different parts of complex sentence, and usually makes mistakes in deciding who performs the action (the subject) and who receives the action (the object). By using various semantic relations, our model constructs a dialogue graph capture long-distance cross-sentence dependencies and combines the elements of events correctly to some extent. Factual knowledge is also enhanced
dual-copy mechanism, which makes generated summaries identify more key information. However, there are some deficiencies in logical reasoning for our models. For example, the future tense not inferred and the logical relationship between "order food" and "for Lilly" is not recognized.
5.5 NON-DIALOGIIC SETTING
To verify the generalization ability of our method, we run our models on a non-dialogic dataset XSum with more abstractive summaries. In this case, remove the edges of speaker dependency. As shown in Table 6, the KGEDC, gets the best scores, which further demonstrates that our method can capture long-distance cross-sentence dependencies and generate more informative and faithful summaries.
CONCLUSION
We propose a Knowledge Graph Enhanced Dual-Copy network for abstractive dialogue summariza- tion. Our model explores the conversational structure via various semantic relations to construct a dialogue graph, which can capture long- disttiit cross-sentence dependencies. tandem with copying from the source dialogue, dual-copy mechanism utilizes factual knowledge graph to improve generated summaries. The human evaluation further confirms that the KGEDC produces more informative summaries and significantly reduces unfaithful errors.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 344-354, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/115- 1034. URL https://wwwa htw.eebb.orr/anthhlooyyye/
Siddhartha Banerjee, Prasenjit Mitra, and Kazunari Sugiyama. Abstractive meeting summariza- tion using dependency graph fusion. In Proceedings of the 24th International Conference on World Wide Web, WWW '15 Companion, pp. 5-6, New York. NY. USA. 2015. Association for Computing Machinery. ISBN 978 9781450334730. /trg/1100 10. 1ew.111142244555 0tt.:/o.rrrr111111/2200000000000 002.440088227727.. 2742751
doi: 10. 00.144527440008.7477757 URL.
htttss://dol .oriii.ogg/iio
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. Faithful to the original: Fact aware neural ab- stractive summarization. In Proceedings of the 32th AAAI Conference on Artificial Intelligence, 2017.
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Proceedings of the Second Inter-
Wellner. The ami meeting corpus: A pre-announcement. national Conference on Machine Learning for Multimodal Interaction. MLMI pp. 28-39, Berlin, Heidelherg, 2005. Springer- Verlag. ISBN 3540325492. doi: 10. 1e.1000711677 1677482 URL.
https: www.orgr...............oo 000/11677/711
Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-sclected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers). pp. 675-686, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1063. URL https:////// wwww aclweb. wwwtahoogg/P1--11111
Marie-Catherine de Marneffe and Christopher D. Manning. Stanford typed dependencies manual.
In Technical report, Stanford University, 2008.
Günes Erkan and Dragomir R. Radev, Lexrank: Graph-based lexical centrality as salience in text summarization. J. Artif. Int. Res., 22(1):457-477) December 2004. ISSN 1076-9757.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summa-
rization. In International Conference Learning Representations, 2019. URL https:
7/ //openrevieeee ew. www.teeforrrm.. d== dotterstttl.
Prakhar Ganesh and Saket Dingliwal. Abstractive summarization of spoken and written conversa- tion, 2019.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pp. 4098-4109, Brussels, Belgium, October-Novemher 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18- 1443. htRLhttp::/ D18 D18-1443
acl web. www..oog//tthhoogg.////
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization,, pp. 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v I/D1/- 1409.,00. /tLh:tt:: www. aclweb. org/antho httpchhooooy/111959900
C. Goo and Y. Chen. Abstractive dialogue summarization with sentence- gated modeling optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 735-742, 2018.
10

---

Under review as a conference paper at ICLR 2021
Beliz Gunel, Chenguang Zhu, Michael Zeng. and Xuedong Huang. Mind the facts: Knowledge- boosted coherent abstractive text summarization. In Proceedings of the Workshop on Knowledge Representation and Reasoning Meets Machine Learning in NIPS, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference the North American Chapter of the Association for Computational Linguistics, pp. 150-157, 2003. URL
https: wwwww/w.wwwww.....mm.. org/ antat wooggg/////00000 1020]
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye, Automatic dialogue summary genera- tion for customer service. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '19, pp. 1957-1965, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016. BtR.httts::/// otwpoii.org//// 1145/ / 32925002200.00 3330683
doi: 10.1 10.1145/2925000333663.
Z. Liu, A. Ng. S. Lee, A. T. Aw, and N. F. Chen. Topic-aware pointer-generator networks for sum- marizing spoken conversations. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 814 124-821,222 2019.
Christopher Manning. Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55 60, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115//1/144-50000 URL https:t::: /www. ac 1 web
Yashar Mehdad, Giuseppe Carenini, and Raymond T. Ng. Abstractive summarization of spoken and written conversations based on phrasal queries. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1220-1230, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.31 10.3115/v1/P14-111
URL https: / www a C wrw.borr... http///hhhooyy/--11111 http://w.w.......oooooooooooooo/
Rada Mihalcea and Paul Tarau. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 404-411, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL hoLLhttp::/// http:://wwwwccc.
org/ bogg/anhollee Setcchoooyy/DDo -3222
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 379-389, Lisbon, Portugal, September 2015. Association for Com- putational Linguistics. doi: 10.18653/v1/D15- 1044. URL https: www aclweb. orq/
anthelogy/D1551044
Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4))696-735, 1974. ISSN 00978507, 15350665.
URL http::tt///w/t httt:::/www.jss.... jstor. http:///////abbbb/eeooeee
Natalie Schluter. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 41-45, Valencia, Spain, April 2017. Association for Computational Linguisties. URL wttpt:t/////. www.wcccll.cccoo...oooocm..c. web www.wwe...w........e wogg/attoooooooooooooooo ogy/e1y 2007
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 1/118866/11/117-10999 URL https: www. awwweb. org/ http:/occ/hhot-a1111
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. An entity-driven framework for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing

---

Under review as a conference paper at ICLR 2021
(EMNLP-IJCNLP), pp. 3280-3291 Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D19-1B 1323. URL https: //wp:///wwwwww// aclweb w
anthology/D19-1323
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Abstractive document summarization with a graph- based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computationnl Linguistics (Volume 1: Long Papers), Pp. 1171-1181, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1..18653/1//11-111 URL https: / 7/www.co ac web. www.aa//eebb.orrrgatttooooo///// hot.llllyy/pp11/////
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL ht.p:://pppen
pdf
Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges in data-to-documen generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253-2263,, Copenhagen, Denmark, September 2017. Association for Computational Linn-
guistics. doi: 10.18653/w1/D17-11339 URL http::://///...... //wtpp::/www.. wwww.www......oooo.w.o ac www..wlwbb.............. www.ooo/b/aoohoooggggo/ooooo/
D17- 1239
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019. URLptttps: htttps://pperrvi...e.
net. forte/ooooo
Jiacheng Xu, Zhe Gan, Yu Cheng. and Jingjing Liu. Discourse-aware neural extractive text summa- rization, 2019.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. Graph-based neural multi-document summarization. In Proceedings of the 2/ Con- ference Computational Natural Language Learning (CoNLL. 2017), pp. 452-462, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/.//117-10055. / www awwweb.ww........... wttpg//antoooo////111
URL https:
Ming Zhong, Pengfei Liu, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. Searching for effective
neural extractive summarization: What works and what's s next. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 1049-1058, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1111183//1/19--10 1/019-1100 URL https: htww.acceee.oog//atttooooooooooe www.:cclww...or///aatooll////loo
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 EDGE INITIALIZATION FOR DIALOGUE GRAPH
We illustrate the edge initialization process for the dialogue graph and the dialogue content of which is shown in Figure 4. For speaker dependency, Neville, Don, and Wyatt corresponding to (1, 3), (2, 5), and (4, 6), respectively. For sequential context dependency, the size of sliding window is set to 1. For co-occurring keyword dependency, the selected keywords are Neville, remember, date, wedding, Don, and Wyatt, which construct this type edges between (1, 3), (1. 6), (2, 5), (4, 6).
Spenker Dependancy (SD)
Nevill: 3): Don (2. 5h Wysth (4, 6)
Sequentiaa Content Dependency (SCI)
(1, 2i (2, 3u G, 4% (5, 6)
1000
Co-occuring Keyward Dependercy (CKII)
(1.3): rememer (1. (1.
wedding: (1, 3) Don: (2, 5t: Wyat (4. 6)
Matts
Mantk
CKD Mabix
Figure 3: (a) A detailed illustration of the edge initialization for the dialogue graph. The dotted box contains all the ids of utterance pairs for three edge labels. (b) Three types of dependency matric in different colors and white color indicates there is no semantic relation between nodes.
A.2 FACT TUPLE EXTRACTION
Although OpenIE is able give a complete description of parts of fact, the relation triples are not always extractable. In fact, about 18% and 21 % of the OpenlE outputs are empty on the SAMSum corpus and Automobile Master corpus, respectively. These empty instances are likely to damage the performance of our model. As observed, each utterance almost contains some binary tuples, while the complete fact tuples are not always available. Therefore, we adopt a strategy to solve this issue. Firstly, we give the part of speech of each token in the utterance via Stanford CoreNLP (Manning et al. 2014). The noun, numeral, adjective, and notional verb, which can express practical significance, are selected as keywords. Then we leverage the dependency parser to dig out the appropriate tuples to supplement the fact descriptions. The dependency parser converts an utterance into the labeled tuples and only the tuples related to keywords tuples containing the same words, collapse coreferential mentions of the same entity into one word, and order words based on the original sentence form the fact descriptions. As shown in Figure 4, we give a demonstration of fact tuple extraction in detail. The outputs of OpenIE are empty for some utterances in the dialogue. For 1-th utterance, we filter out the verb-related and noun-related tuples:
chosen. Finally, merge the
(anyone, remember), (remember, wedding), and (wedding, date) to form a fact description: anyone remember the wedding date. For 3-th utterance, we filter out the noun-related .adjective- related, and verb-related tuples: (Tina, mad), (mad, something). (wedding, anniversary), and (have, check) to form a fact description: Tina is mad wedding anniversary. For 6-th utterance, we filter out the noun-related, numeral-related and verb-related tuples: (September, 17), (hope, remember), and (remember, date) to form a fact description: I hope you to remember the date September 17. All these tuples merged to form facts of the dialogue.
A.3 STATISTICS OF SAMSUM CORPUS AND AUTOMOBILE MASTER CORPUS
As shown in Figure 5-8, we analyze the length distributions of the dialogue and summary for SAM- Sum corpus and Automobile Master corpus, respectively.
13

---

Under review as a conference paper at ICLR 2021
nealy
Neville: Hi! Does anyone remember what date my wedding Is? UN
(anyone, remember); (remember, wedding, date)
PSiPs
VBz
sadt
Don: Are you sertbus?
novdg
oop
Neville: We are on vacation, and Tina mind me about something. NNIP
think is about our wedding anniy ersary,
PRP
vae
IN
NN 0C
NKP VBZ
IN
PRF
IN
XN
PRP
VEP
PSPVBZ
IN
PRPS
NM
XM
cowwp
But have Hlowhere to check.
(Tina, mad, something); (wedding, anniversary); (have, check)
Wyatt: Hang OIL will BSK my Wife.
(ask, wife)
NXP
PB PRB MD
PRPS NN
Don: Haha, someone is in lot of truble.
(someone, lot, trouble)
moubd
obétmee
Wyatt:
September
17.
hope you remember the date.
(September, 17); (hope, remember, date)
NNP
KNP
PRP
Vn
Figure 4: An example of fact tuple extraction in dialogue. Words with shadings in different colors are selected keywords. The purple arrow denotes the keyword-related tuples and the meaning of the dependency labels can be referred to de Marneffe & Manning 2008). The extracted tuples shown in the text boxes.
12000 11485
7000
Frequency
8003
Frapunccy
4000
3073
3000
4002
3676
2000
2000
154
(6. 100)
(100) 200)
(200, 300]
(300, 400)
(400) S00]
(500L 1000)
10
(10,20)
(20, 40j
(20,50)
100)
Length of dialogue
Length summary
Figure 5: The length distribution of the dialogue Figure 6: The length distribution of the sum- for SAMSum corpus.
mary for SAMSum corpus.
70000
70000
00020
MCS
00000
Frequeney
S0020
Frequency
50000
30020
30000
20030
20000
11227
10020
10. 100]
(10), 200]
(200, 300)
(30), 400)
(400, 500)
(500, 1500]
(0.10) 10)
(10.20)
(20,30)
[30 40] (60,50)
(50. 150)
Length of dialogue
Longth of summary
Figure 7: The length distribution of the dialogue Figure 8: The length distribution of the sum- for Automobile Master corpus.
mary for Automobile Master corpus.
14

---

Under review as a conference paper at ICLR 2021
A.4 HYPER- PARAMPARR SETTINGS
We tune the hyper-parameten according to results on the development set. We choose the number of blocks пı from (1, 2,3). and the number self-attention heads n from (2, 4, 6). The learning rate À is searched within the range of (0.0005, 0.0007, 0.001, 0.005). and the batch size Nbttch is in the range {8, 16, 32}. The settings of hyper-parameters are shown in Table 7.
Table 7: Hyper-paramess settings.
Parameter
Parameter Name
Value
ld
maximum length of dialogue
500
Lu
maximum length of utterance
20
mazzs
maximum length of summary
50
Imin.s
minimum length of summary
15
dword
word embedding size
128
dedge
edge label embedding size
32
ds hidden
hidden embedding size of sequence encoder
256
Og hidden
hidden embedding size of graph encoder
128
da hidden
hidden embedding size of factual knowledge graph
128
de hidden
hidden embedding size of CNN
128
wy
filter size for CNN sliding window size
3-4-5
number of blocks number of self-attention heads beam size
71
Thheam
5
Fiiter
number of training iteration
30000
Tbatch
batch size
8
learning rate
0.001
A.5 BASELINE DESCRIPTION
In this subsection, we describe baselines detail.
Longest-3: This model is commonly used in the news summarization task, which treats 3 longest utterances order of length as summary.
Seq SeSSeq+Attentionn Attention: This model is proposed by Rush et al. (2015), which uses an attention-base encoder that learns latent soft alignment over the input text help inform the summary.
Transformer: This model is proposed by Vaswani et al. (2017), which relies entirely on an attention mechanism to draw global dependencies between the input and output.
LightConv: This model is proposed by Wu et al. (2019). which has a very small parameter footprint and the kernel does not change over time-steps.
DynamicConv: This model is also proposed by Wu al. (2019), which predicts different convo- lution kernel every time-step and the dynamic weights are a function the current time-step only rather than the entire context.
Pointer Generator: Fast Abs RL: This model is proposed by Chen & Bansal 2018), which con- structs hybrid extractive-abstractive architecture, with the policy-based reinforcement learning to bridge together the two networks.
Fast Abs RL Enhanced: This model is variant of Fast Abs RL, which adds the names of all other interlocutors at the end of utterances.
A.6 DETAILS HUMAN ANNOTATIONS
A.6.1 READABILITY ANNOTATION GUIDELINES
For readability, we make the annotators focus on how fluent and grammatical the summary and provide them the following guidelines:
15

---

Under review as a conference paper at ICLR 2021
1. First, the annotators judge whether the given sentence is complete not. If the sentence is incomplete, the annotators rate the score as for it.
2. The annotators understand the meaning of complete sentence through their analysis. but there are many grammatical problems in the sentence. The annotators rate the score as or for this sentence.
The annotator can easily understand the meaning of the sentence, and there are only minor grammatical problems in it. The annotators rate the score 4 or 5 for this sentence.
A.6.2 RELEVANCE ANNOTATION GUIDELINES
For relevance, we make the annotators focus on how much faithful information the summary con- tains and we consider two types of unfaithful errors: (a) deletion or substitution error-mistalee mistakistklyl deleting or substituting subjects, objects, or clauses, and (b) hallucination error-creating content not present in the input. The guidelines are as follows:
1. Firstly, ask the annotators to avoid using general knowledge and check whether the given sentence is consistent with the source texts.
2. If the given sentence is faithful to the source texts, the annotators rate the score or 5.
If the given sentence not faithful to the source texts, the annotators will label each type of unfaithful errors 1 for the existence of errors and 0 otherwise and give the scores (1, 2, or 3) of relevance according to the percentage of errors.
A.7 GATE ANALYSIS
0.56
Asivossd: Mare!
0.52
0.46
0.42
10
30
40
30
70
80
100
Epocis
Figure 9: Gates change during training on test set of SAMSum corpus and Automobile Master corpus. Shaded area spans std. The blue line represents the result of the SAMSum corpus and the orange line represents the result of the Automobile Master corpus.
16

---

Under review as a conference paper at ICLR 2021
A.8
CASE STUDY
Table 8: A case study on the summary generated by different models.
Lilly: sorry, I'm gonna be late.
Lilly: don't 't wait for me and order the food.
Gabriel: no problem, shall we also order something for you?
Dialogue
Gabriel: that you get it soon as you get us?
Lilly: good idea!
Lilly: pasta with salmon and basil is always very tasty there.
Reference
Lilly will be late. Gabriel will order pasta with salmon and basil for her.
Longest-3
Gabriel: no problem, shall also order something for you?
Gabriel: so that you get it as soon as you get to us? Lilly: pasta
with salmon and basil is always very tasty there.
Pointer Generator Separator (PGS)
Lilly gonna be late. Lilly order pasta.
Fast Abs RL Enhanced (FARE)
Gabreil order the food, Lilly thinks pasta with salmon and basil
is always very tasty.
KGEDC
Lilly gonna be late. Gabriel order pasta with salmon and basil.
17


Reviews:
[{"review_id": "qnWue5IpXqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new neural pipeline for dialogue summarization that jointly includes word-by-word decoding, an utterance graph, and a factual knowledge graph. The proposed methods is assessed on the SamSUN and  Automobile Master dataset.\nThe method improves the top reported baselines by 0.5 to 2 points (RED) in both cases.\n\nThe paper contains a well-motivated introduction and perform a sound related work section (as far as my judge). They correctly detail their methods in a step-by-step pedagogical procedure. Overall the paper is pretty well-written. \n\nHowever, I have a few concerns about the experimental section. While this paper's main contribution is a (sensible) mixture of neural blocks, a single page of experiments to evaluate the method is not enough from my perspective. Besides, a 0.5-2 pts increase compared to a Pointer network may be a bit limited in light of the model complexity.\nHowever, my main concern is mostly the lack of analysis of the method. As there are many design choices, this requires multiple ablation studies to validate all of them. Yet, the authors only ablate the factual knowledge graph and graph encoder. Furthermore, the score differences are small, and a few run + std would have been useful to determine whether one change is significant.\nIdeas for other ablation studies could be:\n - the choice of edge\n - the attention block\n - removing both graphs\n - fitler-out some specific POS-tagging to see whether some keywords are more important.\nIt is also sometimes interesting to point out what part of the network may require additional capacity: should we make the bi-LSTM bigger first? The decoder? \n\nAnother interesting experiment would be to run the same architecture on a non-dialogue dataset (by naively setting some of the edges). Therefore, it could show whether the architecture can also perform well on non-dialogue long dependency.\nFinally, the authors (rightly) complain that there is a lack of abstract summarisation. However, it is still possible to take the Ubuntu dataset, run a baseline + KGDEC, and perform a human evaluation. \nI here list some potential experiments, and I am aware that it is unreasonable to perform them all. I mostly want to point out that there is a large spectrum of things that can be done to further demonstrate the validity of their model. \n\nOn a second note, some of the neural hyperparameters are missing, such as the convolution kernel/hidden_size to process the vertices. \n\nOverall, the paper has some merits. Easy to read, plenty of implementation details (although some parameters are missing). However, i cannot recommend paper acceptance without at least two or three additional experiments and potentially the error bar.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "n6KdF4NgFR_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the paper about, what contributions does it make, what are the main strengths and weakness?\n\nThe paper proposes a novel framework, Knowledge Graph Enhanced Dual-Copy network (KGEDC) for abstractive dialogue summarization. Conversational structure and factual knowledge are incorporated in this framework based on graph network to deal with long-distance cross-sentence dependencies and faithfulness respectively. This framework can be decomposed of 1) a sequence encoder to capture contextual information of dialogues flowing along the sequence, 2) a graph encoder via sparse relation graph self-attention network for cross-sentence dependencies, 3) a factual knowledge graph for representing relational tuples extracted from dialogues, 4) a dual-copy decoder to focus on both the input tokens and the factual knowledge. Experimental results on two datasets show the performance gains of the proposed methods over several previous baselines.\n\nSTRENGTHS:\n\n1. The motivation is clear and is in line with the model architecture. Two main unresolved problems in abstractive dialogue summarization are consistently concerned in the paper.\n\n2. The experimental results is rather strong and solid. The paper includes the results of different baseline methods for comparison, showing considerable boosts in both automatic and human evaluation metrics.\n\n3. The paper organization and writing is coherent. Besides, the model description is also well detailed.\n\nWEAKNESS:\n\n1. Dialogue graph may not necessarily be sparse. In section 3.2.3, “Sparse Relational Graph Self-Attention Layer” paragraph, the author writes “However, our sparse self-attention operation …, which reduces the quadratic computation to linear”, but no proof is given, actually this seems not to be right. In a dialog where only two persons interchange words in turn, the number of edges for speaker dependency will still be quadratic to turn length.\n\n2. The constructed factual knowledge graph seems to be sparse since the topics various in different dialogues. As shown in figure 2 (b), there are 7 different edge types in one dialogue session. How does the model deal with this issue? \n\n3. Maybe the paper should provide more content on ablation and case study. For example, in equation (3), edge type is considered in attention computation, while the impact for distinguishing 3 different edges is not studied. Moreover, cases, where the proposed method doesn’t perform well, may also be interesting while not included.\n\n4. In figure 1, both Factual Knowledge Graph and Sequence Encoder are used in predicting the next word at the decoder stage, in an attention manner, but why graph encoder is excluded here? Now that utterance representations have been encoded in a graph encoder, attending to these representations may help in doing better decoding.\n\nTypos. Grammar, and Style\n\n1. just beneath equation (3) in section 3.2.3:  w_i^r should be w_i^R. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AT7jJ1BlQzN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a knowledge graph enhanced network to improve abstractive dialog summarization with graphs constructed from the dialog structure and factual knowledge. The dialog graph is composed of utterances as nodes and 3 heuristic types of edges (such as utterances of the same speaker, adjacent utterances). The factual graph is constructed via openIE and dependency parsing, which the authors claim are complementary as the triplets (results of openIE) are not always available. \n\n---\nPros:\n+ The proposed method outperforms all the compared baselines on two dialog summarization datasets.\n+ Human evaluation shows that the proposed method leads to increased relevance and readability.\n\n---\nCons:\n- In the ablation study (Table 3), the performance of each variant is close to the full model, and removing either module still outperforms the compared baselines. Given such performance, I wonder if the difference between the ablated variants and the full model is statistically significant. Also, what is the performance of the proposed method without graph information? Is it effectively the same as a Pointer-Generator?\n- Details of human evaluation are lacking. Who are the annotators and how many annotators are there? What is the inter-annotator agreement?\n- The paper is poorly represented with unclear descriptions (not only typos/grammar but also definitions of various concepts), which makes it hard to follow. To name a few, in 3.2, the definition of edges gives one the impression that it is a fully connected graph. In 3.2.1, “keyword” is defined after the description of the use of the keyword, and I can’t really tell what is the “keyword neighborhood”.  A general suggestion is to define them beforehand (when they first appear) instead of describing them later or in the footnote.\n\n---\nComments for rebuttal and revised paper\n\nThanks for providing a detailed response and an improved version of the paper. One thing that I am still concerned with is how come the updated ablation study is so different from the initial results. Originally, the differences between KGEDCg and KGEDCg-GE and KGEDCg-FKG were very minor (one of my questions above), but now the margins are as large as 7+ pts. Given such discrepancies without explanation, I'd hold my original evaluation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ay9m7PByns", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve dialogue summarization by encoding the text with a sequential encoder (for token-level contextualization) and a graph encoder (for long-distance and semantic contextualization). A KG is built and considered to be a surrogate for \"factual knowledge\". A dual-copy mechanism is used while decoding in the hope that direct access to this factual knowledge will enhance the faithfulness of the generated summaries.\n\nThe authors use a biLSTM to encoder the utterances. They build a dialogue graph where each utterance is a node and 2 nodes are connected if they have the same speaker, are within a distance d sequentially, or if they have common keywords (nouns, numerals,\nadjectives or notional verbs). They also build the KG using OpenIE triples and tuples from the utterance dependency trees. They use a GNN to encode both the utterance graph and the KG and use either concatenation or gated fusion to get the context vector. They copy from both the text sequence and KG during decoding.\n\nThey evaluate the method on two datasets: SAMSum corpus, Automobile Master corpus. They get a Rouge-L improvement of ~1.8 and ~2.5 respectively over string baselines. \n\nPros:\n- The method is interesting. Incorporating facts into text generation is an important and interesting area.\n- The results look good. They also perform human evaluation which is appreciated.\n\nCons/Questions:\n- It is not clear how the biLSTM encoding and the GE encoding are combined? Is it the same way as done with h^S and h^G?\n- I am not satisfied with the ablation study. One important ablation to run would be an experiment without both GE and FKG. Without those two, the model simply becomes (seq2seq + attention). And from Table 1, we see that the R-L for that simple model is 28.16 and 29.37. It is hard for me to convince myself that adding either GE or FKG increases R-L by more than 12 points.\n- It would also be interesting to know in what ratio h_G and h_S are combined in the case of gating fusion.\n- Since there is already sequential context dependency in GE, what additional advantage does the sequential encoder provide? Why not train only with the graph encoders?\n\nI would consider updating the rating once the authors respond.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the important task of improving dialogue summarization using conversation structure and factual knowledge.

Pros:
1) Clearly written and well motivated (as acknowledge by all reviewers)
2) Technically sound (the proposed architecture is clearly in line with the problem that the authors are trying to solve)
3) Significant upgrades to the paper after the reviewer comments (in particular the authors have added detailed ablation studies and results on non-dialogue datasets)

Cons:
1) There is a significant difference between the results in the ablation studies in the original version and in the new version. Originally, the differences between KGEDCg and KGEDCg-GE and KGEDCg-FKG were very minor, but now the margins are as large as 7+ pts. I would request the authors to explain this in the final version

The reviewing team felt that while many Qs were sufficiently addressed by the authors, the large difference in the numbers reported for the ablation study in the initial and final version of the paper raises some new Qs which need to be addressed before the paper can be accepted. 
Decision: Reject