Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new neural pipeline for dialogue summarization that jointly includes word-by-word decoding, an utterance graph, and a factual knowledge graph. The proposed methods is assessed on the SamSUN and  Automobile Master dataset.
The method improves the top reported baselines by 0.5 to 2 points (RED) in both cases.

The paper contains a well-motivated introduction and perform a sound related work section (as far as my judge). They correctly detail their methods in a step-by-step pedagogical procedure. Overall the paper is pretty well-written. 

However, I have a few concerns about the experimental section. While this paper's main contribution is a (sensible) mixture of neural blocks, a single page of experiments to evaluate the method is not enough from my perspective. Besides, a 0.5-2 pts increase compared to a Pointer network may be a bit limited in light of the model complexity.
However, my main concern is mostly the lack of analysis of the method. As there are many design choices, this requires multiple ablation studies to validate all of them. Yet, the authors only ablate the factual knowledge graph and graph encoder. Furthermore, the score differences are small, and a few run + std would have been useful to determine whether one change is significant.
Ideas for other ablation studies could be:
 - the choice of edge
 - the attention block
 - removing both graphs
 - fitler-out some specific POS-tagging to see whether some keywords are more important.
It is also sometimes interesting to point out what part of the network may require additional capacity: should we make the bi-LSTM bigger first? The decoder? 

Another interesting experiment would be to run the same architecture on a non-dialogue dataset (by naively setting some of the edges). Therefore, it could show whether the architecture can also perform well on non-dialogue long dependency.
Finally, the authors (rightly) complain that there is a lack of abstract summarisation. However, it is still possible to take the Ubuntu dataset, run a baseline + KGDEC, and perform a human evaluation. 
I here list some potential experiments, and I am aware that it is unreasonable to perform them all. I mostly want to point out that there is a large spectrum of things that can be done to further demonstrate the validity of their model. 

On a second note, some of the neural hyperparameters are missing, such as the convolution kernel/hidden_size to process the vertices. 

Overall, the paper has some merits. Easy to read, plenty of implementation details (although some parameters are missing). However, i cannot recommend paper acceptance without at least two or three additional experiments and potentially the error bar.

review_title: Improving Abstractive Dialogue Summarization

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to improve dialogue summarization by encoding the text with a sequential encoder (for token-level contextualization) and a graph encoder (for long-distance and semantic contextualization). A KG is built and considered to be a surrogate for "factual knowledge". A dual-copy mechanism is used while decoding in the hope that direct access to this factual knowledge will enhance the faithfulness of the generated summaries.

The authors use a biLSTM to encoder the utterances. They build a dialogue graph where each utterance is a node and 2 nodes are connected if they have the same speaker, are within a distance d sequentially, or if they have common keywords (nouns, numerals,
adjectives or notional verbs). They also build the KG using OpenIE triples and tuples from the utterance dependency trees. They use a GNN to encode both the utterance graph and the KG and use either concatenation or gated fusion to get the context vector. They copy from both the text sequence and KG during decoding.

They evaluate the method on two datasets: SAMSum corpus, Automobile Master corpus. They get a Rouge-L improvement of ~1.8 and ~2.5 respectively over string baselines. 

Pros:
- The method is interesting. Incorporating facts into text generation is an important and interesting area.
- The results look good. They also perform human evaluation which is appreciated.

Cons/Questions:
- It is not clear how the biLSTM encoding and the GE encoding are combined? Is it the same way as done with h^S and h^G?
- I am not satisfied with the ablation study. One important ablation to run would be an experiment without both GE and FKG. Without those two, the model simply becomes (seq2seq + attention). And from Table 1, we see that the R-L for that simple model is 28.16 and 29.37. It is hard for me to convince myself that adding either GE or FKG increases R-L by more than 12 points.
- It would also be interesting to know in what ratio h_G and h_S are combined in the case of gating fusion.
- Since there is already sequential context dependency in GE, what additional advantage does the sequential encoder provide? Why not train only with the graph encoders?

I would consider updating the rating once the authors respond.
review_title: Incorporating factual knowledge into dialogue summarization

final_score: rating: 6: Marginally above acceptance threshold
review_content: What is the paper about, what contributions does it make, what are the main strengths and weakness?

The paper proposes a novel framework, Knowledge Graph Enhanced Dual-Copy network (KGEDC) for abstractive dialogue summarization. Conversational structure and factual knowledge are incorporated in this framework based on graph network to deal with long-distance cross-sentence dependencies and faithfulness respectively. This framework can be decomposed of 1) a sequence encoder to capture contextual information of dialogues flowing along the sequence, 2) a graph encoder via sparse relation graph self-attention network for cross-sentence dependencies, 3) a factual knowledge graph for representing relational tuples extracted from dialogues, 4) a dual-copy decoder to focus on both the input tokens and the factual knowledge. Experimental results on two datasets show the performance gains of the proposed methods over several previous baselines.

STRENGTHS:

1. The motivation is clear and is in line with the model architecture. Two main unresolved problems in abstractive dialogue summarization are consistently concerned in the paper.

2. The experimental results is rather strong and solid. The paper includes the results of different baseline methods for comparison, showing considerable boosts in both automatic and human evaluation metrics.

3. The paper organization and writing is coherent. Besides, the model description is also well detailed.

WEAKNESS:

1. Dialogue graph may not necessarily be sparse. In section 3.2.3, “Sparse Relational Graph Self-Attention Layer” paragraph, the author writes “However, our sparse self-attention operation …, which reduces the quadratic computation to linear”, but no proof is given, actually this seems not to be right. In a dialog where only two persons interchange words in turn, the number of edges for speaker dependency will still be quadratic to turn length.

2. The constructed factual knowledge graph seems to be sparse since the topics various in different dialogues. As shown in figure 2 (b), there are 7 different edge types in one dialogue session. How does the model deal with this issue? 

3. Maybe the paper should provide more content on ablation and case study. For example, in equation (3), edge type is considered in attention computation, while the impact for distinguishing 3 different edges is not studied. Moreover, cases, where the proposed method doesn’t perform well, may also be interesting while not included.

4. In figure 1, both Factual Knowledge Graph and Sequence Encoder are used in predicting the next word at the decoder stage, in an attention manner, but why graph encoder is excluded here? Now that utterance representations have been encoded in a graph encoder, attending to these representations may help in doing better decoding.

Typos. Grammar, and Style

1. just beneath equation (3) in section 3.2.3:  w_i^r should be w_i^R. 

review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes a knowledge graph enhanced network to improve abstractive dialog summarization with graphs constructed from the dialog structure and factual knowledge. The dialog graph is composed of utterances as nodes and 3 heuristic types of edges (such as utterances of the same speaker, adjacent utterances). The factual graph is constructed via openIE and dependency parsing, which the authors claim are complementary as the triplets (results of openIE) are not always available. 

---
Pros:
+ The proposed method outperforms all the compared baselines on two dialog summarization datasets.
+ Human evaluation shows that the proposed method leads to increased relevance and readability.

---
Cons:
- In the ablation study (Table 3), the performance of each variant is close to the full model, and removing either module still outperforms the compared baselines. Given such performance, I wonder if the difference between the ablated variants and the full model is statistically significant. Also, what is the performance of the proposed method without graph information? Is it effectively the same as a Pointer-Generator?
- Details of human evaluation are lacking. Who are the annotators and how many annotators are there? What is the inter-annotator agreement?
- The paper is poorly represented with unclear descriptions (not only typos/grammar but also definitions of various concepts), which makes it hard to follow. To name a few, in 3.2, the definition of edges gives one the impression that it is a fully connected graph. In 3.2.1, “keyword” is defined after the description of the use of the keyword, and I can’t really tell what is the “keyword neighborhood”.  A general suggestion is to define them beforehand (when they first appear) instead of describing them later or in the footnote.

---
Comments for rebuttal and revised paper

Thanks for providing a detailed response and an improved version of the paper. One thing that I am still concerned with is how come the updated ablation study is so different from the initial results. Originally, the differences between KGEDCg and KGEDCg-GE and KGEDCg-FKG were very minor (one of my questions above), but now the margins are as large as 7+ pts. Given such discrepancies without explanation, I'd hold my original evaluation.

review_title: Official blind review

Meta-review: The authors address the important task of improving dialogue summarization using conversation structure and factual knowledge.

Pros:
1) Clearly written and well motivated (as acknowledge by all reviewers)
2) Technically sound (the proposed architecture is clearly in line with the problem that the authors are trying to solve)
3) Significant upgrades to the paper after the reviewer comments (in particular the authors have added detailed ablation studies and results on non-dialogue datasets)

Cons:
1) There is a significant difference between the results in the ablation studies in the original version and in the new version. Originally, the differences between KGEDCg and KGEDCg-GE and KGEDCg-FKG were very minor, but now the margins are as large as 7+ pts. I would request the authors to explain this in the final version

The reviewing team felt that while many Qs were sufficiently addressed by the authors, the large difference in the numbers reported for the ablation study in the initial and final version of the paper raises some new Qs which need to be addressed before the paper can be accepted. 
Decision: Reject