Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes SURGE, a GNN-based context-relevant subgraph retrieval method for KG-augmented dialogue generation. The proposed method considers the permutation invariance and relation inversion invariation, and uses graph-text contrastive learning to generate knowledge-consistent response.
strength_and_weaknesses: ### Strengths
* The paper is overall written clearly.
* The proposed method is a thoughtful combination of important details such as the invariant graph encoding and graph-text contrastive learning, which convincingly can help the training.
* The proposed evaluation metric KQA could be useful.

### Weaknesses
* The paper misses references to prior works on dialogue generation priorly or jointly extracts subgraphs, which to my understanding are closely related to this work, such as:
  * Jung, Jaehun, Bokyung Son, and Sungwon Lyu. "Attnio: Knowledge graph exploration with in-and-out attention flow for knowledge-grounded dialogue." In EMNLP 2020. 
  * Zhou, Hao, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. "Commonsense knowledge aware conversation generation with graph attention." In IJCAI 2018.
  * Tuan, Yi-Lin, Sajjad Beygi, Maryam Fazel-Zarandi, Qiaozi Gao, Alessandra Cervone, and William Yang Wang. "Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems." In Findings of ACL 2022.
  * Yang, Shiquan, Rui Zhang, and Sarah Erfani. "GraphDialog: Integrating Graph Knowledge into End-to-End Task-Oriented Dialogue Systems." In EMNLP 2020.
* Since the proposed method is a combination of subgraph retrieval and dialogue generation, I would like to see comparisons to prior works that also do dialogue generation attending to parts of a graph, e.g., the methods in above listed references.
* About the main motivation across the paper, I have a few concerns:
  * The title is too general according to the scope of this paper. If the main novelty of the proposed method is the subgraph-retrieval augmented method for dialogue generation, I would suggest the title to be more specific, e.g., mentioning “subgraph-retrieval”.
  * About the second paragraph in the introduction, it may not always be true in real applications that only a few hops in a KG can compose a good response. I would suggest revising the paragraph to be more general instead of motivating by one specific dataset that was collected for research purpose and only contained few cases in the real world.
  * The paper claims to jointly retrieve context-relevant subgraph, however, in the experiments, it seems that the “subgraph-retrieval” is based on a manually selected subgraph candidates beforehand i.e., only considering 1-hop in OpenDialKG and 2-hop in KOMODIS. Am I misunderstanding this?
  * Following question c, if the proposed method is based on pre-selected candidates, many prior works already consider such cases (as listed in question (a)). How general is this method when applying to cases with a variant number of hops? How would you analyze its scalability and complexity?


clarity,_quality,_novelty_and_reproducibility: #### Clarity:
The paper is overall clear to get the idea. 

#### Quality:
The paper overall has a good quality for writing, method design, and experiments given its current scope.

#### Novelty:
The paper misses some highly relevant works, which did dialogue generation by attending only parts of a KG. These methods are not discussed or compared in this paper.

#### Reproducibility:
The paper could be reproducible given the provided code. For the paper side, the writing of Section 3-4 can be more clear for reproducibility. For example, the first time defining $y_{<t}$ may be changed to define $y_T$ first, which was also mentioned afterwards but was not defined. Furthermore, about the KQA evaluation, I’m not sure if you calculated the KQA metric with another trained model (BERT according to the appendix) or you used the same dialogue generation model but utilized a mismatched input format to query the output? Both seem a bit weird to me.

summary_of_the_review: With my listed weakness, I would suggest considering more about the line of knowledge graph attention and reasoning works into motivation, related work, and experiments (some works are listed above), or adjusting the scope of the current claim.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an approach to perform knowledge grounded dialogue generation. Their GNN based approach retrieves a relevant subgraph from the KG to efficiently generate an appropriate response. They also propose a permutation and relation inversion invarient yet efficient way to encode the graph information along with the text using a contrastive learning objective that ensures that the generated responses reflect the retrieved knowledge. Finally, they propose a novel metric called Knowlege-verifying Question Answering to evaluate whether the generated responses contain the correct knowledge.
strength_and_weaknesses: Strengths:
* This paper is really well written with extensive experiments and analysis to support the claims in the paper.
* Their approach and proposed metric can be a valuable contribution to the community in the space of knowledge grounded generation especially since it is more efficient than previous approaches using KGs.
*  They talked about their design choices and assumptions in detail.

Weakness:
* Some of the parts in this paper could be better explained, especially the new proposed metric in section 4. It was unclear what this metric is exactly. 
* This paper lacks comparison with approaches that use commonsense KGs in a similar way.
* Using subgraphs is not entirely novel. (Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A Survey of Knowledge-Enhanced Text Generation. ACM Comput. Surv. Just Accepted (March 2022). https://doi.org/10.1145/3512467 talk about it in much more detail), though the way the authors find a subgraph might be novel.
* The mapping function is unclear. Why did you split New York into New and York? Why not just use the base model's tokenizer? If you did use the tokenizer, then instead of calling this a "mapping function" you can simply refer it as "the entity name was directly passed to the PLM to get its encoding". It is also unclear how you did entity linking to get the KG entities from the context. Do you do exact match? 
* What is a) in Figure 2 caption?
* From reading section 3.3 It is not clear if you're using all triplets present in the KG at each step to filter and obtain a subgraph. After reading the appendix (limitations) as well as line 2 in last para of 3.3, it's clear that you're using only the k hop neighborhood of the entities present in the context. Again, how do you do the linking? Also, wont you miss out on potential entities that are not exactly mentioned in the context? Did you consider also using relations? (eg: (Barack Obama, president of, Usa) is present in your KG but the context is "Who is the top-person of the american armed forces" or something, then you'll miss out on Barack Obama).
  * In the limitations, you do discuss limitations on entity linking but have not provided how you are doing the same (I might have missed it).
* Please describe the relation embedding matrix in the main paper section 3.3.
* Human evaluation using just 30 samples might not be sufficient for statistical significance. If possible, please try with 100 samples. In any case, please report statistical significance.
* The metric is only slightly correlated with human judgement and more evaluation of the metric is needed before it can be widely used. 


Questions:
* How did you arrive at the 87% figure for facts that are irrelevant to the context in OpendialKG dataset (2nd para 4th line in Intro)?
* Why did you choose zero vector for initialization of node embeddings?
* What is EM that is reported in Table 1? ( I think you can fix this by describing the proposed metric a bit more).
* What is F1 vs KF1 in Table 3?


Grammar and nitpicks:
* The related works section lacks some discussion on works which use GNNs in different ways in dialogue systems. Some quick google search examples at the end.
* Figure 2 caption last line - "finally we use ~a~ contrastive learning to enforce".

Suggestions:
* Update the hyperlinks to Figures, and tables to include link in the "Figure" or "Table" keyword too instead of just linking the number.
* Please add more details in section 4. 

Some other works that use GNNs in dialogue (from a quick google search) (I'm not asking you to cite them just that it would be nice to talk about other dialogue works that use GNNs) :
1) - Chen, L., Lv, B., Wang, C., Zhu, S., Tan, B., & Yu, K. (2020). Schema-Guided Multi-Domain Dialogue State Tracking with Graph Attention Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 7521-7528.
2) Mauajama Firdaus, Nidhi Thakur, and Asif Ekbal. 2020. MultiDM-GCN: Aspect-guided Response Generation in Multi-domain Multi-modal Dialogue System using Graph Convolutional Network. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2318–2328, Online. Association for Computational Linguistics.
3) W. Nie, R. Chang, M. Ren, Y. Su and A. Liu, "I-GCN: Incremental Graph Convolution Network for Conversation Emotion Detection," in IEEE Transactions on Multimedia, doi: 10.1109/TMM.2021.3118881.
4) X. Zhao, L. Chen and H. Chen, "A Weighted Heterogeneous Graph-Based Dialog System," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2021.3124640.
5) L. Qin, W. Che, M. Ni, Y. Li and T. Liu, "Knowing Where to Leverage: Context-Aware Graph Convolutional Network With an Adaptive Fusion Layer for Contextual Spoken Language Understanding," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1280-1289, 2021, doi: 10.1109/TASLP.2021.3053400.
6) Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan Black, and Yulia Tsvetkov. 2021. Dialograph: Incorporating interpretable strategygraph networks into negotiation dialogues. In International Conference on Learning Representations.
7) [AttnIO: Knowledge Graph Exploration with In-and-Out Attention Flow for Knowledge-Grounded Dialogue](https://aclanthology.org/2020.emnlp-main.280) (Jung et al., EMNLP 2020)
8) Zhou, Li and Kevin Small. “Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering.” ArXiv abs/1911.06192 (2019): n. pag.
9) C. -M. Wong et al., "Improving Conversational Recommender System by Pretraining Billion-scale Knowledge Graph," 2021 IEEE 37th International Conference on Data Engineering (ICDE), 2021, pp. 2607-2612, doi: 10.1109/ICDE51399.2021.00291.
10) Liu, Z., Niu, Z., Wu, H., & Wang, H. (2019). Knowledge Aware Conversation Generation with Explainable Reasoning over Augmented Graphs. EMNLP.
11) Jing Li, Qingbao Huang, Yi Cai, Yongkang Liu, Mingyi Fu, Qing Li, Topic-level knowledge sub-graphs for multi-turn dialogue generation, Knowledge-Based Systems, Volume 234, 2021,
12) Chen, Yu, Lingfei Wu, and Mohammed J. Zaki. "Toward Subgraph Guided Knowledge Graph Question Generation with Graph Neural Networks." arXiv preprint arXiv:2004.06015 (2020).
clarity,_quality,_novelty_and_reproducibility: This paper is very well written with sufficient justification on the claims made in this work. The paper is of high quality with certain novel proposals that might be beneficial for the community. The paper has sufficient details to reproduce the work and after a high level look on the code added, I can say that I might be able to reproduce it satisfactorily.
summary_of_the_review: This paper provides a novel approach to filter knowledge from KGs efficiently and make dialogue responses follow the retrieved knowledge.  This is a well written paper with adequately supported claims using extensive experiments and analysis. The metric proposed in this work can be used by the community for evaluating knowledge grounded generation better. The strengths far outweigh the weaknesses, which are mostly to do with clarifications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: What is the goal of the paper?

* Knowledge-consistent dialogue generation with the KG

What has been done before?

* Many existing methods do not guarantee that the model utilizes a relevant piece of knowledge from the KG before generating knowledge-consistent dialogues. The ones which do simply match and retrieve all facts for entities that appear in the dialogue context, which either may mislead the agent to generate out-of-context responses from irrelevant facts or can increase the computational overheads for prepending tokens for all facts in PLMs. This work differs from those existing works, since it aims at retrieving only a context-relevant subgraph among all associated facts with a novel GNN-based subgraph retriever, which is end-to-end trainable along with a dialogue generation model.

What are the contributions of the paper?

* Propose SUbgraph Retrieval-augmented GEneration (SURGE), a end-to-end dialogue generation framework for generating context-relevant and knowledge-consistent dialogues with a KG that considers all aspects from knowledge retrieval, encoding, and reflection along the generation process.

* Introduced an additional performance metric, referred to as Knowledge-verifying Question Answering (KQA), which evaluates whether the generated responses contain the correct knowledge with an additional extractive question answering scheme.

* Proposed a GNN-based context-relevant subgraph retrieval method for KG-augmented dialogue generation, to extract only the relevant piece of the knowledge for the dialogue context from the entire knowledge graph. 

* Proposed an invariant yet efficient graph encoder and a graph-text contrastive learning objective to ensure that the generated responses faithfully reflect the retrieved knowledge. 

* Evaluated SURGE against relevant baselines on the OpendialKG and KOMODIS datasets , demonstrating its efficacy in generating responses that are more informative by retrieving and reflecting the relevant knowledge from the KG.





strength_and_weaknesses: 
Strengths

* Empirically showed the importance of retrieving the more relevant subgraph knowledge rather than using all the relevant knowledge graphs when generating knowledge-grounded response

* Proposed multiple components for retrieval, encoding, and graph-text representation learning

* Better results than baselines on OpendialKG and KOMODIS datasets

* Source code is available for reproducibility


Weaknesses

* KQA correlation with human ratings is small

* Is human evaluation performed on only 30 examples?


clarity,_quality,_novelty_and_reproducibility: Paper was not an easy read.

Work looks novel and reproducible due to availability of source code.

Improvements in writing needed.
summary_of_the_review: Paper has many strengths with scope of improvements in writing. 


Strengths

* Empirically showed the importance of retrieving the more relevant subgraph knowledge rather than using all the relevant knowledge graphs when generating knowledge-grounded response

* Proposed multiple components for retrieval, encoding, and graph-text representation learning

* Better results than baselines on OpendialKG and KOMODIS datasets

* Source code is available for reproducibility


Weaknesses

* KQA correlation with human ratings is small

* Is human evaluation performed on only 30 examples?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Existing knowledge-grounded dialogue generation models with KG suffer from irrelevant knowledge and inconsistent generation. To attack the above two problems, this paper proposes a GNN-based context-relevant model to retrieve relevant subgraphs from KG, an invariant yet efficient graph encoder, and a graph-text contrastive learning objective which overcome the exposure bias in teacher forcing to guarantee knowledge-consistent generation. Extensive experiments have been conducted on OpendialKG and KOMODIS and the results show the superiority of SURGE over baselines.
strength_and_weaknesses: **Strengths**

The paper is generally well-written. 

The study of generating context-relevant and knowledge-consistent dialogues with a KG is important and interesting. I appreciate the authors' investigation of this important problem. 

The three components (the GNN-based context-relevant subgraph retrieval method, the graph encoder, and a graph-text contrastive learning objective) seem to be new. Experimental results show the empirical advantage of the new methods.

**Weaknesses**

The overall framework is not new, very similar to (Lewis et al., 2020b). In fact, the authors should make this point more explicit.

Although the three components are somewhat new, neither is of high originality, and neither is throughly investigated (see more comments below). This paper seems to belong to a kind of assembly innovation: putting several pieces together, while each piece brings a small adjust. I suggest the authors to focus on major novelty and fully validate the novelty.

More ablation studies are needed to show some significant benefit of each component, by fixing other components while modifying one component. Now the significance of each component is not clear. It is not good to simply say that "We empirically verify that the use of GNN as the triplet embedding yields the better retrieval performance compared to previous PLM-based methods."

In Table 1 and Table 3, the advantages of SURGE (contrastive) are not significant, compared with SURGE (unsupervised) and SURGE (semi-supervised), especially considering SURGE (contrastive) also consists of semi-supervised learning.

In Table 1, SURGE (unsupervised) outperforms the other two SURGE performance on BLEU, ROUGE and Unigram, is there any explanation for that?

Using only unique entities does not guarantee permutation invariant.

Some problems with writing:

Section 3.4 (INVARIANT GRAPH ENCODING) is hard to be understood, even after reading the Appendix.

The two examples in Figure 1 illustrate the two problems - irrelevant knowledge and inconsistent generation. Somewhat confusing. "Irrelevant facts" and "factually wrong" actually may express almost the same meanings.

maaping -> mapping

duplicate definition of SURGE (semi-supervised) on the bottom of page 7

Some places should be re-written to be clearer in both grammar and meaning:

we utilize the Graph Neural Networks (GNNs) in the triplet embedding function 
-> we utilize the Graph Neural Networks (GNNs) for the triplet embedding function ?

The 2nd paragraph, Section 3.1: Any entity and relation are mapped to a sequence of l tokens. Is the length l fixed for different entities and relations?

--- **To reply to Follow-up Response from the authors (Provided here, since the comment button is disabled 2022/12/12)** ---

Using $\eta$ alone should be an important baseline to be compared.

If you think that Eq.(7) is the first in developing GNN-based entity embedding in the literature (not necessarily for dialog generation), then you can clearly say so in the paper. Suppose this is yes, then the contribution should be stated around this, rather than putting more emphasis on "Invariant and Efficient Graph Encoding". To my understanding, "Invariant and Efficient" are actually down to only encoding the unique entities, which essentially is entity embedding. Clarity issues remain.

I can see the ablation result in Table 3. My concern is that the overall performance of SURGE (contrastive) is limited. In real applications, the model needs to work with retrieved knowledge.
clarity,_quality,_novelty_and_reproducibility: see above.
summary_of_the_review: see above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach to knowledge-grounded dialogue. The key problems the paper intends to tackle is the selection and encoding of relevant knowledge according to the dialogue context. To this end, subgraphs are selected using the context. Subgraph encoding is asked to meet some conditions: permutation invariance and relation inversion invariance. Two tricks are used to satisfy these conditions: by sorting the triples, and by adding the inverse relation.
The proposed method is compared to a set of baselines, in which either random knowledge selection, or a selection using T5 is used. The experiments show that the proposed method outperforms the baselines.
In addition, a knowledge verifying metric - KF1 - is used to measure if the generated response is consistent with the known knowledge.
The overall ideas of locating relevant subgraphs and doing subgraph encoding are interesting.
strength_and_weaknesses: The idea of selecting relevant subgraphs from a knowledge graph is well motivated. This is a known problem. There are approaches proposed to do it. This paper also relies on an existing tool to identify the entities in the knowledge graph and to select subgraphs.
The two conditions for subgraph encoding are important in some cases. This paper proposes interesting ideas to satisfy them.
However, the tricks used to satisfy these conditions may raise some questions:
1. The permutation invariance condition is satisfied by sorting the triples. This is a kind of normalization, that transforms a subgraph to a normalized form. Even though the output will be permutation invariant, it does not require that the encoding process after the transformation be permutation invariant. The latter is what one intend to obtain by imposing the condition. So, it is not sure that making this transformation would make the encoding better.
2. The encoding process then considers a set of triples as a sequence of tokens. that is submitted to a PLM for encoding. In this step, one may lose the specific triple structure for relations. Interpreting a set of triples as a sentence makes the encoding easier, but may alter the meaning.
3. In the approach to increase efficiency, it seems that the entities from different triples are mixed up, then sorted. The purpose is to encode an entity only once. However, this process would alter the meaning of the initial subgraph. For example, the initial set of triples [(a,r1,b), (c,r2,d)] would become the same as [(a,r2,c),(d,r1,b)] after this operation. I understand that this operation would transform both into a set of entities like [a,b,c,d,r1,r2] (this part is not very clear). However, the two different sets of triples may mean very different things and we may not want to unify them.
4. The relation inversion invariance is imposed to all the repations. If the relation is kept the same after inversion, this does not make sense. It seems that the inversed relation is different:  ~relation. In this case, I do not see what new information this inversion could add. For example, saying that (a, born-in, b) would be the same as saying (b,  ~born-in, a). What's the benefit of adding the latter?

In the experiments, it seems that the way subgraphs are retrieved is quite simple. There could be more sophisticated methods in the literature. For example [1] proposes a knowledge selection network to make the selection. Given this fact, it is unsure that the method is compared to the state-of-the-art knowledge selection approaches.

[1] Yutao Zhu, Jian-Yun Nie, Kun Zhou, Pan Du, Hao Jiang, and Zhicheng Dou, Proactive Retrieval-based Chatbots based on Relevant Knowledge and Goals, SIGIR 2021.
clarity,_quality,_novelty_and_reproducibility: The description is mostly clear and easy to follow.
The experiments are done correctly. The use of knowledge verifying measure is very interesting.
The implementation details are described in detail in annex. The source code is provided. The method can be reasonably reproduced.
summary_of_the_review: The idea is very well motivated. The paper proposes some way to encode the retrieved subgraphs so that the pieces of knowledge can be better encoded.
However, there are questions about the way the subgraphs are encoded. It may lead to undesirable cases.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes subgraph retrieval-augmented generation (SURGE), a framework for knowledge-grounded dialogue. It can be divided into 3 major components:
(i) a subgraph-retriever, which retrieves only  context-relevant triples from a KG,
(ii) a permutation and inversion invariant graph encoding scheme, and
(iii) a contrastive learning objective to enforce consistency between the retrieved triples and generated text

The authors also propose a new metric called knowledge-verified question-answering (KQA) to evaluate the generated responses. They evaluate the method on OpendialKG and KOMODIS. Both see fair improvements over the baselines on KQA and minimal improvements on lexical overlap metrics like bleu and rouge. 

The paper is well-written and understandable. However,
(i) some space could be saved in section 3.4 and used for ablations and analysis.
(ii) I would have liked to see more information about sensitivity analysis on graph encodings and a discussion on why the authors think retrieving proper subgraphs is easier/more important than say, teaching the decoder to look for/ignore relevant/irrelevant knowledge conditioned on the history. Some toy experiments would have cemented the premise of the paper.
(iii) contrastive training does not give any significant benefit. Why is that? One would intuitively think otherwise. Some analysis there would be appreciated.

There are minor writing errors. E.g.: in section 5.2, the last line should be SURGE (contrastive)
strength_and_weaknesses: -
clarity,_quality,_novelty_and_reproducibility: -
summary_of_the_review: -
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose an integrated solution to combine knowledge subgraphs and language models for dialogue systems. The authors propose SUbgraph Retrieval-augmented GEneration (SURGE), which retrieves a subgraph from the KG. Then, it learns a latent representation space using contrastive learning. The AE and the reviewers have reached a consensus about the following strengths: 

+++ This is a very important direction in dialogue systems research.
+++ The authors try to explicitly model the contextual relevance, which is a good idea.

The panel has also found several limitations and concerns:
--- The novelty is incremental. All techniques are not new, and the authors assemble the three techniques, especially the idea of subgraph retrieval is not very new. 
--- Several reviewers are concerned about the invariance of the triples encoding. Reviewers are not satisfied with the authors' reply. The proposed method might depend on the order. The implementation might have an issue, and imposing the order is not the best solution. 
--- The related work still needs work, and some subgraph retrieval work is ignored in related studies. It is unclear if the comparison with DiffKG is conducted in a fair setting. 
--- Contrastive learning results are not very impressive. 
--- Ablation experiments are lacking: different graph encoding mechanisms would be experiments (no comparisons).

Based on the panel discussion and the AC-reviewer Zoom meeting, the panel has concluded that this paper is not ready to be accepted.

justification_for_why_not_higher_score: Novelty and other issues.
Decision: Reject