Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The work aims at avoiding generating factually incorrect responses by introducing knowledge from knowledge graph (KG) into a conversation. It retrieval only context-relevant sub-graph from the global KG to avoid unnecessary encoding overhead and propose an invariant graph encoder to encode the graph.
strengths_and_weaknesses: Strength:

The proposed new graph encoding achieves permutation invariance and relation inversion invariance, which I like.

Weaknesses:

1. Inadequate baseline. Some classic and recent methods are not compared or discussed. For example, [1][2][3]

2. Missing ablation study. The effectiveness of the newly-proposed graph encoder is unclear as there is no ablation experiment.

[1]Wu, Sixing, et al. "Diverse and informative dialogue generation with context-specific commonsense knowledge awareness." Proceedings of the 58th annual meeting of the association for computational linguistics. 2020.

[2]Zhang, Houyu, et al. "Grounded conversation generation as guided traverses in commonsense knowledge graphs." arXiv preprint arXiv:1911.02707 (2019).

[3]Cui, Fuwei, et al. "Syntactically Diverse Adversarial Network for Knowledge-Grounded Conversation Generation." Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.
questions: None
limitations: limitations are sufficiently considered in Supplementary materials.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The authors introduce a method, SURGE (subgraph retrieval augmented generation) for generating consistent dialogue responses, augmented with knowledge from knowledge graphs. SURGE consists of several elements. The first is a context-relevant knowledge retriever, which simply retrieves knowledge from a graph given a dialogue context. The second is an invariant graph encoding; given a set of entity-relation tuples from a subgraph, the authors propose a method for jointly encoding the context and knowledge such that the encoding is invariant to both the order in which facts are presented, and the order in which entities are presented (i.e., relation-invariant). Finally, to encourage consistent generation, the authors propose a contrastive learning objective, which encourages model graph encodings (context, knowledge) to be similar to their decoded output representations, and dissimilar to decoded output representations given other sets of knowledge; as the authors say, “given a different graph, the model would generate different tokens fro the same graph” (L235).
- The authors compare their method against others on the OpenDialKG dataset, measuring standard generation metrics as well as a new proposed Knowledge QA metric, in which the authors examine the performance of a QA model when given a question and model response as input. The authors find that their method outperforms baselines handily, in all metrics considered. Human evaluations and several ablation studies support the authors’ claims and methods.

strengths_and_weaknesses: ### Originality
- **Strengths**: The exact methods themselves may not be new (retrieval-augmented semi-supervised learning, contrastive learning, etc.), however the authors indeed provide a novel combination of techniques. The invariant graph encoding is novel as well, and it is clear how the work differs from previous contributions.

### Quality 
- **Strengths**: The authors’ claims are generally well supported and technically sound; the authors present a complete work with several contributions that improve significantly compared to baseline performance. Human evaluations emphasize the gap between the authors’ proposed work and previous methods. The choices of using retrieval-training and invariance encodings are well supported with ablation studies.
- **Weaknesses**: While the three proposed main models all outperform baselines (SURGE with unsupervised, semi-supervised, and contrastive learning losses), it is not clear which aspect of the model truly shines as the best contribution. The gap between SURGE and using sparse retrieval seems to imply that graph retrieval is driving nearly all of the performance gains; this intuitively makes sense, given that OpenDialKG was built using the Freebase KG. Beyond that, however, different methods yield different results; i.e., in some cases, semi-supervised outperforms contrastive, and vice-versa. The performance of naive encoding of (context, subgraph) is also not too far behind the invariance proposals. 

### Clarity
- **Strengths**: The paper is well written and clearly outlines the proposed methods and the rationale for using them. Results are generally well presented.
- **Weaknesses**: Figures and tables are very small and compact. While this is not as big of an issue when presenting numbers of graphs, this sees to really hurt other visualizations; Figure 2 (the framework overview) is a bit tough to follow, as are the results in Figure 7.

### Significance
- **Strengths**: This paper provides several methods that other researchers could use in their work, e.g., the invariance encodings (which is helpful in situations with lots of knowledge conditioning), or the various ways in which to train the retriever (e.g. via contrastive learning with the decoder representations). The paper also highlights how a good pipeline can improve performance significantly on a dialogue task requiring knowledge from a knowledge graph.
- **Weaknesses**: The contributions may be limited to instances where a knowledge graph is the most appropriate knowledge source; though the authors claim that using such a knowledge source is more efficient than using unstructured documents, it still limits the types and amount of knowledge at one’s disposal, and it is not immediately clear how to apply this method to situations in which unstructured knowledge is preferred.

questions: - Several times, the authors mention that KG models are more efficient or preferred more than those with unstructured knowledge (e.g., L36-37, L100-103); could you please explain why that is the case?
- The authors mention that the Enc and Dec functions from a transformer are “permutation sensitive” (L135); is it not the case that Enc could in theory be permutation insensitive, due to bidirectional attention?
- Do you have some clarity as to which specific situations the contrastive learning is better than using just semi-supervised, given that, in Table 1, both KQA and BLEU performance is better for the latter?

limitations: The authors adequately address several limitations within the supplementary material.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposed a KG-based dialogue generation approach. 

The proposed approach consists of a triplet selection, a subgraph construction and retrieval, a graph encoding and a basic encoder-decoder framework. 

Contrastive learning is used for ensuring graph-text consistency in generation. 

A KQA metric is proposed for the evaluation of generated dialogue in fact-level.

Experiments in a benchmark dataset verified the effectiveness of the proposed approach. 
strengths_and_weaknesses: Strengths:

1. A novel subgraph construction and retrieval scheme. 

2. The proposed invariant graph encoding considered the order and inverse relation of a triplet. 

3. The use of a loose evaluation metric for dialogue generation.

Weaknesses:

1. The lack of SOTA methods for empirical comparisons on triplets selection. 

2. The lack of verification of different graph-based neural networks for encoding. 

3. I am afraid p(Z|x) defined in this paper is not a probability. 

4. The lack of empirical comparisons with other triplets ranking methods or using other methods for the performance verification of dialogue generation. 

5. A minor point that only one dataset is used in this paper. I concerned the extension of the proposed approach.

6. Another minor point is that there are a lot of grammar errors, a careful proofread is needed. 
questions: Please see the weaknesses. 
limitations: The lack of comparisons with SOTA methods limited the novelty of the proposed approach. 

The only one experimental dataset limited the extension of the proposed approach and the common conclusions presented in the paper.
review_title: null

Meta-review: metareview: The paper presents a method for dialogue generation with knowledge graphs, where the goal is to increase faithfulness towards the provided knowledge graph. The research topic, while somewhat narrow, is well-motivated, and they propose a context-relevant subgraph retrieval method (with specialized graph encoding method preserving the permutation/relation inversion invariant objective, new loss function for the generation that encourages consistency with the knowledge subgraph), which shows a promising performance on OpendialKG benchmark dataset. 

While I do some values and contributions in this paper, I am not convinced this merits a publication at NeurIPS — each component of the pipeline (contrastive learning, etc) is not novel and it addresses a fairly small domain. If they show this can be applied to more diverse settings, for example, different datasets (I saw that results on KOMODIS are presented in supplementary material, but they are not explained carefully and not very convincing as is. I’d recommend integrating it into the main paper). I also suggest novel metric (section 4) should be more carefully verified before being used to evaluate systems — e.g., how they align with human evaluation, etc. 

It is unfortunate that two reviewers did not respond to the author's responses, the area chair examined the paper independently (and also read through the review/responses) to write this meta-review. 
confidence: Certain
Decision: Reject