Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a graph network (called HGN), aiming to better leverage commonsense knowledge graphs (KGs) to solve commonsense question answering and reasoning tasks, by jointly generating representations for new triples from KGs, determining relevance of the triples, and learning graph model parameters. The proposed model is tested on several tasks: CommonsenseQA, OpenbookQA, and CODAH.

Pros:
-  Overall, the paper is easy to follow, although there are a number of typos or grammatical errors that need to be fixed.  The overall idea is clear.
-  Jointly learning (pruning) the graph structure with the network parameters is interesting.
-  The proposed model outperforms the baselines in comparison.
-  Human evaluation is provided.

Cons:
-  My major concern about this paper is the novelty and contributions in terms of methodology. Compared to existing methods (e.g., those PG models proposed (Wang et al. 2020)), the novelty of the current submission is rather limited---the proposed model of jointly generating new triples and learning (pruning) the graph structure with the network parameters is an interesting, but a pretty incremental idea. 

-  The empirical comparison to previous work (e.g., Wang et al. 2020) needs to be clearer to help understand the empirical advantages of the proposed models. The paper mentioned some reason of excluding PG-Full from comparison, but since PG-Global does not include static knowledge embedding and PG-full does, is the latter a more reasonable baseline to be compared with? The model does not achieve better performance than existing models on some tasks, which casts doubts on its effectiveness; e.g., whether its advantage is orthogonal to that brought by stronger models such as those performing much better on the OpenbookQA task.

More comments:
-  The paper uses much space to discuss neural symbolic approaches. Given the vague benefit of doing so, it may be better to use the limited space to focus more on establishing the contributions w.r.t. existing models; e.g., more details about (Wang et al., 2020) can be provided and compared to in both methodological and experimental analyses.
- The human evaluation was performed on the questions with correct questions. More analyses on the edges and weights generated for questions that were not correctly answers may help better understand the proposed model. 

review_title: Novelty of the proposed model is limited

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a question answering model that is augmented with a common-sense knowledge graph (KG). The paper builds on the following two observations — (a) KGs are incomplete often lacking facts that would be needed for reasoning to answer a question. (b) Current methods over-retrieves facts (edges) from the KG leading to a lot of unrelated facts that potentially makes reasoning noisier and harder.

The paper first retrieves all possible facts from the KG connecting entities in the question and answer. However, due to the incompleteness of the KG, the retrieved subgraph might be missing important edges between entities. To deal with this, they connect all nodes between question and answer entities and initialize the embedding of the newly added edge with hidden layers of a sentence generated from a fine-tuned GPT2 language model (this important detail was mentioned in the appendix). However, currently the graph is over complete and very noisy. The proposed model then sparsifies the graph by learning edge weights via a two-step message passing process. The edge weight is learned as a function of the current edge representation and the textual representation. Lastly, an entropy term is added to the objective function to encourage more peakiness (and sparsity). 

The model is tested on three common-sense QA benchmarks and on three of them they beat the baselines albeit only around 1-2%. Statistical significance of the result was not reported. Ablation study show that the efficacy of including the generated edges and pruning the graph. There was a small human-study also done where the annotators were shown a binarized graph and were asked to rate each edge. Annotators had moderate agreement between themselves in finding that the pruned graph was better than the original retrieved graph.


Strengths:
* Developing models that can use symbolic external knowledge present in common-sense KGs and also overcome the sparsity in KG is important and this model is a step in that direction
* The paper achieves a little improvement in performance in all three datasets and ablation experiments are helpful in understanding the results
* The paper is clearly written and it was easy to follow for the most part

Weaknesses & clarifying questions for the authors:
* My biggest complain of the paper in its current form is that several modeling choices were not motivated at all. For example, generating edges between nodes using GPT-2 language model is fairly non-standard. However, the paper lacks any motivation on why this is the right approach to generate facts which are not captured in a KG. What is the guarantee that GPT-2 will not hallucinate and generate a false fact and thereby adding unnecessary noise in the reasoning process.
* Following up on the previous point, there could have been several other modeling choices. For example, instead of generating text via a language model, one could gather text (sentences) from Wikipedia or other text corpora containing the entities (which would mean the text would probably not be a false fact). These modeling choices were not explored and were not discussed. 
* The GPT-2 modeling choice was also moved to the appendix and I think it should definitely be moved to the main section of the paper as it is one of the core technical contribution of the paper.
* Another modeling decision that was not motivated was the graph reasoning part. It is unclear to me why the edge weight is modeled as a part of the message passing process. Another (simpler) alternative could be modeling it as an edge attention, which is computed wrt the text and the current node embeddings. I would be curious to know how this simple model worked and if it didn’t why was the case.
* Even though there are improvements across dataset, the improvements are relatively minor (<1% in few datasets). I think it would be useful to have statistical significance test.
* Regarding the human study, if I understand correctly, was only the node and adjacent matrix shown to the annotators?. Was the relation type (KB relations and generated sentences) included too? If they were not included I think they should be because knowing the relations is also very improvement.
* Can you elaborate on the average helpfulness score of edges in table 5? How many (what proportions) were scored 0, 1 or 2 for both the graphs? I think it would also be helpful to report how many facts all/majority of the annotators found to be helpful for both the graphs. 

Missing Reference: It would be nice to cite Sun et al EMNLP 2019 -- PullNet: Open Domain Question Answering with
Iterative Retrieval on Knowledge Bases and Text since one of the core contributions of that paper was to retrieve and keep only relevant facts from the KG. Relation paths in KG were explored by several works before Wang et al 2020 such as Neelakantan et al ACL 2015 - Compositional Vector Space Models for Knowledge Base Completion, Das et al EACL 2017 -- Chains of reasoning over entities, relations and text etc. It would be nice to cite those work as well.

Recommendation:  In light of the current weaknesses of the paper, I am giving it a score of 5 and I look forward to the discussion.

=======11/22======

I am deciding to keep the same scores as before. Some of the initial concerns remain. I think the paper still lacks motivation wrt the GPT2 model generating missing edges. Thank you for getting the latest results, the paper is stronger than before and with some more work, I am confident it will be a good contribution to the research community.

=====11/24======

After having read through the explanation behind using GPT2 as edge features (and sufficient backing by 2 closely related work), I am increasing my score to 6. I think the discussion helped in somewhat convincing me that this approach would work for ConceptNet because of its limited schema.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: === Summary ===

In this paper, the authors propose a new approach towards incorporating knowledge graphs (KG) into commonsense QA frameworks. KGs are helpful for adding structured "world" information, which neural-symbolic architectures can leverage to do commonsense reasoning, e.g., "what is the expensive resource in printing on paper?" (paper). In such architectures, however, the authors argue that KG quality is a large impediment (e.g., missing or incorrect edges, distracting nodes, etc). Therefore, they propose a "hybrid" KG-based model (accordingly named "Hybrid Graph Network") that jointly learns to refine/augment the graph structure while also optimizing it for inference performance.

Experiments are conducted on a number of commonsense reasoning tasks with multiple KG sources, and compared to relevant baselines. They also perform a user study to examine the "helpfulness" of the refined KGs produced by the HGN.

=== Justification for Score ===

This paper is well-written and well-evaluated. The proposed method is also relatively simple and intuitively motivated. The experiments, however, only show modest (yet still positive) empirical gains. Perhaps not a game-changer for commonsense QA, but still a reasonable contribution that I would recommend for acceptance.

=== Strengths ===

+ The paper is clear and well-written. 
+ The experimental section is strong. The model is compared to strong baselines, and I appreciated the extra user-study on learned graph structure.
+ The method is well-motivated, and provides (modest) empirical gains compared to some baselines.
+ The method shows good performance with respect to increasing data efficiency (Fig. 4).

=== Concerns ===

- The main concern is on the empirical effectiveness of the model. The results appear to give only modest gains at best (against comparable baselines to the best of my knowledge). For a number of the results the variance is large compared to the relative difference---it would helpful to also include tests of significance for these improvements.

- On OpenbookQA the model significantly underperforms T5-based models. Though I appreciate T5 is unwieldy due to its large size, it makes me question if this method indeed presents a complimentary gain, or is climbing the wrong architectural hill.

=== Update After Rebuttal ===

I commend the authors on a through rebuttal and active rewrites/experimentation. I still think the work is good, and can warrant acceptance. However, I still find the empirical results to be only moderate at best (though I appreciated the authors' rebuttal and significance testing). I am keeping my score the same.
review_title: Well evaluated and effective KG-based commonsense QA framework

Meta-review: The paper proposes an interesting step in the direction of neuro-symbolic reasoning. While there is no consensus among reviewers about the key novelty of the method, all acknowledge the interest of the direction. All of them also recognize that the submission improved greatly during the discussion phase: clarification of motivations, of experimental settings and results, of discussion with previous work.

However, despite those improvements, the submission is not yet ready for publication at ICLR. We encourage the authors to use the very detailed reviews and comments to improve the work. In particular, we encourage them to pay attention at three aspects:

1/ Comparison with large language models: the discussion wrt T5 is important. A key motivation for the proposed model is that it is bringing information and elements for QA (or other reasoning tasks) that purely scaling up language models can not bring. Or maybe they can bring the same kind of improvement but at a much lower computational cost. In any case, this is a very important point to justify the interest of such approach, and neuro-symbolic reasoning overall, empirically.

2/ Using GPT2 (or equivalent): the discussion on using GPT-2 for generating new facts is key too. It is essential to bring this description from appendix to the core of the paper. But more discussion are expected.  For instance, what if GPT-2 generates facts that are false and lead to answering and justifying a wrong answer? In other words, how does it impact the integrity of the contextualized KG? This is an essential point that needs to be worked on more thoroughly. 

3/ Overall there have been a lot of discussion to improve the motivations and the contributions. But they are not reflected in the paper necessarily. Following R2, we encourage the authors to "refocus the existing version (e.g., from vague discussion about neural-symbolic models towards establishing solid comparison to the most related previous work in various sections of the submission)"




Decision: Reject