Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a novel method of fusion of information from two modalities: text (context and question) and a Knowledge Base, for the task of question answering. Concretely, they propose GreaseLM layers that combine information from a Language Model (LM) and a Graph Neural Network (GNN) by mixing representations of embeddings from two special tokens (one in the LM and one in the GNN). The embedding in the LM passes information to the text modality via self attention; while the embedding in the GNN passes information to the KB modality via connections to other nodes. The modality interaction layer (dubbed MInt) is a simple MLP for combining the embeddings.
The paper demonstrates superior performance compared to baseline methods on 3 different datasets. Additional ablations quantitatively show that the model performs better on questions with negations, hedging and constraints (modelled with a proxy of number of prepositional phrases). 
main_review: Strengths of the paper: 

1. The paper presents a novel way of combining information from text and a KB in a bidirectional way. 
2. The results presented in the paper show strong gains against baseline methods on 3 different datasets.
3. Ablation studies show that the model achieves good performance on more complex questions.

Weaknesses, suggested improvements and requested clarifications

1. Regarding preposition phrases as a proxy for complexity: since the hypothesis is that the more the number of prepositional phrases in a question, the harder it is to answer. But from Table 5., the trend of performance seems to increase with the increased prepositional phrases (with 84.7 being the max for 4 PPs). It would be good if the authors could provide some discussion around this observation. Additionally, it would be good if the authors could report the number of examples (either raw number or as a fraction of the total dev set) for each of the categories: having that would help draw better conclusions. 

2. The paper mentions that the entity extraction was done following Yasunaga et al. Was relevance scoring was done for the entities ?

3. Alongside with qualitative analysis, some quantitative analysis would be good to show what the model learns. Specifically, having the BFS analysis of the attention weights as a function of different GreaseLM layers (as done by Yasunaga et al.) can help demonstrate how having the bidirectional context information flow helps improve reasoning. Additionally, showing this for negations and / or examples which GreaseLM gets correct but QA-GNN does not (and vice-versa) can shed some light on what the model improves on (and what are the limitations). 

4. Having an ablation on the number of GreaseLM layers would also be quite useful to answer if performance improves with more GreaseLM layers, are there diminishing returns or do we need just a few GreaseLM layers, beyond which it is detrimental to the model's performance.

5. The Graph connectivity ablation states that connecting the e_int node to all entities (instead of just the input text entities) hurts performance. It would be good if the authors could provide some intuition / insight as to why that might be the case. 

Presentation Suggestions:

1. Page 5, line 1: should \tilde{e}^{(l-1)} be \tilde{e}^{l} instead ? 

2. Page 5, Equations (6, 7, 8): should e^{l}_{s} and e^{l}_{j} be e^{(l-1)}_{s} and e^{(l-1)}_{j} respectively ?

Note to the authors:

There is a contemporaneous work submitted to ICLR 2022 [GNN is a Counter? Revisiting GNN for Question Answering](https://openreview.net/forum?id=hzmQ4wOnSb), whose hypothesis seems to be that by only using embeddings for node types and relation types, the models are able to attain good performance (86.67 acc on OpenBookQA) without needing any cross-modal information. While I understand this is contemporaneous work, but since the work is so relevant to this paper and seems to directly contradict the premise of this paper, it might be good to have a short discussion on this (just a suggestion).

summary_of_the_review: The paper presents a novel way of combining information from text and knowledge base modalities. The results on 3 datasets show the models improved performance, and ablations show quantitatively that the model performs better on complex questions. While I believe the paper can benefit from some qualitative analysis, and there are some open questions that need clarification, I believe the contributions presented are enough to merit an acceptance.

Edit: I have updated my score after reading the author response and edits made to the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a KG augmented LM which fuses graph and contextual representations. The model, when evaluated on multi-choice QA, outperforms strong baselines.

The context, question, answer pairs are encoded with a standard LM and the nodes of a relevant subgraph which is retrieved from a KG are encoded with a GNN. The representations of (i) a special interaction token (same as CLS), and (ii) a special interaction node, are input to custom layers that mix/fuse them which potentially makes them more informative to the downstream MLP head.

The authors experiment on three MCQA datasets: commonsenseQA, openbookQA, and medQA-USMLE (from the biomedical domain). ConceptNet is used as the KG for the first two datasets and a self-constructed KG is used for the third. The evaluation and analysis show that this method outperforms strong baselines (sometimes with a much higher parameter count). especially when the questions are complex and have prepositional phrases, negation terms, etc.
main_review: Overall I like this paper since the method is simple and the results are good. I have only one problem: the initialization of LMs. The authors say that they initialize different LMs for different datasets to show that the method is agnostic to them. I am not convinced by this argument. A better way to do this would be to experiment with different LMs on one dataset to show that the method is agnostic to it and then use the best one for the other dataset(s) (Note: I understand that this would not apply to the biomedical dataset).

Other than this, I enjoyed reading the paper and liked the analysis and the detailed ablations. I am willing to increase my rating if the above concern is resolved.

EDIT: changed my rating after the author response.
summary_of_the_review: A good paper with a small and easily fixable issue regarding LM initialization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce a new model for the QA tasks. They improve the existing approaches by adding the interaction components (the interaction token for the LM and interaction node for the GNN) to fuse knowledge from LM and knowledge graph in a more interactive manner. The LM can be any Transformer encoder-based pretrained LMs, where they tested RoBERTa-Large, AristoRoBERTa, and SapBERT in the experiments. The GNN is a simplification of the method of [1]. Experiments have been conducted to compare the proposed model and the prior KG+LM and LM-only baselines. Their model proved to be effective on three benchmarks, CommonsenseQA, OpenbookQA and MedQA-USMLE.

main_review: Strength:
- A novel architecture to enable deeper interaction between LM and GCN.
- Clear explanation of the proposed model.
- Extensive ablation studies to demonstrate the importance of modality interaction layer 
- selection, parameter sharing, graph connectivity, and parameter initialization.
-  Consistent performance improvement over the baseline models on all of the three QA benchmarks.

Weakness:
- paper title is misleading, not directly related to LM
- no citation and description for the baseline method T5+KB (Table 4)
- As shown in Table 7, the proposed method is very sensitive to many factors.  The performance drops significantly when we change any of them.
- The data preprocessing and training steps are complex.


Questions:
- According to the parameters presented in Table 8, the knowledge from LM and GNN are only fused at the last 5 layers (parameter M) when 24-layer LMs are used, and at the last 3 layers when 12-layer LMs are used. It may be useful to show how the performance changes when using different M.
- According to [2], the CommonsenseQA IH-dev set contains 1,221 questions in total. May I know many questions are in each data split shown in Table 5?
- Is there any case that no entities are found during KG retrieval (Sec. 3.1)? If there are any, how do you handle these cases? What’s the percentage of such cases.

Reference:
[1] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA- GNN: Reasoning with language models and knowledge graphs for question answering. ArXiv, abs/2104.06378, 2021.
[2] Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Empirical Methods in Natural Language Processing (EMNLP), 2019.

summary_of_the_review: In summary, the paper has some nice contributions, such as a novel architecture to enable deeper interaction between LM and GCN, clear writing, and extensive analysis. Although there remain some issues and questions, the paper should be able to bring to the community some new aspects.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model which fuses representations from language models and graph neural networks through multiple interactions layers for multi-choice question answering tasks, which outperforms baseline methods over three datasets from two different domains.
main_review: Strengths:
1. The idea of multi-layer fusion of two modality through additional interaction token and node is interesting.
2. The model achieves good results on three multi-choice question answering datasets.
3. The ablation study shows some interesting finds such as the importance weight sharing in Mint layers.

Weakness:
1. The technical contribution is somewhat small: except the multi-layer fusion part, others are very similar to QA-GNN including the GNN model. 
2. As the main contribution is about the modality interaction layer, more analysis should be made such as the effects of number of fusion layers $M$, why only perform interaction through one single embedding (interaction token and node) instead of directly applying cross attention through all the tokens and nodes? There are also some other ways to perform embedding interaction such as the one in ERNIE [1] Figure 2 (b). To answer these questions, both intuitive analysis and empirical results are needed.

Additional Reviews:
1. Since this paper only focuses on question answering tasks, I suggest adding "for question answering" into the title similar to previous works QA-GNN and MHGRN to be more accurate.
2. There are also some related but missing references such as GLM[1], JAKET[2], CoLAKE[3]. Although they focus on language model pre-training, the idea of combining graph reasoning and language modeling is related.

[1] Zhang Z, Han X, Liu Z, et al. ERNIE: Enhanced language representation with informative entities[J]. arXiv preprint arXiv:1905.07129, 2019.
[2] Shen T, Mao Y, He P, et al. Exploiting structured knowledge in text via graph-guided representation learning[J]. arXiv preprint arXiv:2004.14224, 2020.
[3] Yu D, Zhu C, Yang Y, et al. Jaket: Joint pre-training of knowledge graph and language understanding[J]. arXiv preprint arXiv:2010.00796, 2020.
[4] Sun T, Shao Y, Qiu X, et al. Colake: Contextualized language and knowledge embedding[J]. arXiv preprint arXiv:2010.00309, 2020.
summary_of_the_review: In summary, this paper proposes an interesting idea to combine language model and graph neural networks for question answering and shows good results. But more experiments and analysis should be conducted to justify their design on modality interaction modeling. Adding them will make it a strong paper.
review_title: null

Meta-review: The paper presents a novel method of fusion of information from two modalities: text (context and question) and a Knowledge Base, for the task of question answering. The proposed method looks quite simple and clear, while the results show strong gains against baseline methods on 3 different datasets. Ablation studies show that the model achieves good performance on more complex questions. While the reviewers raise some concerns, e.g., on the sensitivity of the proposed method, the technical novelty against prior works, they see values in this paper in general. And the authors did a good job in their rebuttal. After several rounds of interactions, some reviewers were convinced to raise their scores by a little bit. As a result, we think the paper is in a good shape and ICLR audience should be interested in it.
Decision: Accept (Spotlight)