Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents several experiments that make use of combined knowledge from language models and unstructured knowledge bases. The paper is mostly well-written, though some choices in presentation made it difficult to follow (leading me to go back and forth). I will provide some suggestions to improve this below. The paper also contains some claims that, in my opinion, are not supported by the evidence. Since the paper does not need these claims in order to provide a nice contribution, I suggest they be adapted.

The claims of which I am not convinced based on this research are the following:

- the claim that the results show the models capability of reasoning with implicit knowledge. 
=> what it means for a model to do this is a very complicated question and requires much more reflection on what implicit knowledge is and what reasoning with it means (and how that can be shown). I do not believe that solving something despite lack of lexical overlap provides evidence for this, since language models are known to capture similarity in meaning.
- "The significant improvement over BERT (3-14%) shows the ability for these methods to utilize implicit knowledge, which is especially useful for low-resource languages, target domains where we can pre-train using fewer data and use ad-hoc knowledge to solve a target task and have smaller vocab and params."
=> BERT is still huge and the quality of similar models for real low-resource languages will be much lower. For such a claim, I would expect experiments that play with very small language models and limited training data (because this also tends to be the case for low-resource languages). What is more relevant from these results is that indeed the results with BERT come closer to those of RoBERTa and that this is therefore potentially interesting for reducing training costs. It also leads to the future research question of what size vs knowledge is doing (this current work does not dive into this question, but it seems the most logical thing to examine: what mistakes does each method still make and are they similar). 

As mentioned earlier, I think the paper provides sufficient contributions without these claims and I therefore suggest they be revised.

Suggestions for improvement on the presentation.

About the structure:

- the choice of spreading the general approach over various sections (Section 2 about the sets, Section 3 about the knowledge sources and their preparation and Section 4 with the rest of the method made the overall setup more difficult to follow. The first sentence in Section 4 illustrates this (it is starting with `After' which shows there is a strong connection with the previous sections). I suggest merging the three sections and use the saved space to take the reader a bit more by the hand (replacing the headers by short motivations). 
- I almost missed that both validation and a separate test set were present since only the size of the validation set was originally reported and had already made a note about using separate validation for tuning: this could be prevented by introducing all three components of the data right away and clearly stating how each set is used.
- I am still a bit confused about what has actually been tested to support the hypotheses and this is largely due to the use of the term `baseline' for what, if I understand correctly turn out to be the actual experiments. I recommend removing the full table for tuning and use the additional space to explain better what the reported results are.
- The paper states "As a result, the knowledge does not overlap with the gold answer, making the task trivial." This statement (in particular the choice of the word ``trivial'') can be a bit confusing, since trivial is also often used to state something is extremely obvious. I assume the problem is in fact the opposite; that using knowledge correctly would lead to the wrong outcome and that therefore the outcome on these instances do not contribute to insights on the methods?
review_title: Interesting experiments, but containing some overly strong claims

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper asks how to improve commonsense question answering by infusing or providing potentially useful knowledge to BERT-style models. Authors explore using finetuning on and/or retrieving of useful or related knowledge for questions, and experiment with multiple ways to combine the useful knowledge with the questions. Results show that simultaneously finetuning on related knowledge source and retrieving a few examples for each question improves performance the best. 

The paper is well-executed, provides convincing experiments and analyses, and yields interesting results about the usefulness of external-ish knowledge.

Some questions or comments:
- According to the paper, the finetuning is done on both the MLM and the NSP objectives; however, the RoBERTa paper showed that the NSP objective isn't that useful, and wasn't used to train RoBERTa. I believe the results will not change that much, but it would probably be better if authors re-ran their experiments only using the MLM objective.
- The method section is a little hard to follow; I wish the authors had included a high-level modelling diagram that illustrates how each of the {concat, max, ...} strategies work exactly, and how they are combined to feed into BERT/RoBERTa.
- [Minor] the introduction mentions the WSC example and some retrieved passage that is supposedly helpful to answer the WSC example. I didn't fully understand why it was the same kind of reasoning, so it would be useful to point out directly why the example is useful.
review_title: An interesting exploration of how retrieved knowledge can be useful for commonsense reasoning QA tasks

final_score: rating: 7: Good paper, accept
review_content: The paper presents a detailed study on the commonsense reasoning ability of pre-trained Transformer-based LMs such as BERT and RoBERTa. The authors build an experimental framework for evaluating different models with different knowledge sources under different infusion strategies, e.g., continually pre-training (revision) or retrieval-based learning (openbook), or both. The authors present a few methods that can improve the results and reduce the gap between smaller LMs and larger ones. 

Questions and suggestions

- I suggest the authors consider adding the CommonsenseQA task [1] and an additional dataset and use the OMCS corpus (the text-version of ConceptNet) or GenericsKB [2] as the knowledge resource. Also, the OpenCSR [3] seems to share the similar goal of the openbook setup for answering commonsense questions.  

- The authors claim in the conclusion that the proposed methods "surpass complex graph-based methods"? I don't find the supporting facts from the experiments. 

- I would suggest the authors do not claim the methods are part of the novelty of the paper as they are very trivial. I would rephrase the Introduction and Section 4.1 so that the key contribution is the analysis and the evaluation protocol. The proposed methods here are just some default options as baselines in the framework. 

- The experiment section does not show any qualitative analysis or case studies. It is unclear how well the retrieved sentences are and how they can improve the LMs for reasoning.

- The retrieval module can be replaced with pre-trained DPR [4], which is generally better than BM25.

[1] CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. 
Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant. NAACL-19

[2] https://allenai.org/data/genericskb

[3] Differentiable Open-Ended Commonsense Reasoning. 
Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William W. Cohen. NAACL-21

[4] Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020


review_title: A solid work with detailed empirical analysis.

Meta-review: This paper proposes new ways to combine pretrained models with external knowledge for commonsense reasoning tasks. All the reviewers thought the paper was well-executed and interesting. One reviewer raised some concerns about overclaiming and paper organization, which the authors have promised to address.
Decision: Accept