Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors propose DRAGON - a self-supervised method to pretrain a language-knowledge model from raw text and KQ at scale. A training instance is formulated from a text segment and the related sub-knowledge graph extracted based on the entities appearing in the text segment. The model bidirectionally fuses information from both modalities. The pre-training step is done on BookCorpus and ConceptNet and fine-tuning is performance on specific test-data. Different from GreaseLM where there is only the finetuning step, the pretraining step of DRAGON helps the finetuned model and shows noticeable gain in downstream QA datasets.
strengths_and_weaknesses: Originality: Authors present an effective approach to learn deep fusion of text data and knowledge graph at scale, applied for pre-training language-knowledge model. The pretrained model has promising performance when being finetuned on downstream QA tasks. 

Quality: The proposed idea is technically sound, and the experiment results are convincing. Authors also discuss different design choices of DRAGON, which provides insights and explain why the model works.

Clarity: The paper is well written and easy to follow. I like that authors use the main example in Fig 1 to explain the key idea of the designed components (MLM and LinkPred losses). Another comment is that it will be useful to show an example of text segment and its related sub-graph (can be in the Appendix if there is no space).

Significance: Given the novelty and effectiveness of the proposed idea, the contribution of this work is significant.

questions: 1. In the KG retrieval for each text segment, it only considers matching with the entities but not the relationships between them. Can you discuss the impact if the edge's relationship doesn't match with the one described in a text segment? 

2. Did authors try to evaluate the performance of model on knowledge graph embedding tasks?
limitations: Yes, authors have discussed about the ethics, limitations and risks in Appendix A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: **Summary**

This work proposed a deep bidirectional language-knowledge pre-training, by leveraging both text and knowledge graphs. The proposed model has two core components: a cross-modal model that fuses text and KG bidirectionally, and a bidirectional self-supervised objective that learns joint reasoning over text and KG. The pre-training process contains two self-supervised reasoning tasks: masked language modeling (MLM), which masks and predicts tokens in the input text; and link prediction, which drops and predicts edges in the input KG. Experiments on two domains demonstrated the effectiveness of the proposed model. 
strengths_and_weaknesses: **Strengths** 

(1) The idea of jointly pre-training text and knowledge graph representations is well motivated. Most of the recent works focused on augmenting the model by fine-tuning it with a knowledge graph. Still, they were not pre-trained to learn deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and knowledge graphs. 

(2) The proposed method is able to take pairs of text and relevant knowledge graphs as input and bidirectionally fuses information from both. The experiments were conducted on two domains, both of which demonstrate superior performance to baseline methods. 

**Weaknesses**

(1) The novelty of the proposed method is limited. First, jointly pre-training text and knowledge graph are not new, which has been studied in existing works [1, 2]. Second, the multi-modal MLM objective is widely used in vision-language pre-training works; the link prediction objective was first proposed by JAKET [2]. Third, the deep fusion between text and knowledge graph was proposed in GreaseLM [3].

(2) The experimental results of CSQA and OBQA were not consistent with existing works. Recent models have achieved 80+ accuracy on CSQA and OBQA, e.g., QA-GNN 76.1 on CSQA leaderboard, GreaseLM 84.8 on OBQA leaderboard. However, the performance in the paper is lower than the baseline methods by a large margin. Did the authors submit the results to the leaderboard?

(3) The title of this paper is confusing, especially the phrase “language-knowledge”. Why does language not include knowledge? 

[1] JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs. ACL 2021 (This is a missing reference.)

[2] Jaket: Joint pre-training of knowledge graph and language understanding. AAAI 2022

[3] Greaselm: Graph reasoning enhanced language models for question answering. ICLR 2022


questions: See weakness above.
limitations: It seems the proposed method can only work on multi-choice QA problems. As shown in Figure 2, the proposed method needs to use answer choices to find relevant entity nodes on the knowledge graph. So, I am not sure how this method can be applied to other tasks, such as binary question answering tasks (e.g., CSQA2.0), and language generation tasks.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method for jointly pre-training a BERT-style LM with a knowledge-graph component.
The paper builds on an existing architecture called GreaseLM, which can fuse information across a piece of text and a knowledge-(sub)graph using cross-attention. The contribution is that they extent Grease-LM by using pretraining, and demonstrate stronger results on a set of downstream tasks. 

This is achieved by taking a chunk of text to be used in masked-language modelling, and running entity linking, and constructing a small knowledge graph from ConceptNet (or UMLS in the case of biomedical text).
The model is jointly to trained on masked-language modelling for the text, and link prediction for the KB subgraph.

The authors go on to finetune and test their model on a suite of multiple-choice commonsense-like QA tasks, and observe consistent improvements over a non-KG-augmented Baseline, and GREASE-LM (i.e. the non-pretrained variant of their model). They also show similar results on Biomedical tasks.

They finally present detailed ablations, demonstrate wider performance gaps for smaller training sets, and present some analysis. 
strengths_and_weaknesses: Strengths:
* Written clearly and easy to read. 
* The authors experiment in two domains, (general English and Biomedical texts) and show consistent gains, strengthening their claims
* The authors demonstrate a good practice to treat the baseline Roberta-only model fairly by continuing to pretrain it for as long as the other model. 
* the method works well compared to their baselines
* the ablations and analysis section are detailed and generous  

Weaknesses/areas for improvement:
* There is a certain lack of novelty here, as the authors acknowledge, but the have done a good job of really trying to understand their design space well, and they ablations are thorough.
* The method requires an entity linker to work, which may not always be available or effective in the domain of interest. 
* The construction of the pretraining data is probably one of the bigger factors in the success or failure of this model, I would like to see more details here in the main paper, rather than this be demoted to the appendix.
* less of a weakness, more of a suggestion: I think this would be significantly more impactful as a model/piece of work, if it was generative, and could generate text.
* If I've understood correctly, the method is static with respect to the knowledge graph (i.e. if the KG gets changed after pretraining time, the model will not have a good way to embed novel entities or relations) I think this is a missed opportunity, e.g. the nodes and relations are usually lexicalized, so the text-understanding part of the model could be used to create the node embeddings, so that the model could consume new entities and relations after pretraining.
* The quoted numbers for CSQA for the baseline look to be weak c.f. literature - a quick look at the leaderboard suggest Roberta achieves 72.1, whereas the authors quote 68.7, significantly lower. (https://www.tau-nlp.sites.tau.ac.il/csqa-leaderboard2) - Dragon is still better, but it makes me worry a little about the baseline numbers. 
* Figure 2 is nice but really small, and pushes what is acceptable in my opinion.  
* The gains in Table 4 gains seem very small - these should be accompanied by confidences/statistical significance tests.
* My main complaint is that the downstream experiments are quite narrow, limiting impact. Could this be used well on other kinds of NLP tasks? Further, The focus is entirely on what the model can do to improve NLP tasks using KGs, but I think it would be interesting to explore whether this model can be used to improve performance on KG tasks by using retrieved text - this would increase the scope and impact of this paper.
* commonsenseQA is constructed using data from ConceptNET, so the results you achieve here may be unfair compared to a text-only model. (I don't know if this is the case for the other datasets, but please address it in writing)

originality: Some lack of novelty, as mentioned above, but the authors have done enough to convince me this is a novel-enough contribution (combining existing things in new and interesting ways is a contribution) 
quality: the paper is of good quality in my view - I appreciate the depth of ablation and analysis 
clarity: very clear and easy to follow
significance: I have included above some areas which would improve significance and impact. 
questions: What happens when you use more (or fewer) nodes in the retrieved KG at fine-tune/test time? If I want a bigger neighbourhood (and slower inference), will my accuracy improve? 

CommonsenseQA is constructed using data from ConceptNET, so the results you achieve here may be unfair compared to a text-only model. (I don't know if this is the case for the other datasets, but please address it in writing)

limitations: No specific concerns from me here. The may be a little bit of over-claiming in terms of the strength of the results.
review_title: null

Meta-review: metareview: This paper describes a pretraining approach that can leverage both text and knowledge graphs. The model has a cross-modal model that fuses text and KG bidirectionally, and a bidirectional self-supervised objective that learns joint reasoning over text and KG. For pretraiing the models uses traditional masked language modeling (MLM) as in BERT and link prediction, which drops and predicts edges in the input KG. Experiments on two domains demonstrate the effectiveness of the pretraining method.

Reviewers agree that the paper will make a good addition to NeurIPS, but disagree in their level of enthusiasm. The concerns expressed is that the approach is not particularly novel and that it requires entity linking, which adds more complexity to pretraining. Nevertheless, the reviewers agree that adding knowledge graph information to pretrained models is important, the experimental results are convincing, and the paper is clear and easy to read.

There was a productive discussion between the reviewers and the authors. As result, the paper was improved by adding results and clarifications. The improved version will make a good contribution to the conference program.
confidence: Less certain
Decision: Accept