Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper uses domain-specific pretrained language models to biomedical knowledge graph completion.

- I like the research problem of the paper. Pretrained language models are definitely a way to extract knowledge from unstructured texts. It would be great if we could find a way to extract the knowledge from LM to structured or semi-structured stuff. Using LMs for knowledge graph completion is a clever way, which is easy to evaluate.

- Why is the domain-specific task evaluated in the paper? How about the open-domain setup?

- It's good to see the proposed method is complementary to previous methods. There would be some opportunities to better integrate these methods together.

- In terms of interpretation, there is a related paper "Knowledge Neurons in Pretrained Transformers", which also tries to align the neurons in LMs and knowledge.

- Is this a good way to evaluate pretrained LMs? A metric could be defined for this.
review_title: nice contribution

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method for link prediction in biomedical knowledge graphs that combines the predictions of knowledge graph embedding models with pre-trained language models. The main contribution of this paper is a routing method that acts as a switch between a set of KG embedding models and language models (that are fine-tuned for the link prediction task) and outputs the best model’s prediction for a given triple (h, r, t).

There are similar methods to this paper proposed in the general KG completion task (cited in this paper). This paper extends those methods to biomedical knowledge graphs. The key difference is that existing methods use a weighted average of different model’s predictions whereas the proposed model chooses one of the model’s predictions using a routing method. Empirical results show that this approach is effective for transductive settings. The main finding of the paper is that language models and knowledge graph embedding models are complementary.

**Reasons to accept**
1. The paper is well-written and performs a thorough empirical study by considering a wide range of methods and various combinations of them.
2. Performed detailed analysis of the model’s predictive performance.

**Reasons to reject**
1. The proposed model performs much worse than the fine-tuned LM model in an inductive setting. Also, the reason to choose the nearest neighbor of the unseen entity is not well justified.  Since the embedding of the unseen entity is obtained using a fine-tuned LM, why can’t we directly use that? 
2. There are other strong baselines for inductive settings such as R-GCN [1], CompGCN [2], and [3] that are not considered for comparisons. Also, integrating these models into the proposed method might further improve the performance.

**Comments**
Please mention the number of relations in each dataset in Table 1.

**Missing references**

[1] Schlichtkrull, M.S., Kipf, T.N., Bloem, P., van den Berg, R., Titov, I., Welling, M.: Modeling relational data with graph convolutional networks. In: Proceedings of ESWC 2018. LNCS, vol. 10843, pp. 593–607. Springer (2018)

[2] Vashishth, S., Sanyal, S., Nitin, V., Talukdar, P.: Composition-based multi-relational graph convolutional networks. In: International Conference on Learning Representations (2020)

[3] R Bhowmik, G de Melo. Explainable Link Prediction for Emerging Entities in Knowledge Graphs.  International Semantic Web Conference, 2020

review_title: Strong empirical study for transductive setting, but weaker in the inductive setting

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Summary**:

This paper presented a comprehensive empirical study of applying various scientific language models to the biomedical knowledge graph completion task. The authors found that language models and knowledge graph embedding models have complementary strengths, and further proposed a routing method, which is able to integrate these two models. Domain-specific biomedical language models are carefully evaluated.

**Reasons to accept**:

1. The paper is well-organized and nicely presented. A wide range of methods including the very recent ones are surveyed. I am not an expert in this research direction, and I learned a lot from this paper.

2. Extensive experiments are conducted to investigate the existing models under different settings and verify the effectiveness of the proposed routing method under the transductive settings. Details of the experimental settings such as datasets, hyper-parameters, baselines are included. 

**Reasons to reject**:

No strong reasons for rejection. However, as mentioned by the other reviewer, graph neural network-based models should also be considered for discussions and comparisons.

**Questions**: ​

1.Though investigating general-domain language models is out of the scope of the paper, I am wondering if the proposed method for domain-specific LM also works for general-domain language models? 

2.For biomedical knowledge base completion, do we have any domain-specific issues when applying LM to it? 

3.The last question is whether integrating LM-based models with KG embedding models is an under-explored research question for the general-domain settings.

review_title: A comprehensive emprical study

Meta-review: The paper presents an in-depth evaluation of biomedical language models to the task of biomedical knowledge graph completion. A main takeaway is that language models and knowledge graph embedding models capture complementary signals. The paper then proposes a routing method to switch between the two, obtaining good results. The paper presents a thorough empirical study by considering a wide range of methods.
Decision: Accept