Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Knowledge Graph Embedding (KGE) methods solve link prediction problems by learning node embeddings and a score function such that the correct triples are ranked the highest. On the hand, AnyBURL is a symbolic alternative which learns rules for ranking triples. The paper proposes a simple aggregation approach for combining the ranking results from KGE methods with those from AnyBURL. The motivation of this approach is to ultimately harness the strengths of both symbolic and sub-symbolic methods. The method is evaluated by studying the improvement in link prediction performance using benchmark datasets.

Overall, the idea presented in the paper is interesting because of its simplicity. Despite the weak results obtained in the empirical study, I think it presents an exciting new direction for the integration of symbolic and sub-symbolic methods. However, I strongly advice the authors to improve the text to make it read better.

Strengths:
* The aggregation function (in Section 5) is simple and the intuition behind this approach is easy to understand.
* Structure of the paper is easy to follow.
* The analysis presented in Section 4.2 are insightful.
* The results from the rule-based approach are explainable. However, this feature has not been thoroughly studied.

Weaknesses:
* The writing style of the paper could be improved to make it read better. There are many awkwardly phrased sentences throughout the text: “We abstain here from...“; “Unfortunately, AnyBURL is to our knowledge the only symbolic approach...“; “Within this work we have not presented a sophisticated new knowledge base completion method.”
* The results reported in the paper show only very little improvements (1-3% improvement in Mean Reciprocal Rank). Therefore, I feel that the title of the paper is misleading.

Minor Typos:
Section 4.2 (page 5) “We first look at two predictions” → “three predictions”
Abstract (page 1) “embbedings” → “embeddings”
Related Work (page 2) “the the” → “the”
Section 6.1 (page 8) “Further” → “Furthermore”

review_title: Some insights, but only minor improvements obtained.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a comparative study on symbolic methods and embedding methods for knowledge graph completion. From the manually observed ranking results of two kinds of approaches, the authors find that rule-based method AnyBURL tends to perform well on complicated rules that express transitivity of multiple relations, and that KG embedding methods sometimes will output meaningless result due to its learned similarity measure, which may be remedied by rule-based methods. Accordingly, a simple aggregation reranking approach is also proposed, and it shows promising performance gain for several embedding approaches in the KG completion task. 

I find qualitative analysis in comparing symbolic and embedding-based methods is insightful. The illustrated examples are closely related to the proposed approach, which is simple yet shown to be effective in boosting prediction performance. In general, the paper is well organized and easy to follow. Besides, I still have some questions regarding the analysis and experiments.

- It is intuitive to examine the cherry-picked examples and discover the potential patterns of various methods, but from the macro perspective, it would be better to provide more global statistical information, for instance, how many examples are there similar to the Darwin example? If it accounts for a very large proportion, it is not surprising the proposed method can work well. If there are only very limited similar cases, the performance gain may be caused by other reasons. In other words, larger-scale of analysis is preferred to demonstrate global statistical information.

- Since KG embedding methods can be also used to predict head entity or relation, is the proposed aggregation method able to be applied to these two tasks as well?

- It seems that the coefficient beta in the scoring function plays an important role in balancing the contribution of rule-based confidence scores and embedding-based scores. It would be more convincing to show the ablation study on this parameter.
review_title: An interesting paper with well-motivated examples and a simple aggregation approach

final_score: rating: 7: Good paper, accept
review_content: The authors investigate ways to ensemble symbolic and soft KB completion models. The symbolic approach infers rules from the training data, and the soft approach learns to embed the KB into a geometric space.
The authors start by analyzing the behavior of the two approaches and specify cases where each of them fail.
This analysis is used to come up with several heuristics for combining the predictions of the two approaches. 
On several datasets, the authors show that the proposed ensemble is better (although not drastically) than the two approaches on their own.

The paper is well motivated, and can give insights about the task and its leading models. The paper is also well written and easy to follow.

Having said that, the improvements the paper shows are somewhat limited (few MRR points), and ensembling two different approaches together is standard for pushing the performance.
review_title: Well motivated ensemble of symbolic and soft KB completion models.

Meta-review: This paper presents an ensemble scheme to combine predictions from embedding and rule-based KG completion methods. While the proposed method is extremely straightforward, the aggregated method does perform consistently better compared to the component methods. 

Overall, we think the paper is interesting, well-motivated, and adds value to the AKBC program. Hence, we recommend acceptance.
Decision: Accept