Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studies the problem of link prediction for knowledge graph completion with rule-based methods. Specifically, it builds on a recent strong rule-based method, AnyBURL, for mining rules from a KG, and mainly studies the application of the mined rules. Recognizing the limitations of the Maximum and Noisy-OR rule aggregation strategies used in prior work, this paper proposes SAFRAN, a rule application strategy that clusters rules of the same relation based on their similarity with the hope that redundant rules will be clustered together. The Maximum strategy is then used within each cluster and the Noisy-OR strategy is used to aggregate clusters. Empirical results on FB15K-237, WN18RR, and YAGO3-10 are strong, outperforming embedding-based methods as well as the Maximum and Noisy-OR baselines.

Strengths
- Rule-based link prediction is an important problem with clear advantages in interpret ability
- This paper makes a focused and clear contribution in rule application
- Strong empirical results

Weaknesses
- Sec 2.2, the description of AnyBURL is not very self-contained and one has to resort to the original paper to get a sufficient understanding for the rest of this paper. I understand that we don’t want to spend too much space on repeating prior work, but it could have been described more clearly with the same amount of space. Typos and confusing sentences in this section make it even harder. For example, “ In a rule r uppercase letters represent variables and lowercase letters represent variables.” I believe the second “variables” should be “constants” (after reading the original AnyBURL paper). The next sentence is also very confusing.
- Examples may be taking a bit too much space throughout the paper. It’d be better to use some of the space to have a more in-depth, quantitative analysis of the clustering quality as well as error analysis and case studies. Currently the experiments are a bit thin.
review_title: A focused contribution that shows strong empirical gains

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces SAFRAN rule application framework to improve upon the existing AnyBURL interpretable rule-based link prediction approach. SAFRAN helps to solve the redundancies that affect the aggregation step in AnyBURL and introduces Non-redundant Noisy-OR (NRNO) to detects and clusters redundant rules before the aggregation step. In several link prediction benchmarks, SAFRAN achieves SOTA performances for rule-based algorithms and in two out of three datasets even beats multiple embedding-based algorithms with good explainability.

Strengths: 
1) The paper is generally well-written and easy to follow, the code is also shared for reproducibility.
2) This paper is well-motivated, in order to reduce the redundant rules which limit the performance of AnyBURL, the author proposes to conduct the rule clustering first based on a redundant estimation metric, then aggregate rules in a cluster using the Maximum approach, and finally aggregate rules in different clusters with Noisy-OR method. The resulting aggregation technique is called Non-redundant Noisy-OR (NRNO), which possesses very good explainability as shown in the Figure 1 example.
3) Experimental results show that SAFRAN achieves state-of-the-art performances over latent embedding-based and interpretable-based approaches on FB15K-237 and WN18RR datasets. While in YAGO3-10,  SAFRAN beats previous interpretable baselines. Overall, the performance improvements look impressive.

Weakness and Questions: 
1) Though effective, the NRNO approach proposed in the paper is just a combination of the well-known Maximum and Noisy-OR aggregation methods. Hence, the technical novelty of NRNO is relatively limited.
2) SAFRAN relies on the threshold t_{hd}, which is a list of hyperparameters that may take additional effort to decide with grid search or random search.
3) The paper forgets to compare with HAKE (Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. “Learning hierarchy-aware knowledge graph embeddings for link prediction”, AAAI 2020), which is also a recent competitive baseline. Also, the ComplEx performance numbers in [Rossi et al., 2021] are slightly higher than in Table 6 of the NRNO paper.
4) I didn’t very well understand the meaning of ”VS“ in Table 6. What do the numbers in the VS line indicate? Could you elaborate more on this? 
5) There are some minor typos, for example, on Page 4, the fourth last line - “Is is” should be “It is”; on Page 5, the tenth last line - “(i.e)” seems to be redundant, if not it should be “i.e.” instead.

===== After Rebuttal =====

The author's response solves my concerns and I will keep my original rating (6).
review_title: Generally a paper with good motivation and considerable performance improvements

final_score: rating: 6: Marginally above acceptance threshold
review_content: --------

**Summary**

The paper proposes an interpretable rule-based approach for link prediction in knowledge graphs (KGs). The authors identify a common weakness (redundancy) in the following two prior ideas:
* maximum aggregation [Anytime bottom-up rule learning for knowledge graph completion, IJCAI'19]
* noisy-OR aggregation [Fast rule mining in ontological knowledge bases with amie+, VLDB'15].

Experiments on real-world KG datasets demonstrate that a simple combination of the two ideas can overcome redundancy to an extent and compete with state-of-the-art embedding-based / deep-learning-based methods in terms of link prediction performance.

--------

**Quality**

The technical contents of the paper (problem definition, methodology, evaluation) are sound. Claims are well-supported by experiments. However, the authors need to carefully evaluate the limitations of their proposal.

--------

**Clarity**

The illustrations are helpful to understand the main ideas of the approach. The authors have shared the code. A pseudo code would enhance the clarity further and facilitate an expert to reproduce the results of the paper.

--------


**Originality**

The tasks and methods are not new. The main proposal (Non-redundant Noisy-OR) is a simple combination of max aggregation and Noisy-OR which are well-known.  The fact that the simple combination overcomes the common weakness to an extent (Figure 1) is somewhat interesting.

--------

**Significance**

The paper is relevant to the AKBC community. The results seem significant in comparison to rule-learning baselines. However, the paper can be strengthened by 
* positioning with explainable KG embeddings,  e.g., (i) Interaction Embeddings for Prediction and Explanation in Knowledge Graphs, WSDM'19, (ii) Faithfully explaining predictions of knowledge embeddings, ENIAC'19 and
* conducting experiments on inductive relation prediction (learned rules can be easily applied to unseen entities at test time), e.g., (i) Communicative Message Passing for Inductive Relation Reasoning, AAAI'21, (ii) Topology-Aware Correlations Between Relations for Inductive Link Prediction in Knowledge Graphs, AAAI'21.

--------
review_title: Good paper but can be improved

Meta-review: The reviewers see many merits in the paper, and most of their concerns are addressed by the feedback. We are happy to recommend acceptance. Please make sure to account for all reviewer comments when preparing the camera-ready version.
Decision: Accept