Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a model for probabilistic rule learning to automate the completion of probabilistic databases. The proposed method uses lifted inference which helps in computational efficiency given that non-lifted inference in rules containing ungrounded variables can be extremely computationally expensive.
The databases used contain binary relations and the probabilistic rules that are learned, are also learned for discovering new binary relations. The use of lifted inference restricts the proposed model to only discover rules that are a union of conjunctive queries.
The proposed approach uses AMIE+, a method to generate deterministic rules, to generate a set of candidate rules for which probabilistic weights are then learned. The model initializes the rule probabilities as the confidence scores estimated from the conditional probability of the head being true given that the body is true, and then uses a maximum likelihood estimate of the training data to learn the rule probabilities. 

The paper presents empirical comparison to deterministic rules and ProbFOIL+ on the NELL knowledge base. The proposed approach marginally performs better than deterministic rule learning.

The approach proposed is straightforward and depends heavily on the candidate rules produced by the AMIE+ algorithm. The paper does not provide insights into the drawbacks of using AMIE+, the kind of rules that will be hard for AMIE+ to propose, how can the proposed method be improved to learn rules beyond the candidate rules. 

‘End-to-end differentiable proving’ from NeurIPS (NIPS) 2017 also tackles the same problem and it would be nice to see comparison to that work. 

review_title: Sound approach for rule learning but heavy dependence on black-box algorithm to propose candidate rules

final_score: rating: 6: Marginally above acceptance threshold
review_content: In general, the paper presents a routing practice, that is, apply lifted probabilistic inference to rule learning over probabilistic KBs, such that the scalability of the system is enhanced but being applicable to a limited scope of rules only. I would not vote for reject if other reviewers agree to acceptance.

Specifically, the proposed algorithm SafeLearner extends ProbFOIL+ by using lifted probabilistic inference (instead of using grounding), which first applies AMIE+ to find candidate deterministic rules, and then jointly learns probabilities of the rules using lifted inference.

The paper is structured well, and most part of the paper is easy to follow.

I have two major concerns with the motivation. It reads that there are two challenges associated with rule learning from probabilistic KBs, i.e., sparse and probabilistic nature. 
1) While two challenges are identified by the authors, but the paper deals with the latter issue only? How does sparsity affect the algorithm design?

2) The paper can be better motivated, although there is one piece of existing work for learning probabilistic rules from KBs (De Raedt et al. [2015]). Somehow, I am not convinced by the potential application of the methods; that is, after generating the probabilistic rules, how can I apply the probabilistic rules? It will be appreciated if the authors can present some examples of the use of probabilistic rules. Moreover, if it is mainly to complete probabilistic KBs, how does this probabilistic logics based approach compare against embedding based approach?

review_title: Good but not surprising

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper was interesting and rather clearly written, as someone who didn't have much background in rule learning.

Section 5.1.1.1 : "(ii) tuples contained in the answer of Q' where Q' is the same as Q but without the rule with empty body, but not in the training set"  is unclear.

In the algorithm, line 22, aren't rules removed until H leads to a "safe" UCQ ?

Section 6.1 : "In line 7 we formulate a UCQ Q from all the candidate rules in H (explained in 5.2 with an example)". I was unable to find the example in section 5.2 

It would be interesting to have an idea of the maximum scale that ProbFoil+ can handle, since it seems to be the only competitor to the suggested method. 

In section 7.2 does the "learning time" include the call to AMIE+ ? if not, it would be interesting to break down the time into its deterministic and learning components, since the former is only necessary to retrieve the correct probabilities.

Being new to this subject, I found the paper to be somewhat clear. However, I found that it was sometimes hard to understand what was a part of the proposed system, and what was done in Amie+ or Slimshot.

For example, "But, before calling Slimshot
for lifted inference on the whole query, we first break down the query to independent subqueries such that no variable is common in more than one sub-query. Then, we perform
inference separately over it and later unify the sub-queries to get the desired result."

This is described as important to the speedup over ProbFoil+ in the conclusion, yet doesn't appear in Algorithm 1. 

Similarly, "it caches the structure of queries before doing inference " is mentioned in the conclusion but I couldn't map it to anything in Algorithm 1 or in the paper.


I lean toward an accept because the work seems solid, but I feel like I don't have the background required to judge on the contributions of this paper, which seems to me like a good use of Amie+/Slimshot with a reasonable addition of SGD to learn rule weights. Some of the components which are sold as important for the speed-up in the conclusion aren't clear enough in the main text. Some numbers to experimentally back-up how important these additions are to the algorithms would be welcome. 

review_title: review

Meta-review: metareview: The paper presents a method of learning probabilistic rules from
a probabilistic dataset of KB tuples.  They first use existing
deterministic rule-learning algorithm AMIE+ to get candidate
rules and then learn probabilistic rules using lifted inference.
The paper is written clearly.  Authors have responded the
reviewers' concerns well.  Overall there are some concerns that the
contributions of the paper are not substantial enough in quantity
and depth.  Given the vast existing literature on the topic,
the authors should try to resolve the questions of comparisons
that naturally arise.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)