Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes an extension of NeuralLP that is able to learn a very restricted (in terms of expressiveness) set of logic rules involving numeric properties. The basic idea behind NeuralLP is quite simple: traversing relationships in a knowledge graph can be done by multiplicating adjacency matrices, and which rules hold and which ones don't can be discovered by learning an attention distribution over rules from data.

The idea is quite clever: relationships between numeric data properties of entities, such as age and heigh, can also be linked by relationships such as \leq and \geq, and those relations can be treated in the same way as standard knowledge graph relationship by the NeuralLP framework.

A major drawback in applying this idea is that the corresponding relational matrix is expensive to both materialise, and use within the NeuralLP framework (where matrices are mostly sparse). To this end, authors make this process tractable by using dynamic programming and by defining such a matrix as a dynamic computation graph by means of the cumsum operator.  Furthermore, authors also introduce negated operators, also by defining the corresponding adjacency matrices by means of computation graphs.

Authors evaluate on several datasets - two real world and two synthetic - often showing more accurate results than the considered baselines.


One thing that puts me off is that, in Table 2, AnyBurl (the single one baseline authors considered other than the original NeuralLP) yields better Hits@10 values than Neural-LP-N, but the corresponding bold in the results is conveniently omitted.

Another concern I have is that the expressiveness of the learned rules can be somehow limited, but this paper seems like a good star towards learning interpretable rules involving multiple modalities.


Missing references - authors may want to consider citing https://arxiv.org/abs/1906.06187 as well in Sec. 2 - it seems very related to this work.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposed several extensions to the Neural LP work. Specifically, this paper addresses several limitations, including numerical variables, negations, etc. To efficiently compute these in the original Neural LP framework, this paper proposed several computation tricks to accelerate, as well as to save memory. Experiments on benchmark datasets show significant improvements over previous methods, especially in the case where numerical variables are required. 

I think overall the paper is written clearly, with good summarization of existing works. Also I like the simple but effective tricks for saving the computation and memory.

One main concern is, how general this approach would be? As it is a good extension for Neural LP, it is not clear that the framework of Neural LP is flexible or powerful enough in general. For example, if rules contain quantifiers, how would this be extended? 

Minor comments:

1) 4.1,  “O(n^2/2)” -- just put O(n^2) or simply write as n^2/2.
2) How are the rules from in Eq (2)? i.e., how is \beta_i selected for each i? In the extreme case it would be all the permutations.
3) I would suggest a different name other than Neural-LP-N, as it is somewhat underselling this work. Also it makes Table 2 not that easy to read.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes an interesting extension to the Neural LP framework for learning numerical rules in knowledge graphs. The proposed method can handle predicates involving the comparison of the numerical attribute values. The authors demonstrate its effectiveness on both synthetic knowledge graphs and the parts of existing knowledge graphs which consider numerical values.

I recommend the paper to be rejected in its current form for the following 3 reasons:

(1) Although the idea of making numerical rules differentiable is interesting, the current proposed method can only deal with one form of numerical predicate, which is numerical comparison. The limitation to such a special case makes the paper somewhat incremental. 

(2) The paper does not do a great job of convincing the reader that the problem it is trying to solve is an important matter, or the proposed method is indeed effective in some applications. Although the proposed method does a good job in synthetic experiments, outperforming existing methods by a large margin, its performance on the numerical variants of Freebase/DBPedia dataset does not show consistent significant improvement. The authors should try to find a real-world domain which can really demonstrate the effectiveness of the method.

(3) The experiment section lacks more detailed analysis which can intuitively explain how well the proposed method performs on the benchmarks. A good place to start with is to visualize(print out) the learned numerical rules and see if they make any sense. The experiment section needs significant improvement, especially when there is space left.


The authors can consider improving the paper based on the above drawbacks. I encourage the authors to re-submit the paper once it's improved. 

review_title: Official Blind Review #3

Meta-review: This paper presents a number of improvements on existing approaches to neural logic programming. The reviews are generally positive: two weak accepts, one weak reject. Reviewer 2 seems wholly in favour of acceptance at the end of discussion, and did not clarify why they were sticking to their score of weak accept. The main reason Reviewer 
 1 sticks to 6 rather than 8 is that the work extends existing work rather than offering a "fundamental contribution", but otherwise is very positive. I personally feel that
a) most work extends existing work
b) there is room in our conferences for such well executed extensions (standing on the shoulders of giants etc).

Reviewer 3 is somewhat unconvinced by the nature of the evaluation. While I understand their reservations, they state that they would not be offended by the paper being accepted in spite of their reservations.

Overall, I find that the review group leans more in favour of acceptance, and an happy to recommend acceptance for the paper as it makes progress in an interesting area at the intersection of differentiable programming and logic-based programming.
Decision: Accept (Poster)