Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to mine explicit rules from KB embedding space, and casts it into a sparse reconstruction problem. Experiments demonstrate its ability of extracting reasonable rules on a few link prediction datasets.

The solution part sounds plausible. However, it confuses me that why we need to mine rules from learned KB embeddings. 

- It is still unclear what information these KB embeddings encode and it looks strange that we aim to learn rules including negation / disjunction from them.

- If the goal is to extract useful rules (for other applications), it is necessary to compare it to “graph random walk” (http://rtw.ml.cmu.edu/papers/lao-emnlp11.pdf) which could learn rules from KB graph directly.

- As there is only one KNN baseline, the experimental results seem pretty weak. At the least, it is necessary to show the original precision / recall of RESCAL, together with the proposed rule mining approach (with different max length), so we know how much the current information the current rule miner could recover.

In addition, the four datasets are all very small. Would it be able to scale it to WordNet or Freebase?

[Minor comments]

“Relational embedding” and “relation embedding” are used mixedly throughout the paper. I am not sure if they are well-defined terms (it is better to cite relevant paper).

review_title: review

final_score: rating: null
review_content: Thank you for this interesting paper. The weighting of different paths for link prediction was also present in Lao et al. Random Walk Inference and Learning in A Large Scale Knowledge Base (EMNLP 11); the current paper proposes to learn from paths directly in embedding space, which may be much faster. I still have a few questions:

1) The experimental section shows the relative performance of different rule miners, but there does not seem to be any reference with respect to the original algorithm (here, nonnegative RESCAL). How can we use these rules together with the original algorithm to perform link prediction? What performance gains can be expected? 

2) While the method to learn the paths seems applicable to many embedding-based algorithms for relational learning, I wonder what properties of the underlying algorithm are necessary for this principle to effectively work well. For instance, the authors use nonnegative RESCAL; to what extent is the non-negativity a critical feature for the approach to work?
review_title: questions

final_score: rating: null
review_content: - Nations, Kinship and UMLS are standard relational learning benchmarks. But most previous works use area under the precision recall curve as metric. How does the proposed approach compare to previous work with this metric?
- All datasets used in the experiments are rather small. How would FRM scale with larger datasets (high number of entities and/or of relation types and/or of triples)?
review_title: Questions

final_score: rating: 3: Clear rejection
review_content: This paper proposes a process to mine rules from vector space representations learned from KBs (using nonnegative RESCAL).

The paper is nicely written. 
But its motivations are unclear: what is the underlying motivation to mine rules from embedding spaces?
- If it is for better performance on link prediction then the paper does not show this. The experiments do not compare FRM against the performance of the original vector space model.
- If it is for a better interpretability and debugging of the representations learned by vector space models, then there should have more elements on this in the paper.

Other remarks:
- The fact that the performance of the methods in Figure 1 and 2 are not compared to any baseline is problematic.
- The scalability of the rule miner is a big drawback that should be addressed.
- Figure 3 does not do a good job at convincing that rule based systems should be used for prediction or interpretation. The learned rules are bad for both cases.


review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a nice idea of directly finding rules such as brother(father) => uncle in knowledge bases, by directly searching in embedding space. The idea is to interpret the successive application of relationships as the multiplication of the relation-dependent matrices in non-negative RESCAL. 

The experimental section provides an evaluation of the rules that are found by the algorithm. Nonetheless, the work seems only at its first stages for now, and many questions are left open:

1) while the approach to find rules seems very general, the reason why it should work is unclear. What properties of the embedding space or of the initial algorithm are required for this approach to find meaningful rules? Can we apply the same principles to other algorithms than non-negative RESCAL?

2) there is no real evaluation in terms of link prediction. How can we use these rules in conjunction with the original algorithm to improve link prediction? What performance gains can be expected? Can these rules find links that would not be found be the original algorithm in the first place?

3) scaling: for now the number of parameters of the rule miner is (#relationships)^(max. path length + 1). How does this method scale on standard benchmarks such as FB15k where there is more than a 1000 of relationships?

review_title: unconclusive experiments and missing theoretical motivation

Meta-review: The reviewers concur that the paper is well written and the topic is interesting, but that the authors have not put sufficient effort into motivating their approach and evaluating it. The baselines seem too simple, the evaluation is incomplete. It is furthermore disappointing that the authors not only did not respond to the reviews, but did not respond to the pre-review questions. There is little in this review process that would support the paper being accepted, and therefore I concur with the reviewers' opinion and support rejection.
Decision: Reject