Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a new method for knowledge graph reasoning: modeling the entity and relation with a probabilistic distribution rather than a single vector. Then the operations can be regarded as a function of probability distribution transformation. With an assumption of Gaussian distribution families, the entire process can be considered in a tractable manner. Empirical results on multiple datasets show that the method outperforms previous SOTA systems. The general idea is novel and very interesting. The experimental results are convincing. The writing is clear and well-structured. I like the idea a lot.


main_review: Pros:

1. A novel and interesting idea for modeling the entities and relations with a random variable with gaussian probabilistic distributions.
2. The designed operations are cleverly designed and seem reasonable to me.
3. The empirical results are convincing
4. The analysis is very interesting. I like Sec 5.5 a lot.

Cons: 

If my understanding is correct, the logical reasoning step (Union->Intersection-> … ) must be given in both training and testing. If so, I am not sure it is okay to claim that the proposed approach is “to solve FOE queries in a closed form solution”. My intuition is that inferring logical steps might be the most difficult problem.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
In this paper authors proposed a method to solve logical reasoning over knowledge graphs. They proposed the use of gaussian density function instead of previous methods like union of boxes. This shows good results in the experiments and also good visualization of the distribution.
main_review: Originality: 

The work is a novel addition of gaussian density function into the existing logical reasoning over knowledge graphs work. I like the intuitive explanation from the authors on how it gives better expressibility for queries over the knowledge graph.

Quality: 

The submission seems technically sound. The motivation is well explained with examples. And the result also mostly supports the hypothesis.

Although the final average is better for PERM vs CQD, but in some queries like 2t, 3t, t\intersection etc CQD seems to outperform PERM. Some analysis into these particular queries may shed more light about why this happens. It might give some query specific intuition and may help improve the work more.

I also have questions regarding how many queries were used for each type. And how were they averaged. Is the average weighted on number of queries? If some query class dominates the average. Because we are showing efficacy based on the average of all queries, I think an analysis of how the queries were chosen is important.

Drug recommendation experiment seems interesting. How was the ground truth set was constructed for this? Is it all the drugs that currently are in trial? It would be good to know how good each drug works, but that data is probably not available at this point. But this shows good potential for drug recommendation.

I liked the visualization of the gaussian representation. It gives some intuitive sense to the work.

Clarity: 

The submission is clearly written and seems well organized and easy to read. 

Significance: 

I think this is a very important problem to solve in this area. It should be impactful work for researchers in this domain.


limitations_and_societal_impact: did not see this section
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: For the multi-hop logical reasoning in the knowledge graph, this paper presents a probabilistic entity representation model. The proposed model encodes entities in knowledge graphs using a multivariate Gaussian density function. Then, this paper defines logical operations such as translation, intersection, and intersection and chained operations using Gaussian properties with the proposed model.
main_review: The proposed method that models an entity in knowledge graphs using the Gaussian density function seems reasonable. Furthermore, the logical operations and the chain reasoning are properly defined.

But, this paper leaves some negative or uncertain points.
As this paper mentioned, some previous studies have modeled both entities and queries by probabilistic distributions such as hyperboloid embeddings [8] and beta embeddings [11]. So, there is one question about the motivation of the proposed method. What is the main advantage of using the Gaussian density function? It would be better to include some analyses and/or discussions about the comparison with the methods using another probabilistic distributions instead of comparing box geometric approach.

The proposed method only supports existential positive first-order queries. That is, this paper cannot consider negation logical operation. Compared to the previous study (Beta embeddings [11]) that deals with the negation logical operation, the proposed method lacks the coverage of first-order logic.

The main concerns of this paper involve the experimental settings and results. First, the FB15k-237 and NELL995 datasets used in this paper are different from the ones used in previous studies [7, 10]. For example, according to Section C in the appendix, the number of relations and edges of FB15k-237 are 474 and 149,689, respectively. However, in [7, 10] papers, those are 237 and 272,115. The reviewer was wondering why this paper does not use the same dataset used in the previous studies [7, 10]. The reviewer speculates that the dataset used in this paper is the one used in [8]. However, the dataset of [8] is not the same as the one of [7, 10].

Second, the performances in Table 1 are quite strange. It looks like that the performances of BQE and CQD are borrowed from each paper, respectively. Since the dataset is different, their performances cannot be borrowed directly. For example, the BQE paper [10] provides the performances of HITS@1 and MRR, not the one of HITS@3. However, this paper uses MRR as HITS@3. In addition, the performance of BetaE in Table 1 and the one in Table 5 (in appendix) is the same. For these reasons, the reviewer cannot trust the experimental results.

It would be better to mention/include a training objective of the proposed model in detail. 

In Section 5.5, it would be better to include entity distribution as well as query distribution.

==

* Update after author responses *

I want to thank the authors for the additional information provided in a short period. The authors have provided experimental results on standard dataset. Since the concerns about the experiments were addressed, I raised my score.
limitations_and_societal_impact: It is properly addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents Probabilistic Entity Representation Model (PERM), a technique that models entities in a Knowledge Graph (KG) through a mixture of Gaussian densities.
main_review: Strengths:
- The paper is well written and easy to follow.
- The proposed technique unlocks not-trivial gains with respect to the state-of-the-art on several datasets.
- Although gaussian densities are not a new concept, and have been successfully applied to domains such as NLP, their application to chain reasoning over KG can be considered novel.
- The code to reproduce the experiments is available and will be made public.

Weakness:
- The numbers for CQD in Table 1 seems not to match those reported in their paper. For instance, for NELL995 1t should be 0.667 instead of  0.567, and so on. Please double check.
- I would have appreciated a discussion section, potentially highlighting some of the limitations of the proposed approach. Interesting questions that comes to mind are: (1) Is PERM robust against missing information in a KG? To what extent? (2) Is the proposed solution differentiable? Can it be trained end2end in the context of a broader framework?

Presentation comments:
- The first sentence in the abstract is a bit hard to parse. Why does logical reasoning over KG limit its application on large DB? Please rephrase or add a sentence to elaborate.
- The fact that KG's organisation provides a *fluid*-schema can be disputed (line 21), it is extremely static from my point of view. Please elaborate on the fluid adjective or rephrase. 
- Figure 1 is very hard to read on a A4 paper, you should make the font larger.
- Most of the Figures are not colour-blind friendly (in particular, Figure 3, Table 3 and Figure 4). Please make them so.
- Examples of good and bad PERM predictions could be useful.

limitations_and_societal_impact: the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: This paper presents a probabilistic treatment of KG entities, and introduces a logic over such probabilistic entity representations which supports multi-step reasoning about relationships in graphs. It shows the strengths of this approach on a number of graph tasks, and is clearly written. There is a fairly unanimous consensus from the reviewers that this is a strong paper, and worth publishing.

From a cursory read through, given the focus on logical reasoning on graphs, I was surprised not to see mentions of [1,2] which build on [3] (I should indicate that these are not my works, but just that I am familiar with this set of approaches) in at least related work, if not direct comparison (which would be possible on some datasets). I see the approach presented here as being very different in spirit and execution, so while this absence is surprising I don't think it is damning. I just indicate this remark FYI, and did not take it into consideration when making a recommendation for this paper.

I am happy to go with the reviewer consensus and recommend acceptance for this paper at NeurIPS.

[1] Minervini, Pasquale, et al. "Differentiable reasoning on large knowledge bases and natural language." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.

[2] Minervini, Pasquale, et al. "Learning reasoning strategies in end-to-end differentiable proving." International Conference on Machine Learning. PMLR, 2020.

[3] Rocktäschel, Tim, and Sebastian Riedel. "End-to-end differentiable proving." arXiv preprint arXiv:1705.11040 (2017)
Decision: Accept (Poster)