Efficient Probabilistic Logic Reasoning with Graph Neural Networks

Abstract: Markov Logic Networks (MLNs), which elegantly combine logic rules and probabilistic graphical models, can be used to address many knowledge graph problems. However, inference in MLN is computationally intensive, making the industrial-scale application of MLN very difficult. In recent years, graph neural networks (GNNs) have emerged as efficient and effective tools for large-scale graph problems. Nevertheless, GNNs do not explicitly incorporate prior logic rules into the models, and may require many labeled examples for a target task. In this paper, we explore the combination of MLNs and GNNs, and use graph neural networks for variational inference in MLN. We propose a GNN variant, named ExpressGNN, which strikes a nice balance between the representation power and the simplicity of the model. Our extensive experiments on several benchmark datasets demonstrate that ExpressGNN leads to effective and efficient probabilistic logic reasoning.

Reviews and author responses:

In this paper the authors propose a system, called ExpressGNN, that combines MLNs and GNNs. This system is able to perform inference and learning the weights of the logic formulas.

The proposed approach seems valid and really intriguing. Moreover the problems it tackles, i.e. inference and learning over big knowledge graphs, are of foremost importance and are interesting for a wide community of researchers.
I have just one concern and it is about the experiments for the knowledge graph completion task. In fact, this task was performed only on one KG. I think the proposed system should be evaluated on more KGs.

For these reasons I think the paper, after an extension of the experimental results, should be accepted.

[Minor]
Page 3. “The equality holds” which equality are you talking about?


Thanks for your comments. We briefly respond to a couple of points as follows.


> The integration of variational EM and MLN has been explored in another work pLogicNet.

We have to clarify that our ExpressGNN work was proposed earlier than the pLogicNet. In fact, we have submitted an earlier version of our work to arXiv 15 days before the pLogicNet appeared on arXiv ( https://arxiv.org/abs/1906.08495 ). Due to the ongoing anonymous period, we could not provide the link of our arXiv submission here.


> With pLogicNet, the contributions are not surprising or novel.

1) As claimed above, we proposed the idea of integrating stochastic variational inference and MLN before the pLogicNet work appeared. As a concurrent and later work, pLogicNet also employs variational EM for MLN inference, which should not hurt the originality and novelty of our work.

2) Compared to pLogicNet, our work employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph. For example, an entity can be affected by its neighborhood entities, which is not modeled in pLogicNet but can be captured by GNNs. Our work models such implicit knowledge encoded in the graph structure to supplement the knowledge from logic formulae, while pLogicNet has no graph structure knowledge and only has a flattened embedding table for all the entities.

3) Our method is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN part and the embedding part. Thus, pLogicNet can be viewed as a special case of our work with the embedding part only.

4) We compared our method with pLogicNet in the experiments. Please refer to Table 3 for the experimental results. Our method achieves significantly better performance than pLogicNet (MRR 0.49 vs 0.33, Hits@10 60.8 vs 52.8) on the FB15K-237 dataset.

We have updated the paper to incorporate the discussions above.

First of all, thank you for your valuable comments. We briefly respond to a couple of points as follows.


> Why traditional MLN is computationally inefficient? Provide the inference time complexities.

The computational complexity of probabilistic MLN inference is known to be #P-complete when MLN was proposed [1]. To make it feasible, there are three categories of approximate inference methods: Monte Carlo methods, loopy belief BP, and variational methods [2]. Previous methods (including MCMC, BP, lifted BP) require to fully construct the ground Markov network before performing approximate inference, and the size of the ground Markov network is O(M^d) where M is the number of entities and d is the highest arity of the logic formula. Typically, there are a large number of entities in a practical knowledge graph, making the full grounding infeasible.

With mean-field approximation, our stochastic inference method avoids to fully construct the grounded Markov network, which only requires local grounding of the formulae in each sampled minibatch. Our method has constant time complexity for each sampled minibatch, and the overall time complexity is O(N) where N is the number of iterations. We have compared the inference efficiency on two benchmark datasets. Experimental results reported Fig. 4 show that our method is both more efficient and scalable than traditional MLN inference methods.


> Does Lifted BP reduce the computational cost of grounding?

Lifted BP constructs the minimal lifted network via merging the nodes as the first step, and then performs belief propagation on the lifted network to save the computational cost. However, there is no guarantee that the lifted network is much smaller than the ground network. In the worst case, the lifted network can have the same size as the original ground network [2]. Moreover, the construction of the lifted network is also computationally expensive, which is even slower than the construction of the full network as reported in Table 3 of their paper [2]. In fact, our experiments demonstrate that Lifted BP is NOT efficient even on small dataset like UW-CSE and Kinship (please refer to Fig. 4 in our paper), and it certainly cannot scale up to the FB15K-237 dataset.


> Why use Neural LP to learn the rules?

The FB15K-237 dataset is not designed for evaluating MLN inference / learning methods, and hence, have no logic formulae provided. Our work focuses on MLN inference and learning with a set of logic formulae, thus we need to generate the rules first. Similarly, recent work [3] uses simple brute-force search to generate the rules for MLN. However, brute-force rule search can be very inefficient on large-scale data. Instead, our method employs Neural LP to efficiently generate the rules. We use the training set only for rule learning, which guarantees that there is no information leakage during the evaluation on the test set.


> Why not compare to BoostSRL?

The BoostSRL work uses MC-SAT as the inference method, which has been compared with our work in the experiments. According to the inference time reported in Fig. 4, our method is much more efficient and scalable than MC-SAT.

Moreover, BoostSRL is not directly comparable to our method, since the task is completely different. Our method is designed for MLN inference and rule weight learning with logic rules provided, while BoostSRL was proposed for MLN structure learning, i.e., learning logic rules for MLN. We chose Neural LP instead of this method to generate the rules, since Neural LP has been demonstrated to be effective in rule induction on the Freebase dataset. In the updated paper, we have included BoostSRL as related work to supplement our literature review.


> MLN is fairly general, does GNN result in any loss of expressivity?

We have discussed the expressive power of GNNs in our paper in the section titled “Why combine GNN and tunable embeddings”. To make it more clear, in the updated paper, we change the section title to: “Expressive power of GNN as inference network ”. In this section, we have shown an example in Fig. 3 where GNN produces the same embedding for nodes that should be distinguished. We have also formally proved the sufficient and necessary condition to distinguish any non-isomorphic nodes in the knowledge graph. Inspired by this, we augment GNN with additional tunable embeddings to trade-off the compactness and expressiveness of the model.


> Related work should appear in the main paper.

Thanks for the suggestion. In the updated paper, we’ve added the related work section right after the introduction to provide a clear background of statistical relational learning and Markov Logic Networks.


References

[1] Richardson, Matthew, and Pedro Domingos. “Markov Logic Networks.” Machine Learning.

[2] Singla, Parag, and Pedro M. Domingos. “Lifted First-Order Belief Propagation.” AAAI.

[3] Qu, Meng, and Jian Tang. “Probabilistic Logic Neural Networks for Reasoning.” arXiv.


Thanks for your review comments. We briefly respond to your questions as follows.


> The proposed system should be evaluated on more KGs.

In fact, our method is evaluated on four benchmark datasets with four different KGs: UW-CSE, Cora, Kinship, and Freebase. These knowledge graphs are of different knowledge types and data distributions, and are widely used as benchmark datasets to evaluate MLNs and knowledge graph reasoning methods.


> Page 3. “The equality holds” which equality are you talking about?

“The equality holds” points to the equality in Eq. (2). To make it more clear, we have added a reference to Eq. (2) in the updated paper.


This paper proposes a framework for solving the probabilistic logic reasoning problem by integrating Markov neural networks and graph neural networks to combine their individual features into a more expressive and scalable framework. Graph neural networks are used for learning representations for Knowledge graphs and are quite scalable when it comes to probabilistic inference. But no prior rules can be incorporated and it requires significant amount of examples per target in order to converge. On the other hand, MLN are quite powerful for logical reasoning and dealing with noisy data but its inference process is computationally intensive and does not scale. Combining these two frameworks seem to result in a powerful framework which generalizes well to new knowledge graphs, does inference and is able to scale to large entities. 

Regarding its contribution, the paper seems to consider a training process which is done using the variational EM algorithm. The variational EM is used to optimize the ELBO term (motivation for this is the intractability of the computing the partition term). In the E-step, they infer the posterior distribution and in the M-step they learn the weights. The integration of variational EM algorithm and MLN has been explored in another work (pLogicNet: Probabilistic Logic Neural Networks for Reasoning), but this paper proposes a new pipeline of tools: MLN, GNN and variational EM which seem to outperform all the existing baseline methods.The paper looks technically sound to me and the evaluations results are delivered neatly, however the flow of the paper makes it a bit difficult to follow sometimes due to many topics covered in it.
Regarding the significance of the paper, it tries to combine logic reasoning and probabilistic inference which is of great interest among the researchers recently. ExpressGNN proves to generalise well and perform accurate inference due to the tunable embeddings added at the GNN.

Overall, the work of this paper seems technically sound but I don’t find the contributions particularly surprising or novel. Along with plogicnet, there have been many extensions and applications of Gnns, and I didn’t find that the paper expands this perspective in any surprising way. 


The paper proposes to use graph neural networks (GNN) for inference in MLN. The main motivation seems to be that inference in traditional MLN is computationally inefficient. The paper is cryptic about precisely why this is the case. There is some allusion in the introduction as to grounding being exponential in the number of entities and the exponent being related to the number of variables in the clauses of the MLN but this should be more clearly stated (e.g., does inference being exponential in the number of entities hold for lifted BP?). In an effort to speed up inference, the authors propose to use GNN instead. Since GNN expressivity is limited, the authors propose to use entity specific embeddings to increase expressivity. The final ingredient is a mean-field approximation that helps break up the likelihood expression. Experiments are conducted on standard MLN benchmarks (UW-CSE, Kinship, Cora) and link prediction tasks. ExpressGNN achieves a 5-10X speedup compared to HL-MRF. On Cora HL-MRF seems to have run out of memory. On link prediction tasks, ExpressGNN seems to achieve better accuracy but this result is a bit difficult to appreciate since the ExpressGNN can't learn rules and the authors used NeuralLP to learn the rules followed by using ExpressGNN to learn parameters and inference.

Here are the various reasons that prevent me from rating the paper favorably:

- MLNs were proposed in 2006. Statistical relational learning is even older. This is not a paper where the related work section should be delegated to the appendix. The reader will want to know the state of inference and its computational complexity right at the very beginning. Otherwise, its very difficult to read the paper and appreciate the results.

- Recently, a number of papers have been tried to quantify the expressive power of GNNs. MLN is fairly general, being able to incorporate any clause in first-order logic. Does the combination with GNN result in any loss of expressivity? This question deserves an answer. If so, then the speedup isn't free and ExpressGNN would be a special case of MLN, albeit with the advantage of fast inference.

- Why doesn't the paper provide clear inference time complexities to help the reader appreciate the results? At the very least, the paper should provide clear time complexities for each of the baselines.

- There are cheaper incarnations of MLN that the authors should compare against (or provide clear reasons as to why this is not needed). Please see BoostSRL (Khot, T.; Natarajan, S.; Kersting, K.; and Shavlik, J. 2011. Learning Markov logic networks via functional gradient boosting. In ICDM)

Meta-review: This paper is far more borderline than the review scores indicate. The authors certainly did themselves no favours by posting a response so close to the end of the discussion period, but there was sufficient time to consider the responses after this, and it is somewhat disappointing that the reviewers did not engage.

Reviewer 2 states that their only reason for not recommending acceptance is the lack of experiments on more than one KG. The authors point out they have experiments on more than one KG in the paper. From my reading, this is the case. I will consider R2 in favour of the paper in the absence of a response.

Reviewer 3 gives a fairly clear initial review which states the main reasons they do not recommend acceptance. While not an expert on the topic of GNNs, I have enough of a technical understanding to deem that the detailed response from the authors to each of the points does address these concerns. In the absence of a response from the reviewer, it is difficult to ascertain whether they would agree, but I will lean towards assuming they are satisfied.

Reviewer 1 gives a positive sounding review, with as main criticism "Overall, the work of this paper seems technically sound but I don’t find the contributions particularly surprising or novel. Along with plogicnet, there have been many extensions and applications of Gnns, and I didn’t find that the paper expands this perspective in any surprising way." This statement is simply re-asserted after the author response. I find this style of review entirely inappropriate and unfair: it is not a the role of a good scientific publication to "surprise". If it is technically sound, and in an area that the reviewer admits generates interest from reviewers, vague weasel words do not a reason for rejection make.

I recommend acceptance.

Decision: accept-poster