Efficient Probabilistic Logic Reasoning with Graph Neural Networks

1 INTRODUCTION . Knowledge graphs collect and organize relations and attributes about entities , which are playing an increasingly important role in many applications , including question answering and information retrieval . Since knowledge graphs may contain incorrect , incomplete or duplicated records , additional processing such as link prediction , attribute classification , and record de-duplication is typically needed to improve the quality of knowledge graphs and derive new facts . Markov Logic Networks ( MLNs ) were proposed to combine hard logic rules and probabilistic graphical models , which can be applied to various tasks on knowledge graphs ( Richardson & Domingos , 2006 ) . The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data , while the graphical model formalism provides a principled framework for dealing with uncertainty in data . However , inference in MLN is computationally intensive , typically exponential in the number of entities , limiting the real-world application of MLN . Also , logic rules can only cover a small part of the possible combinations of knowledge graph relations , hence limiting the application of models that are purely based on logic rules . Graph neural networks ( GNNs ) have recently gained increasing popularity for addressing many graph related problems effectively ( Dai et al. , 2016 ; Li et al. , 2016 ; Kipf & Welling , 2017 ; Schlichtkrull et al. , 2018 ) . GNN-based methods typically require sufficient labeled instances on specific end tasks to achieve good performance , however , knowledge graphs have the long-tail nature ( Xiong et al. , 2018 ) , i.e. , a large portion the relations in only are a few triples . Such data scarcity problem among long-tail relations poses tough challenge for purely data-driven methods . In this paper , we explore the combination of the best of both worlds , aiming for a method which is data-driven yet can still exploit the prior knowledge encoded in logic rules . To this end , we design a simple variant of graph neural networks , named ExpressGNN , which can be efficiently trained in the variational EM framework for MLN . An overview of our method is illustrated in Fig . 1 . ExpressGNN and the corresponding reasoning framework lead to the following desiderata : Variational EM Posterior Encoding Likelihood Decoding GNNMLN Knowledge Graph 𝜙 '' ( ⋅ ) formula potential predicate posterior Figure 1 : Overview of our method for combining MLN and GNN using the variational EM framework . • Efficient inference and learning : ExpressGNN can be viewed as the inference network for MLN , which scales up MLN inference to much larger knowledge graph problems . • Combining logic rules and data supervision : ExpressGNN can leverage the prior knowledge encoded in logic rules , as well as the supervision from graph structured data . • Compact and expressive model : ExpressGNN may have small number of parameters , yet it is sufficient to represent mean-field distributions in MLN . • Capability of zero-shot learning : ExpressGNN can deal with the zero-shot learning problem where the target predicate has few or zero labeled instances . 2 RELATED WORK . Statistical relational learning . There is an extensive literature relating the topic of logic reasoning . Here we only focus on the approaches that are most relevant to statistical relational learning on knowledge graphs . Logic rules can compactly encode the domain knowledge and complex dependencies . Thus , hard logic rules are widely used for reasoning in earlier attempts , such as expert systems ( Ignizio , 1991 ) and inductive logic programming ( Muggleton & De Raedt , 1994 ) . However , hard logic is very brittle and has difficulty in coping with uncertainty in both the logic rules and the facts in knowledge graphs . Later studies have explored to introduce probabilistic graphical model in logic reasoning , seeking to combine the advantages of relational and probabilistic approaches . Representative works including Relational Markov Networks ( RMNs ; Taskar et al . ( 2007 ) ) and Markov Logic Networks ( MLNs ; Richardson & Domingos ( 2006 ) ) were proposed in this background . Markov Logic Networks . MLNs have been widely studied due to the principled probabilistic model and effectiveness in a variety of reasoning tasks , including entity resolution ( Singla & Domingos , 2006a ) , social networks ( Zhang et al. , 2014 ) , information extraction ( Poon & Domingos , 2007 ) , etc . MLNs elegantly handle the noise in both logic rules and knowledge graphs . However , the inference and learning in MLNs is computationally expensive due to the exponential cost of constructing the ground Markov network and the NP-complete optimization problem . This hinders MLNs to be applied to industry-scale applications . Many works appear in the literature to improve the original MLNs in both accuracy ( Singla & Domingos , 2005 ; Mihalkova & Mooney , 2007 ) and efficiency ( Singla & Domingos , 2006b ; 2008 ; Poon & Domingos , 2006 ; Khot et al. , 2011 ; Bach et al. , 2015 ) . Nevertheless , to date , MLNs still struggle to handle large-scale knowledge bases in practice . Our framework ExpressGNN overcomes the scalability challenge of MLNs by efficient stochastic training algorithm and compact posterior parameterization with graph neural networks . Graph neural networks . Graph neural networks ( GNNs ; Dai et al . ( 2016 ) ; Kipf & Welling ( 2017 ) ) can learn effective representations of nodes by encoding local graph structures and node attributes . Due to the compactness of model and the capability of inductive learning , GNNs are widely used in modeling relational data ( Schlichtkrull et al. , 2018 ; Battaglia et al. , 2018 ) . Recently , Qu et al . ( 2019 ) proposed Graph Markov Neural Networks ( GMNNs ) , which employs GNNs together with conditional random fields to learn object representations . These existing works are simply data-driven , and not able to leverage the domain knowledge or human prior encoded in logic rules . To the best of our knowledge , ExpressGNN is the first work that connects GNNs with first-order logic rules to combine the advantages of both worlds . Knowledge graph embedding . Another line of research for knowledge graph reasoning is in the family of knowledge graph embedding methods , such as TransE ( Bordes et al. , 2013 ) , NTN ( Socher et al. , 2013 ) , DistMult ( Kadlec et al. , 2017 ) , ComplEx ( Trouillon et al. , 2016 ) , and RotatE ( Sun et al. , 2019 ) . These methods design various scoring functions to model relational patterns for knowledge graph reasoning , which are very effective in learning the transductive embeddings of both entities and relations . However , these methods are not able to leverage logic rules , which can be crucial in some relational learning tasks , and have no consistent probabilistic model . Compared to these methods , ExpressGNN has consistent probabilistic model built in the framework , and can incorporate knowledge from logic rules . A recent concurrent work Qu & Tang ( 2019 ) has proposed probabilistic Logic Neural Network ( pLogicNet ) , which integrates knowledge graph embedding methods with MLNs with EM framework . Compared to pLogicNet which uses a flattened embedding table as the entity representation , our work explicitly captures the structure knowledge encoded in the knowledge graph with GNNs and supplement the knowledge from logic formulae for the prediction task . 3 PRELIMINARY Knowledge Graph . A knowledge graph is a tuple K = ( C , R , O ) consisting of a set C = { c1 , . . . , cM } of M entities , a set R = { r1 , . . . , rN } of N relations , and a collection O = { o1 , . . . , oL } of L observed facts . In the language of first-order logic , entities are also called constants . For instance , a constant can be a person or an object . Relations are also called predicates . Each predicate is a logic function defined over C , i.e. , r ( · ) : C × . . .× C 7→ { 0 , 1 } . In general , the arguments of predicates are asymmetric . For instance , for the predicate r ( c , c′ ) : = L ( c , c′ ) ( L for Like ) which checks whether c likes c′ , the arguments c and c′ are not exchangeable . With a particular set of entities assigned to the arguments , the predicate is called a ground predicate , and each ground predicate ≡ a binary random variable , which will be used to define MLN . For a d-ary predicate , there are Md ways to ground it . We denote an assignment as ar . For instance , with ar = ( c , c′ ) , we can simply write a ground predicate r ( c , c′ ) as r ( ar ) . Each observed fact in knowledge bases is a truth value { 0 , 1 } assigned to a ground predicate . For instance , a fact o can be [ L ( c , c′ ) = 1 ] . The number of observed facts is typically much smaller than that of unobserved facts . We adopt the open-world paradigm and treat these unobserved facts ≡ latent variables . As a clearer representation , we express a knowledge base K by a bipartite graph GK = ( C , O , E ) , where nodes on one side of the graph correspond to constants C and nodes on the other side correspond to observed factsO , which is called factor in this case . The set of T edges , E = { e1 , . . . , eT } , connect constants and the observed facts . More specifically , an edge e = ( c , o , i ) between node c and o exists , if the ground predicate associated with o uses c as an argument in its i-th argument position ( Fig . 2 ) . Markov Logic Networks . MLNs use logic formulae to define potential functions in undirected graphical models . A logic formula f ( · ) : C × . . .× C 7→ { 0 , 1 } is a binary function defined via the composition of a few predicates . For instance , a logic formula f ( c , c′ ) can be Smoke ( c ) ∧ Friend ( c , c′ ) ⇒ Smoke ( c′ ) ⇐⇒ ¬Smoke ( c ) ∨ ¬Friend ( c , c′ ) ∨ Smoke ( c′ ) , where ¬ is negation and the equivalence is established by De Morgan ’ s law . Similar to predicates , we denote an assignment of constants to the arguments of a formula f as af , and the entire collection of consistent assignments of constants as Af = { a1f , a2f , . . . } . A formula with constants assigned to all of its arguments is called a ground formula . Given these logic representations , MLN can be defined as a joint distribution over all observed facts O and unobserved factsH as Pw ( O , H ) : = 1Z ( w ) exp ( ∑ f∈F wf ∑ af∈Af φf ( af ) ) , ( 1 ) where Z ( w ) is the partition function summing over all ground predicates and φf ( · ) is the potential function defined by a formula f as illustrated in Fig . 2 . One form of φf ( · ) can simply be the truth value of the logic formula f . For instance , if the formula is f ( c , c′ ) : = ¬S ( c ) ∨¬F ( c , c′ ) ∨S ( c′ ) , then φf ( c , c ′ ) can simply take value 1 when f ( c , c′ ) is true and 0 otherwise . Other more sophisticated φf can also be designed , which have the potential to take into account complex entities , such as images or texts , but will not be the focus of this paper . The weight wf can be viewed as the confidence score of the formula f : the higher the weight , the more accurate the formula is . Difference between KG and MLN . We note that the graph topology of knowledge graphs and MLN can are very different , although MLN is defined on top of knowledge graphs . Knowledge graphs are typically very sparse , where the number of edges ( observed relations ) is typically linear in the number of entities . However , the graphs associated with MLN are much denser , where the number of nodes can be quadratic or more in the number of entities , and the number of edges ( dependency between variables ) is also high-order polynomials in the number of entities .

Summary: This paper proposes a framework for solving the probabilistic logic reasoning problem by integrating Markov neural networks and graph neural networks to combine their individual features into a more expressive and scalable framework. Graph neural networks are used for learning representations for Knowledge graphs and are quite scalable when it comes to probabilistic inference. But no prior rules can be incorporated and it requires significant amount of examples per target in order to converge. On the other hand, MLN are quite powerful for logical reasoning and dealing with noisy data but its inference process is computationally intensive and does not scale. Combining these two frameworks seem to result in a powerful framework which generalizes well to new knowledge graphs, does inference and is able to scale to large entities.