Explainable GNN-Based Models over Knowledge Graphs

1 INTRODUCTION . Knowledge graphs ( KGs ) ( Hogan et al. , 2021 ) are graph-structured knowledge bases describing an application domain as a collection of entities and their relationships . They can be captured in formats such as the Resource Description Framework ( RDF ) , and are used in applications such as Web search , recommendation , fraud detection , and data integration . Many of these applications involve realising a graph transformation . For example , in a recommender system we can represent user-item interactions as a graph ( Aggarwal , 2016 ) and extend it with external KGs to improve accuracy , diversity , and interpretability of recommendations ( Zhang et al. , 2016 ; Wang et al. , 2018 ; 2019a ) ; the system then realises a transformation of the input graph into another graph containing recommended user-item interactions . Moreover , a KG completion system ( Rossi et al. , 2021 ; Bordes et al. , 2013 ; Teru et al. , 2020 ) transforms an input graph into another graph containing relationships missing in the input . Usually , the transformation is not given explicitly , but is learned from examples . Graph Neural Networks ( GNNs ) ( Scarselli et al. , 2009 ; Liu & Zhou , 2020 ) are one of the most popular ML models for graph data . They are usually applied to KGs in three stages ( Hamilton et al. , 2017 ) . First , the input KG is encoded into an embedding space as another graph whose vertices are labelled with numeric feature vectors . Second , this graph is processed by a GNN with several layers ; for each layer , the GNN updates the feature vectors based on the learned model parameters and the vectors of the previous layer . Finally , the vectors of the output layer are decoded to the output KG . While such approaches are effective in practice , GNNs compute predictions through numeric calculations in the embedding space , so the transformation can be difficult to understand . For example , it would be useful to know that a system recommended novel The Idiot because the user liked Crime and Punishment and both novels were written by Dostoevsky . There is growing interest in explaining GNN predictions , and many methods have focused on identifying relevant subgraphs in the embedding space ( Ying et al. , 2019 ; Luo et al. , 2020 ; Lin et al. , 2021 ) . Such methods , however , do not explain predictions symbolically—that is , by showing how each prediction can be derived from the input KG using rules of inference in a knowledge representation formalism . Datalog ( Abiteboul et al. , 1995 ; Ceri et al. , 1989 ) is a prominent formalism based on ‘ if–then ’ rules . When applied to facts , the rules derive new facts . For example , we can represent knowledge about literature using facts Author ( Dostoevsky , Crime and Punishment ) and Author ( Dostoevsky , The Idiot ) , and past interactions using fact Likes ( Crime and Punishment ) . Then , the application of rule Author ( x , y1 ) ∧ Author ( x , y2 ) ∧ Likes ( y1 ) → Recommend ( y2 ) to these facts derives the target fact Recommend ( The Idiot ) . State-of-the-art Datalog engines ( Motik et al. , 2014 ) provide powerful methods for generating human-readable proofs explaining why a target fact follows from the input rules and the data . Such proofs can help fostering trust in system predictions , ensuring compliance with norms , and enabling verification of fairness standards . The main contribution of this paper is a new family of GNN-based transformations of KGs that can be trained from examples as usual , but whose predictions can be explained symbolically using Datalog rules . Our transformation consists of the three steps mentioned earlier : encoding an input KG , applying a GNN , and decoding the result back into an output KG . These steps , however , were carefully crafted to ensure that the transformation can always be described symbolically . We use a novel class of monotonic GNNs ( MGNNs ) , which we designed to ensure an important property : when values in input feature vectors increase , no value of an output feature vector decreases . To capture this intuition formally , we show that our transformation is monotonic under homomorphisms—a well-known property that captures key aspects of Datalog rule application . In contrast to the related approaches ( Teru et al. , 2020 ; Hamaguchi et al. , 2017 ; Schlichtkrull et al. , 2018 ; Wang et al. , 2019b ) , all predictions of our transformation can be explained using Datalog rules . Specifically , for an arbitrary MGNN , there exists an equivalent set of rules so that , on any KG , the rules and the MGNN-based transformation produce the same facts . This opens the door to new ways of using MGNNs : when developing Datalog rules manually is difficult , we can train an MGNN model and use it to provide further predictions , but such predictions can be explained using rules . For example , we could train an MGNN on a graph containing examples of credit applications and then use it to support decision-making on new applications by applying the model to a different graph . Unlike existing approaches , there exists a set of rules implicitly captured by the model that fully explain each credit recommendation ; although computing all such rules can be computationally expensive , we can typically extract a representative subset of them in practice . Thus , we integrate ML and symbolic AI in a single framework , allowing applications to use the best of both worlds . We demonstrate the effectiveness of our approach by applying it to classification tasks in KG completion . Using well-known benchmarks , we compare our system ’ s performance with that of DRUM ( Sadeghian et al. , 2019 ) and AnyBURL ( Meilicke et al. , 2019 ) , both state-of-the-art KG completion systems able to also generate rules . We show that , in addition to providing an exact correspondence between the model and the extracted rules , our technique also offers competitive performance . Proofs are given in the appendix , and the source code is available as supplementary material . 2 AN MGNN-BASED TRANSFORMATION OF KNOWLEDGE GRAPHS . We now describe our GNN-based transformation of knowledge graphs . In this paper , we see a KG as a dataset , which is a finite set of facts of the form A ( a ) or R ( a1 , a2 ) , where A is a unary predicate , R is a binary predicate , and a , a1 , a2 are constants . Constants represent entities from an application domain , unary predicates typically represent entity types , and binary predicates represent relationships between entities . For example , the statements that a user likes Crime and Punishment and it is a novel written by Fyodor Dostoevsky are represented as facts Likes ( Crime and Punishment ) and Author ( Dostoevsky , Crime and Punishment ) . KGs in formats such as RDF can be transformed into such a dataset in the obvious way . For D a dataset , cn ( D ) is the set of constants occurring in D. Our transformation is realised by an operator TM on datasets , which is parameterised by a GNNM . A dataset D is first encoded into a graph GD in the embedding space , where vertices are labelled with numeric feature vectors ; then , GD is processed byM ; finally the result is decoded to the output dataset TM ( D ) . In Section 3 , we show that , by restrictingM to be a GNN of a special form , called monotonic , we can ensure that TM is equivalent to a round of Datalog rule application ; this , in turn , allows us to automatically extract an equivalent set of rules from TM . The embedding space consists of coloured graphs , which extend the standard notion of undirected graphs in two ways . First , each vertex is labelled by a numeric feature vector of some fixed dimension ; this is necessary in order to allow the coloured graph to be processed by M. Second , each edge in the graph is assigned a colour , which allows us to distinguish different types of connections in the graph . Figure 1 illustrates the encoding of the datasetD in the introduction . First , we introduce vertices vDs , vCP , and vI in GD to represent constants Dostoevsky , The Idiot , and Crime and Punishment . The feature vector of each vertex va represents all unary and binary facts of the form X ( a ) and Y ( a , a ) ; for example , vCP has 1 as the first element of its vector representing Likes ( Crime and Punishment ) . Second , we introduce vertices vDs , CP , vCP , Ds , vDs , I , and vI , Ds , whose vectors represent all binary facts over the corresponding constants ; for example , vDs , CP has 1 as the third vector element , representing Author ( Dostoevsky , Crime and Punishment ) . Since The Idiot and Crime and Punishment do not occur together in a fact in D , the encoding of D contains neither vCP , I nor vI , CP . Here , we rely on an order of predicates to determine the corresponding vector positions ; this order is arbitrary , but fixed in advance . Third , we connect all vertices of GD referring to common constants with coloured edges , which allows us to distinguish different types of joins between facts . For example , vDs and vDs , CP refer to Dostoevsky , so we connect them with an edge of colour c1 , thus indicating that Dostoevsky occurs first in the constant pair of vDs , CP . Analogously , we connect vDs , CP and vCP with an edge of colour c2 . The constant pair of vCP , Ds is the inverse of vDs , CP ’ s , so we connect them with an edge of colour c3 . Finally , we use edges of colour c4 to connect vertices of GD representing constants occurring together in a fact , such as vCP and vDs . Formally , we fix a set of four colours Col = { c1 , c2 , c3 , c4 } . We assume that datasets and derived facts will use finitely many predicates , and fix a list A1 , . . . , A , R +1 , . . . , Rδ of available unary and binary predicates . We consider vectors and matrices over the reals R and its nonnegative subset R+ . For v a vector , ( v ) i is the i-th element of v. We apply scalar functions to vectors and matrices element-wise ; for example , for v1 , . . . , vn vectors of equal dimension , max { v1 , . . . , vn } is the vector whose i-th element is max { ( v1 ) i , . . . , ( vn ) i } . Finally , a function σ : R→ R is monotonically increasing if x < y implies σ ( x ) ≤ σ ( y ) , and function σ is Boolean if it ranges over { 0 , 1 } . The following definition describes the structure of our embedding space . Definition 1 . A ( Col , δ ) -graph is a tuple 〈V , { Ec } c∈Col , λ〉 , where V is a finite set of vertices ; for each c ∈ Col , Ec is a set of c-coloured undirected edges ( possibly including self-loops ) over V ; and labelling λ assigns to each v ∈ V a vector λ ( v ) of dimension δ . If λ is implicitly understood , λ ( v ) is written as v. For c ∈ Col and v ∈ V , Nc ( v ) is the set of vertices connected to v by an edge in Ec . We are now ready to define our encoding of an arbitrary dataset D into a ( Col , δ ) -graph GD . Definition 2 . The encoding of a dataset D is the ( Col , δ ) -graph GD that contains a vertex va for each constant a ∈ cn ( D ) , and vertices va , b and vb , a for all ( not necessarily distinct ) constants a and b that occur together in a binary fact in D. For each vertex va , b , GD contains an edge of colour c1 connecting va , b and va , an edge of colour c2 connecting va , b and vb , an edge of colour c3 connecting va , b and vb , a , and an edge of colour c4 connecting va and vb . For each va , labelling ( va ) i is 1 if Ai ( a ) ∈ D and 0 otherwise . For each va , b , labelling ( va , b ) i is 1 if Ri ( a , b ) ∈ D and 0 otherwise . We process GD by a GNN . As we explain in more detail Section 3 , if we are to ensure that our transformation ‘ mimics ’ Datalog rule application , we can not use an arbitrary GNN . Instead , we introduce Monotonic Graph Neural Networks ( MGNNs ) , which restrict GNNs in a way that allows us to attain our objective . Definition 3 also specifies how to apply an MGNN to a coloured graph . Definition 3 . A ( Col , δ ) -monotonic graph neural network ( MGNN ) M with L ≥ 1 layers is a tuple 〈 { A ` } 1≤ ` ≤L , { B ` c } c∈Col,1≤ ` ≤L , { b ` } 1≤ ` ≤L , σ , cls〉 . For each ` ∈ { 1 , . . . , L } and each c ∈ Col , A ` and B ` c are matrices over R+ of dimension m ` × n ` , and b ` is a vector over R of dimension n ` such that n1 = mL = δ , and m ` −1 = n ` for each ` ∈ { 2 , . . . , L } . Moreover , σ : R→ R+ is a monotonically increasing activation function , and cls : R+ → { 0 , 1 } is a monotonically increasing classification function . The dimension ofM is max { n1 , . . . , nL } . ApplyingM to a ( Col , δ ) -graph G = 〈V , { Ec } c∈Col , λ〉 induces the sequence λ0 , . . . , λL of vertex labellings where λ0 = λ , and for each ` ∈ { 1 , . . . , L } and v ∈ V , the value of λ ` ( v ) is given by v ` = σ ( A ` v ` −1 + ∑ c∈Col B ` cmax { w ` −1 | w ∈ Nc ( v ) } + b ` ) . ( 1 ) The resultM ( G ) of applyingM to G is the ( Col , δ ) -graph with the same vertices and edges as G , but where each vertex v ∈ V is labelled by cls ( λL ( v ) ) . Intuitively , an MGNN consists of L layers , where each layer ` ∈ { 1 , . . . , L } is defined by a matrix A ` , a matrix B ` c for each colour c , and a bias vector b ` ; these constitute the learnable parameters of the model . In addition , an MGNN also uses functions σ and cls which are fixed and given explicitly . An MGNN transforms an arbitrary ( Col , δ ) -graph G into another ( Col , δ ) -graphM ( G ) as follows . Each vertex v in G and each ` ∈ { 0 , . . . , L } is associated with a labelling v ` . Labelling v0 is the labelling v of v in G , and , for each layer ` , vertex v is labelled by a feature vector v ` computed as specified in equation ( 1 ) . The feature vector of v in the resulting graph is computed by applying the classification function cls to the final labelling vL . Thus , MGNNs instantiate the GNN framework by ( i ) taking the maximum of the features of a vertex ’ neighbours ( instead of the more commonly used sum or average ) , ( ii ) using nonnegative weights in matrices A ` and B ` c ( but the elements of bias vectors b ` can be negative ) , and ( iii ) requiring the activation and classification functions to be monotonically increasing . We discuss the reasons for these restrictions in Section 3 . The resultM ( GD ) of applying an MGNN to the encoding GD of a dataset D can be decoded into the output dataset of the transformation by essentially inverting the encoder ( see Figure 1 for an example ) . Thus , for each MGNNM , we obtain a distinct operator TM that is specified as follows . Definition 4 . For M a ( Col , δ ) -MGNN and D a dataset , TM ( D ) is the dataset containing the fact Ai ( a ) for each i ∈ { 1 , . . . , } and each vertex va such that ( va ) i = 1 inM ( GD ) , and the fact Ri ( a , b ) for each i ∈ { + 1 , . . . , δ } and each vertex va , b such that ( va , b ) i = 1 inM ( GD ) . Note that the dimensions of the matrices in M do not predetermine the kind of dataset to which TM can be applied ; in fact , M can be trained on some dataset and then applied to a different dataset over the same predicates . This makes our transformation ideally suited for inductive settings in applications such as KG completion and recommendation ( Teru et al. , 2020 ; Hamaguchi et al. , 2017 ; van den Berg et al. , 2017 ) , where the datasets to which a trained model is applied may use constants ( i.e. , entities ) not mentioned in the datasets on which the model was trained .

Summary: This paper proposes a new family of GNN-based transformations for knowledge graphs, which is called monotonic GNNs (MGNNs). The key characteristic of MGNNs is that we can produce an equivalent set of rules for an arbitrary MGNN, and the rules and the MGNN-based transformation can produce the same facts. In other words, the rules can be used to explain how MGNNs make predictions. Moreover, the authors provide dedicated proofs for their formal statements. The experimental results on classification tasks for knowledge graph completion demonstrate the effectiveness of the approach.