SAFRAN: An interpretable, rule-based link prediction method outperforming embedding models

1 . Introduction . Link prediction in knowledge graphs is a vibrant area of research and development and encompasses a wide range of different approaches . Currently , most state-of-the-art approaches use low-dimensional representations ( embeddings ) of knowledge graphs to predict new knowledge . However , as such approaches are black boxes , the validity of their predictions can be questioned due to lack of interpretability . Recent research has focused on learning symbolic rules from knowledge graphs that try to detect correlations of entities in knowledge bases and express them as first-order Horn rules . In contrast to neural models , such rule-based methods are fully transparent and predictions can be explained intuitively . However , while current research on rule-based approaches puts a lot of emphasis on designing rule-mining algorithms , research on methods for applying learned rules and aggregating their predictions in a meaningful way is comparatively neglected . AnyBURL [ Meilicke et al. , 2019 , 2020 ] is currently one of the best performing symbolic rule learners and demonstrated results competitive with embedding methods on link prediction benchmarks such as FB15k-237 . However , the application of rules generated by AnyBURL is currently limited by functional redundancies , making it difficult to meaningfully aggregate predictions made by different rules and thereby limiting predictive performance . In this paper , we introduce the high-performance rule application framework SAFRAN1 ( ’ Scalable And fast non-redundant rule application ’ ) that employs a novel aggregation approach . We propose an algorithm called Non-redundant Noisy-OR that detects redundant rules prior to aggregation , mitigating their negative effects and improving predictive performance . This algorithm for aggregating predictions is not necessarily restricted to rules learned by AnyBURL and could also be applied to other rule-based systems . 2 . Preliminaries . 2.1 Link prediction . Knowledge graphs contain facts , such as “ Amsterdam is the capital of the Netherlands ” or “ Marie Curie was born in Warsaw ” , which are represented as semantic triples containing a subject ( head h ) , predicate ( relation r ) and object ( tail t ) . Formally , a knowledge graph is a directed heterogeneous multigraph G = ( E , R , T ) where E and R are the set of entities and relations respectively and T is a set of triples { ( h , r , t ) } ⊆ E ×R×E . The task of link prediction is to predict either the tail of a given head and relation ( h , r , ? ) or the head of a given tail and relation ( ? , r , t ) such that the resulting triple describes a correct fact which has not been in G before . 2.2 AnyBURL . We now give an overview about the main AnyBURL algorithm , which is explained in detail in [ Meilicke et al. , 2019 ] and [ Meilicke et al. , 2020 ] . AnyBURL ( Anytime Bottom Up Rule Learning ) is a novel walk-based method and is currently one of the best symbolic rule learners , competing with current state-of-the-art embedding approaches [ Rossi et al. , 2021 ] . Walk-based ( bottom-up ) methods are based on the idea that sampled paths from a knowledge graph ( random walks ) are examples of very specific rules , which can be transformed into more general rules . Learned rules can then be applied to the knowledge graph for the prediction of novel links between entities . AnyBURL learns rules in the form of first-order logic Horn rules of length n. In each iteration of the algorithm for mining rules the algorithm samples a random triple h ( c0 , c1 ) , f.e . lives ( max , uk ) from the training set . Starting from either the head or the tail of this triple , AnyBURL performs a random walk of length n , resulting in a ground 1 . Code can be found at https : //github.com/OpenBioLink/SAFRAN path in the form of h ( c0 , c1 ) ← b1 ( c1 , c2 ) , . . . , bn ( cn , cn+1 ) , f.e . speaks ( max , english ) ← lives ( max , uk ) , lang ( uk , english ) . The sampled ground path is subsequently generalized into three predefined types of rules shown in Table 1 . Cyclic rules can be generalized from cyclic paths ( c0 = cn+1 ) , while AC2 rules can be generalized from acyclic paths ( c0 6= cn+1 ) . AC1 rules are hybrid , as they can be both generalized from cyclic ( if c0 = cn+1 ) and acyclic ( if c0 6= cn+1 ) paths . All rules that can be generalized from the previous example ground path can be seen in Table 2 . The following applies for inferring a set of head triples Ĥr from the body of a rule r : Ĥr = { h ( ey , ex ) |ex , ey ∈ E ∧ ∃e2 , ... , en b1 ( ex , e2 ) , . . . , bn ( en , ey ) ∈ T } , if type = C { h ( c0 , ex ) |ex ∈ E ∧ ∃e2 , ... , en b1 ( ex , e2 ) , . . . , bn ( en , cn+1 ) ∈ T } , if type = AC1 { h ( c0 , ex ) |ex ∈ E ∧ ∃e2 , ... , en+1 b1 ( ex , e2 ) , . . . , bn ( en , en+1 ) ∈ T } , if type = AC2 AnyBURL calculates the confidence of a rule as the size of the union of inferred triples by a rule and the triples of the training set divided by the number of inferred triples conf ( r ) = |Ĥr ∩ T |/|Ĥr| , where T is the set of triples contained in the training set . 2.3 Aggregation . Rule-based link prediction methods , such as AnyBURL , produce many rules with different confidences that are subsequently applied to the knowledge graph in order to generate candidates for a link prediction task . Upon a link prediction task p ( s , ? ) , rule r generates the set of predictions { y|p ( s , y ) ∈ Ĥr } that could act as substitutions for the question mark . Rules can either predict zero , one or multiple entities . As the same entities can be proposed by multiple rules which may differ in confidence , an aggregation of these confidences is required to assess the final confidence of a prediction . Table 3 shows the confidences and results of five fictional rules . One aggregation approach called Maximum aggregation is using the maximum of these confidences [ Meilicke et al. , 2019 ] . The score of an entity score ( e ) = max { conf ( r1 ) , . . . , conf ( rk ) } is the maximum confidence of all rules r1 , . . . , rk that predict entity e. If the maximum confidences of two or more entities are the same , these entities are further ranked by their second best confidence and so on , until all top-k candidates can be distinguished or all rules are processed . This approach results in the following list of ranked candidates when applied to the solutions of Table 3 : rankingmax = 〈 ( b , 0.9 ) , ( a , 0.9 ) , ( c , 0.8 ) 〉 . As the second best rule that proposes b has a higher confidence than the second best rule which proposes a , b is ranked before a . Unfortunately , the Maximum aggregation is constrained to relatively simple predictions : Each prediction is primarily informed by only a single rule with the highest confidence ; it is not possible to make predictions based on a weighted combination of different rules . These limitations of maximum aggregation are potentially addressed by noisy-or aggregation , first proposed in this context by [ Galárraga et al. , 2015 ] . Instead of taking the maximum of confidences , the probability that at least one of the rules proposed the correct candidate is used for generating a list of ranked candidates . The score of an entity e can thus be calculated as score ( e ) = 1− k∏ i=1 ( 1− conf ( ri ) ) where r1 , . . . , rk are rules that predict entity e. Applying the Noisy-OR approach to the results of Table 3 , the entities are ranked as follows : rankingnoisy = 〈 ( c , 0.94 ) , ( b , 0.93 ) , ( a , 0.9 ) 〉 . This ranking differs from the maximum approach . As c is proposed by the second and third most confident rules , its aggregated confidence is higher than a or b , entities proposed by the most confident rule . In theory , Noisy-OR allows for more sophisticated confidence aggregation . However , it assumes independence/non-redundancy of all rules . As most rules that can be derived from real-world knowledge bases are in some way redundant with other rules , Noisy-OR was found to perform worse than maximum aggregation in practice . 2.4 The detrimental effect of redundancy on Noisy-OR aggregation . Redundancies in rules can lead to overestimation of confidences of predicted entities when aggregating using Noisy-OR . As two redundant rules generate the same predictions for the same reasons , the confidences of entities predicted by both rules are overestimated due to double counting . Consider the rules shown in Table 4 where this problem becomes apparent . If the training set contains the ground path lives ( john , uk ) , lang ( uk , english ) , each rule would predict speaks ( john , english ) upon the prediction task speaks ( john , ? ) for the same reasons . Aggregating the confidences of all three rules using Noisy-OR would result in an overestimated confidence of 0.988 for the entity english . While these redundancies are apparent , redundancies between rules are often less obvious in practice . Redundancy between two rules can arise from special relationships between relations , such as relations that are nearly equivalent , symmetric relations or relations entailing other relations . Consider the rules shown in Table 5 . The first example shows two rules being redundant based on entailment , as every capital of a country is a city within that country . The second example shows redundancy based on the symmetric relationship married . Both rules generate predictions for languages spoken by a person , based on what language its spouse speaks . The third example shows that such relationships are not restricted to single relations , but can arise from a combination of relations , as the combination parent → brother is a duplicate of the relation uncle . It is important to understand that there is usually no explicit information available about the symmetry of married nor do we know that capital is more specific than city . It is not even possible to grasp this information from the dataset itself as it might be incomplete and noisy . This makes it impossible to decide redundancy via a procedure that makes use of relevant background knowledge . 3 . Approach . 3.1 Algorithm . To overcome the problem of redundancy when using the Noisy-OR aggregation method , we propose an approach to cluster rules based on their redundancy degree prior to aggregation . Predictions of rules in a cluster are aggregated using the Maximum approach , as this approach is not susceptible to redundancies . Predictions of the different clusters are then further aggregated using the Noisy-OR approach . We call the resulting aggregation technique Non-redundant Noisy-OR . A pseudocode of Non-redundant Noisy-OR can be seen in Appendix G. As a metric for redundancy between two rules ri , rj the Jaccard Index sim ( ri , rj ) = |Ĥri ∩ Ĥrj |/|Ĥri ∪ Ĥrj | of the sets of inferred triples is used . As the calculation of the Jaccard coefficient is very inefficient for large sets , the Jaccard coefficient is estimated using the MinHash scheme [ Broder , 1997 ] , which makes time complexity linear and memory usage constant . Naturally rules are pre-clustered by their relation of the head atom , as two rules having a different relation in the head are never used in the same prediction task and the union of their inferred triples is always empty hri 6= hrj → Ĥri ∩ Ĥrj = ∅ . Thus , similarity matrices of rules are calculated for each head relation and used to cluster the rules . Two rules ri , rj are considered redundant and assigned to the same cluster if sim ( ri , rj ) > t , where t is a threshold . However a global threshold t is not capable of defining the cutoff point for redundancy for all relations , rule types and directions of prediction ( prediction of head entities or tail entities ) . As the sets of rules that predict for different relations may vary in distribution of type , length or specificity , the threshold that generates the optimal clustering for a relation may not generate the optimal clustering for a different relation . Furthermore as relations may vary in multiplicity the direction of the prediction has to taken into account . Therefore , a distinct threshold for the prediction of heads and prediction of tails for each relation is required in order to be able to optimally cluster an entire rule set . Furthermore , within a set of rules that predict for the same relation , optimal thresholds may vary between types of rules . As can be seen by the examples in Table 4 , the type combination of C and AC1 often tends to form set-subset relation . The Jaccard Index is very insensitve to such relationships . While e.g . a Jaccard index of 0.25 between a rule of type C and a rule of type AC1 may indicate a high redundancy due to the dominance of set-subset relationships within rules of this type combination , it may be considered as a fairly low evidence level for a redundancy between two rules of type C , where such relationships are not as prominent . Thus a differentiation of rule types is needed when defining the cutoff value for redundancy between two rules of a certain relation . For each combination of rule types an independent threshold parameter is used to determine redundancy . Conclusively , the threshold of similarity for rules that predict entities of direction d ( head or tail ) of a h-triple ( we use h to refer to a relation , while r is used to refer to rules ) is given by thd = [ tC/C , tC/AC1 , tC/AC2 , tAC1/AC2 , tAC1/AC1 , tAC2/AC2 ] . Note that if thd = [ 0 , 0 , 0 , 0 , 0 , 0 ] only the Maximum aggregation is used and if thd = [ 1 , 1 , 1 , 1 , 1 , 1 ] only Noisy-OR . An optimal clustering of rules that predict h-triples is given by the type combination thresholds thd that maximize the fitness of the clusters for predicting new links . This fitness is evaluated on the validation set using the mean-reciprocal rank based on the topk predicted entities . SAFRAN uses one of two search strategies for finding the optimal thresholds : grid search ( parameter sweep ) and random search . For grid search the range of possible thresholds [ 0.0 , 1.0 ] is divided by n equally distant steps and each threshold is subsequently used for clustering . For the grid search , we do not distinguish between rule-type specific parameters but use a single relation-specific fix parameter thd to limit the search space . Contrary to this , the random search randomly samples the six rule-typespecific thresholds of thd .

Summary: This paper studies the problem of link prediction for knowledge graph completion with rule-based methods. Specifically, it builds on a recent strong rule-based method, AnyBURL, for mining rules from a KG, and mainly studies the application of the mined rules. Recognizing the limitations of the Maximum and Noisy-OR rule aggregation strategies used in prior work, this paper proposes SAFRAN, a rule application strategy that clusters rules of the same relation based on their similarity with the hope that redundant rules will be clustered together. The Maximum strategy is then used within each cluster and the Noisy-OR strategy is used to aggregate clusters. Empirical results on FB15K-237, WN18RR, and YAGO3-10 are strong, outperforming embedding-based methods as well as the Maximum and Noisy-OR baselines.