SAFRAN: An interpretable, rule-based link prediction method outperforming embedding models

1 . Introduction . Link prediction in knowledge graphs is a vibrant area of research and development and encompasses a wide range of different approaches . Currently , most state-of-the-art approaches use low-dimensional representations ( embeddings ) of knowledge graphs to predict new knowledge . However , as such approaches are black boxes , the validity of their predictions can be questioned due to lack of interpretability . Recent research has focused on learning symbolic rules from knowledge graphs that try to detect correlations of entities in knowledge bases and express them as first-order Horn rules . In contrast to neural models , such rule-based methods are fully transparent and predictions can be explained intuitively . However , while current research on rule-based approaches puts a lot of emphasis on designing rule-mining algorithms , research on methods for applying learned rules and aggregating their predictions in a meaningful way is comparatively neglected . AnyBURL [ Meilicke et al. , 2019 , 2020 ] is currently one of the best performing symbolic rule learners and demonstrated results competitive with embedding methods on link prediction benchmarks such as FB15k-237 . However , the application of rules generated by AnyBURL is currently limited by functional redundancies , making it difficult to meaningfully aggregate predictions made by different rules and thereby limiting predictive performance . In this paper , we introduce the high-performance rule application framework SAFRAN1 ( ’ Scalable And fast non-redundant rule application ’ ) that employs a novel aggregation approach . We propose an algorithm called Non-redundant Noisy-OR that detects redundant rules prior to aggregation , mitigating their negative effects and improving predictive performance . This algorithm for aggregating predictions is not necessarily restricted to rules learned by AnyBURL and could also be applied to other rule-based systems . 2 . Preliminaries . 2.1 Link prediction . Knowledge graphs contain facts , such as “ Amsterdam is the capital of the Netherlands ” or “ Marie Curie was born in Warsaw ” , which are represented as semantic triples containing a subject ( head h ) , predicate ( relation r ) and object ( tail t ) . Formally , a knowledge graph is a directed heterogeneous multigraph G = ( E , R , T ) where E and R are the set of entities and relations respectively and T is a set of triples { ( h , r , t ) } ⊆ E ×R×E . The task of link prediction is to predict either the tail of a given head and relation ( h , r , ? ) or the head of a given tail and relation ( ? , r , t ) such that the resulting triple describes a correct fact which has not been in G before . 2.2 AnyBURL . We now give an overview about the main AnyBURL algorithm , which is explained in detail in [ Meilicke et al. , 2019 ] and [ Meilicke et al. , 2020 ] . AnyBURL ( Anytime Bottom Up Rule Learning ) is a novel walk-based method and is currently one of the best symbolic rule learners , competing with current state-of-the-art embedding approaches [ Rossi et al. , 2021 ] . Walk-based ( bottom-up ) methods are based on the idea that sampled paths from a knowledge graph ( random walks ) are examples of very specific rules , which can be transformed into more general rules . Learned rules can then be applied to the knowledge graph for the prediction of novel links between entities . AnyBURL learns rules in the form of first-order logic Horn rules of length n. In each iteration of the algorithm for mining rules the algorithm samples a random triple h ( c0 , c1 ) , f.e . lives ( max , uk ) from the training set . Starting from either the head or the tail of this triple , AnyBURL performs a random walk of length n , resulting in a ground 1 . Code can be found at https : //github.com/OpenBioLink/SAFRAN path in the form of h ( c0 , c1 ) ← b1 ( c1 , c2 ) , . . . , bn ( cn , cn+1 ) , f.e . speaks ( max , english ) ← lives ( max , uk ) , lang ( uk , english ) . The sampled ground path is subsequently generalized into three predefined types of rules shown in Table 1 . Cyclic rules can be generalized from cyclic paths ( c0 = cn+1 ) , while AC2 rules can be generalized from acyclic paths ( c0 6= cn+1 ) . AC1 rules are hybrid , as they can be both generalized from cyclic ( if c0 = cn+1 ) and acyclic ( if c0 6= cn+1 ) paths . All rules that can be generalized from the previous example ground path can be seen in Table 2 . The following applies for inferring a set of head triples Ĥr from the body of a rule r : Ĥr = { h ( ey , ex ) |ex , ey ∈ E ∧ ∃e2 , ... , en b1 ( ex , e2 ) , . . . , bn ( en , ey ) ∈ T } , if type = C { h ( c0 , ex ) |ex ∈ E ∧ ∃e2 , ... , en b1 ( ex , e2 ) , . . . , bn ( en , cn+1 ) ∈ T } , if type = AC1 { h ( c0 , ex ) |ex ∈ E ∧ ∃e2 , ... , en+1 b1 ( ex , e2 ) , . . . , bn ( en , en+1 ) ∈ T } , if type = AC2 AnyBURL calculates the confidence of a rule as the size of the union of inferred triples by a rule and the triples of the training set divided by the number of inferred triples conf ( r ) = |Ĥr ∩ T |/|Ĥr| , where T is the set of triples contained in the training set . 2.3 Aggregation . Rule-based link prediction methods , such as AnyBURL , produce many rules with different confidences that are subsequently applied to the knowledge graph in order to generate candidates for a link prediction task . Upon a link prediction task p ( s , ? ) , rule r generates the set of predictions { y|p ( s , y ) ∈ Ĥr } that could act as substitutions for the question mark . Rules can either predict zero , one or multiple entities . As the same entities can be proposed by multiple rules which may differ in confidence , an aggregation of these confidences is required to assess the final confidence of a prediction . Table 3 shows the confidences and results of five fictional rules . One aggregation approach called Maximum aggregation is using the maximum of these confidences [ Meilicke et al. , 2019 ] . The score of an entity score ( e ) = max { conf ( r1 ) , . . . , conf ( rk ) } is the maximum confidence of all rules r1 , . . . , rk that predict entity e. If the maximum confidences of two or more entities are the same , these entities are further ranked by their second best confidence and so on , until all top-k candidates can be distinguished or all rules are processed . This approach results in the following list of ranked candidates when applied to the solutions of Table 3 : rankingmax = 〈 ( b , 0.9 ) , ( a , 0.9 ) , ( c , 0.8 ) 〉 . As the second best rule that proposes b has a higher confidence than the second best rule which proposes a , b is ranked before a . Unfortunately , the Maximum aggregation is constrained to relatively simple predictions : Each prediction is primarily informed by only a single rule with the highest confidence ; it is not possible to make predictions based on a weighted combination of different rules . These limitations of maximum aggregation are potentially addressed by noisy-or aggregation , first proposed in this context by [ Galárraga et al. , 2015 ] . Instead of taking the maximum of confidences , the probability that at least one of the rules proposed the correct candidate is used for generating a list of ranked candidates . The score of an entity e can thus be calculated as score ( e ) = 1− k∏ i=1 ( 1− conf ( ri ) ) where r1 , . . . , rk are rules that predict entity e. Applying the Noisy-OR approach to the results of Table 3 , the entities are ranked as follows : rankingnoisy = 〈 ( c , 0.94 ) , ( b , 0.93 ) , ( a , 0.9 ) 〉 . This ranking differs from the maximum approach . As c is proposed by the second and third most confident rules , its aggregated confidence is higher than a or b , entities proposed by the most confident rule . In theory , Noisy-OR allows for more sophisticated confidence aggregation . However , it assumes independence/non-redundancy of all rules . As most rules that can be derived from real-world knowledge bases are in some way redundant with other rules , Noisy-OR was found to perform worse than maximum aggregation in practice . 2.4 The detrimental effect of redundancy on Noisy-OR aggregation . Redundancies in rules can lead to overestimation of confidences of predicted entities when aggregating using Noisy-OR . As two redundant rules generate the same predictions for the same reasons , the confidences of entities predicted by both rules are overestimated due to double counting . Consider the rules shown in Table 4 where this problem becomes apparent . If the training set contains the ground path lives ( john , uk ) , lang ( uk , english ) , each rule would predict speaks ( john , english ) upon the prediction task speaks ( john , ? ) for the same reasons . Aggregating the confidences of all three rules using Noisy-OR would result in an overestimated confidence of 0.988 for the entity english . While these redundancies are apparent , redundancies between rules are often less obvious in practice . Redundancy between two rules can arise from special relationships between relations , such as relations that are nearly equivalent , symmetric relations or relations entailing other relations . Consider the rules shown in Table 5 . The first example shows two rules being redundant based on entailment , as every capital of a country is a city within that country . The second example shows redundancy based on the symmetric relationship married . Both rules generate predictions for languages spoken by a person , based on what language its spouse speaks . The third example shows that such relationships are not restricted to single relations , but can arise from a combination of relations , as the combination parent → brother is a duplicate of the relation uncle . It is important to understand that there is usually no explicit information available about the symmetry of married nor do we know that capital is more specific than city . It is not even possible to grasp this information from the dataset itself as it might be incomplete and noisy . This makes it impossible to decide redundancy via a procedure that makes use of relevant background knowledge . 3 . Approach . 3.1 Algorithm . To overcome the problem of redundancy when using the Noisy-OR aggregation method , we propose an approach to cluster rules based on their redundancy degree prior to aggregation . Predictions of rules in a cluster are aggregated using the Maximum approach , as this approach is not susceptible to redundancies . Predictions of the different clusters are then further aggregated using the Noisy-OR approach . We call the resulting aggregation technique Non-redundant Noisy-OR . A pseudocode of Non-redundant Noisy-OR can be seen in Appendix G. As a metric for redundancy between two rules ri , rj the Jaccard Index sim ( ri , rj ) = |Ĥri ∩ Ĥrj |/|Ĥri ∪ Ĥrj | of the sets of inferred triples is used . As the calculation of the Jaccard coefficient is very inefficient for large sets , the Jaccard coefficient is estimated using the MinHash scheme [ Broder , 1997 ] , which makes time complexity linear and memory usage constant . Naturally rules are pre-clustered by their relation of the head atom , as two rules having a different relation in the head are never used in the same prediction task and the union of their inferred triples is always empty hri 6= hrj → Ĥri ∩ Ĥrj = ∅ . Thus , similarity matrices of rules are calculated for each head relation and used to cluster the rules . Two rules ri , rj are considered redundant and assigned to the same cluster if sim ( ri , rj ) > t , where t is a threshold . However a global threshold t is not capable of defining the cutoff point for redundancy for all relations , rule types and directions of prediction ( prediction of head entities or tail entities ) . As the sets of rules that predict for different relations may vary in distribution of type , length or specificity , the threshold that generates the optimal clustering for a relation may not generate the optimal clustering for a different relation . Furthermore as relations may vary in multiplicity the direction of the prediction has to taken into account . Therefore , a distinct threshold for the prediction of heads and prediction of tails for each relation is required in order to be able to optimally cluster an entire rule set . Furthermore , within a set of rules that predict for the same relation , optimal thresholds may vary between types of rules . As can be seen by the examples in Table 4 , the type combination of C and AC1 often tends to form set-subset relation . The Jaccard Index is very insensitve to such relationships . While e.g . a Jaccard index of 0.25 between a rule of type C and a rule of type AC1 may indicate a high redundancy due to the dominance of set-subset relationships within rules of this type combination , it may be considered as a fairly low evidence level for a redundancy between two rules of type C , where such relationships are not as prominent . Thus a differentiation of rule types is needed when defining the cutoff value for redundancy between two rules of a certain relation . For each combination of rule types an independent threshold parameter is used to determine redundancy . Conclusively , the threshold of similarity for rules that predict entities of direction d ( head or tail ) of a h-triple ( we use h to refer to a relation , while r is used to refer to rules ) is given by thd = [ tC/C , tC/AC1 , tC/AC2 , tAC1/AC2 , tAC1/AC1 , tAC2/AC2 ] . Note that if thd = [ 0 , 0 , 0 , 0 , 0 , 0 ] only the Maximum aggregation is used and if thd = [ 1 , 1 , 1 , 1 , 1 , 1 ] only Noisy-OR . An optimal clustering of rules that predict h-triples is given by the type combination thresholds thd that maximize the fitness of the clusters for predicting new links . This fitness is evaluated on the validation set using the mean-reciprocal rank based on the topk predicted entities . SAFRAN uses one of two search strategies for finding the optimal thresholds : grid search ( parameter sweep ) and random search . For grid search the range of possible thresholds [ 0.0 , 1.0 ] is divided by n equally distant steps and each threshold is subsequently used for clustering . For the grid search , we do not distinguish between rule-type specific parameters but use a single relation-specific fix parameter thd to limit the search space . Contrary to this , the random search randomly samples the six rule-typespecific thresholds of thd .

Summary: The paper introduces SAFRAN rule application framework to improve upon the existing AnyBURL interpretable rule-based link prediction approach. SAFRAN helps to solve the redundancies that affect the aggregation step in AnyBURL and introduces Non-redundant Noisy-OR (NRNO) to detects and clusters redundant rules before the aggregation step. In several link prediction benchmarks, SAFRAN achieves SOTA performances for rule-based algorithms and in two out of three datasets even beats multiple embedding-based algorithms with good explainability.