Why a Naive Way to Combine Symbolic and Latent Knowledge Base Completion Works Surprisingly Well

1 . Introduction and Motivation . In this paper we compare knowledge graph completion ( KGC ) methods , that are based on non-symbolic representations in terms of embeddings , against a symbolic approach that is based on rules . Given a knowledge graph G , which is a set of triples , a KGC task is to predict the question mark in the incomplete triple r ( e , ? ) or r ( ? , e ) , where r is a relation ( binary predicate ) used in G and e is an entity described by ( usually ) several triples in G. KGC methods solve this task by generating a ranking of possible candidates , which are evaluated by metrics that focus on the rank of the correct prediction within this ranking . A first famous model for solving the KGC problem via a representation in an embeddings space is known as TransE [ Bordes et al. , 2013 ] . The basic idea of such a model , termed knowledge graph embedding ( KGE ) model in the following , is to first randomly map the entities and relations from G to a multidimensional space , e.g. , to IRn . Then the triples from G are used to set up a large optimization problem that is solved via gradient descent or a similar algorithm . How to transfer a triple into a part of the objective function is usually the defining characteristic of the model . By doing this , the final embedding becomes an alternative representation of the knowledge graph and it can be used to answer the completion task . Contrary to this , a rule-based approach searches for regularities in the graph and expresses them usually in the form of horn clauses . These rules are then used to propose candidates for solving the completion task . Within this paper we try to understand the essential differences between both families of approaches . We first tackle this problem by analysing the rankings for several concrete completion tasks . This analysis results into several insights that motivates a specific method to combine both types of approaches . This aggregation method improves each of the incoming rankings , no matter whether rule or embeddings-based input has been better . As an additional benefit our method does not loose the explanatory power of a rule-based approach but integrates it into the combined approach . 2 . Related Work . In [ Zhang et al. , 2021 ] the authors are concerned with different KGC methods with a focus on symbolic reasoning , neural reasoning techniques , and everything in between . However , they discuss differences from a rather abstract level and try to give a comprehensive overview without any connection to their empirical analysis . Symbolic approaches and latent approaches that are based on a projection to an embedding space have already been compared in an experimental study in [ Meilicke et al. , 2018 ] , where the authors identify types of completion tasks for which there are differences between these approaches . However , within the last three years significant improvements have been made . For example , in the 2018 publication the authors reported about top hits @ 10 scores for the FB237 dataset in the range 0.4 to 0.43 , while in [ Ruffinelli et al. , 2020 ] and [ Rossi et al. , 2021 ] scores between 0.5 and 0.55 are reported . This makes insights that follow from these experiments to some degree unreliable , as it might be the case many flaws are meanwhile fixed . Note also that our main interest is not to find types of completion tasks where rules are superior or vice versa but to understand at least partially why this is the case . In [ Rossi et al. , 2021 ] the authors compare a large set of embeddings based methods and included the rule-based approach AnyBURL [ Meilicke et al. , 2019 ] as a baseline . This paper describes an extensive experimental study and comes to the conclusion that AnyBURL is an efficient alternative to non-symbolic methods . To our best knowledge , AnyBURL is the only symbolic approach that has proven to achieve performances on par with the current KGE state-of-the-art and therefore we choose AnyBURL as the representative for rule-based approaches within our experiments . In the second half of this paper we combine the results of rule and embedding based approaches . The core principle of our strategy , combining multiple models into an ensemble , has been studied in [ Wang et al. , 2018 ] and in [ Meilicke et al. , 2018 ] . Although the approach in this paper can be seen as an extension , it differs fundamentally from a typical ensemble . As we discuss in Section 4 , we exclusively allow for candidates which are within the language bias of the rule based approach by only considering its top-k candidates . Our combination strategy ensures that the rule based method and the KGE model operate independently and are able to focus on their strengths before the results are aggregated to achieve an overall improvement . There are several papers that propose , on the contrary , to tightly integrate rule learning and embedding based approaches . Examples can be found in KALE [ Guo et al. , 2016 ] and RUGE [ Guo et al. , 2018 ] .1 However , approaches that belong to this category are mostly restricted to a type of rule which does not allow for constants . More recently differentiable rule learning has been proposed [ Rocktäschel and Riedel , 2017 , Yang et al. , 2017 , Sadeghian et al. , 2019 , Minervini et al. , 2020 ] . These approaches are not the focus of this work . Nevertheless , the aforementioned papers either have not been applied to the common KGC datasets or the results in this paper are better . 1 . A comparison with these models is possible via the performance achieved on the FB15K dataset . Here the base version of AnyBURL alone performs significantly better . 3 . Rule-Based Knowledge Base Completion with AnyBURL . We abstain from a detailed description of the AnyBURL algorithm and refer the reader to [ Meilicke et al. , 2019 , 2020 ] . Nevertheless , for the purpose of this work it is important to know what kind of rules AnyBURL learns and how they are applied to create the final rankings . 3.1 Language Bias . We give concrete examples for the most important rule types within this section . There are two additional rule types that are mainly used for filling up the rankings which are explained in the Appendix in Section C. Each of the supported rules is a horn rule and can be written as h← b . We call h the head of the rule and b , which is a conjunction of atoms , the body of the rule . The first and probably most prominent type of rules are called closed connected rules in [ Galárraga et al. , 2013 ] or cyclic rules according to [ Meilicke et al. , 2019 ] . The attribute cyclic refers to the head variables X and Y being directly connected in the head of the rule , while there is an alternative path expressed in the body of the rule . These rules do not contain constants but variables only . Here are some examples . hypernym ( X , Y ) ← hyponym ( Y , X ) ( 1 ) contains ( X , Y ) ← contains ( X , A ) , adjoins ( A , Y ) ( 2 ) contains ( X , Y ) ← administrative parent ( A , X ) , adjoins ( A , B ) , capital ( B , Y ) ( 3 ) The first rule expresses that the hypernym relation is inverse to the hyponym relation . Rule 2 expresses the transitivity of the contains relation . Rule 3 is a rather complex rule that expresses roughly that X contains Y if X is an administrative parent location of A , A shares a border with B and B has capital Y . Note that this rule is an example of a rule that creates highly ranked correct answers in our experiments . Another important type of rules are acyclic rules with only one variable , that have a constant in the head and a ( in most cases different ) constant in the body . AnyBURL is restricted in its default setting , which we used in our experiments , to learn rules of this type with only one body atom . Here are two examples . citizen ( X , UK ) ← bornIn ( X , London ) ( 4 ) contains ( Australia , Y ) ← contains ( V ictoria , Y ) ( 5 ) Rule 4 expresses that someone born in London is ( probably ) a citizen of the UK . Rule 5 says that locations contained in Victoria are also contained in Australia . The vast majority of the mined rules are longer cyclic rules ( as Rule 3 ) and acyclic rules similar to the ones that we just presented . A regularity that can not be expressed in terms of these rule types is completely invisible to AnyBURL . 3.2 Applying Rules . To compute a prediction for a given completion task r ( a , ? ) AnyBURL applies all rules that might create a triple r ( a , c ) where c is the predicted candidate . This is done by grounding the relevant rules against the training set , i.e. , by replacing the variables with constants ( in other words : entities ) such that the resulting body atoms are triples contained in the training set . Following this procedure2 it will happen quite often that a candidate is predicted by several rules . In this case the maximum of the confidences of these rules is associated as confidence of predicting c. If there are several candidates c and c′ that are predicted with the same confidence , they are ordered in the ranking according to the confidence of the second best rule , ( if this confidence is also the same , the third-best counts , and so on ) . Two candidates that can not be distinguished due to the fact that they are predicted by a set of rules that have exactly the same confidences , are ranked randomly . We call this aggregation method in the following maximum-aggregation . In [ Meilicke et al. , 2020 ] the authors also reported about a noisy-or-aggregation where a candidate predicted by a set of rules will have a higher confidence than a candidate predicted by a subset . It turned out that this method performed worse on all datasets compared to the simpler maximum-aggregation . This was especially caused by the problem that the method can not discriminate between redundant rules that fire for the same reason and rules that capture different aspects . Opposed to a symbolic approach , that requires to explicitly define an aggregation type , KGE approaches have an implicit aggregation technique that is based on the fact that each triple is part of a comprehensive objective function . This is another advantage that KGE models might have compared to rule-based approaches as long as they are based on a rather simple aggregation method . 4 . Case By Case Analysis . It is not easy to distinguish accidental differences , that might be caused by the stochastic nature of the approaches , from systematic differences . We first propose a method to spot essential differences before we discuss what we found with this method . We report occasionally about datasets and KGE models that are first introduced in Section 6.1 .

Summary: The paper presents a comparative study on symbolic methods and embedding methods for knowledge graph completion. From the manually observed ranking results of two kinds of approaches, the authors find that rule-based method AnyBURL tends to perform well on complicated rules that express transitivity of multiple relations, and that KG embedding methods sometimes will output meaningless result due to its learned similarity measure, which may be remedied by rule-based methods. Accordingly, a simple aggregation reranking approach is also proposed, and it shows promising performance gain for several embedding approaches in the KG completion task.